How I fixed PCIe Training Error: Slot 1 on Dell R200 0GU186

I recently picked up a Dell R200 off craigslist\free to teach my nephew about computers/servers by setting up a Minecraft server.  However when I got it home I had a  PCIe Training Error: Slot 1 problem.  Inspecting the card showed some capacitors that were expanding and probably the cause of the issue.  The R200 has the ability to run 3.5" SATA hard drives off the motherboard itself, but if you wanted to add SAS capabilities, you needed an add-on Perc 5/i or Perc 6/iR card.  Neither configuration supported hot swap.  Amazingly enough, Dell had a card that would add SAS...and not add RAID capabilities.

I replaced the 0GU186 nonRaid SATA card with a Perc 6/iR and kept the same cables from the original card.  Since any data on the hard drives wasn't important to me I was able to find a Perc 6/iR card for 5 dollars shipped instead of the 60 dollars people were wanting for direct replacement.  Tested both hard drives with the Drive Scanning tool built into the 6/iR and both drives tested as no broken blocks and installed ESXi 5.5.x like a champ.  However, moving an iso image onto the datastore made 1 hard drive sound like a bench grinder.

Just because a drive utility says everything is fine, doesn't mean all is going to be well in the near future.  Run a few tests.  I'll consider this more learning for my nephew to hear what a hard drive dying sounds like.

note: I would not recommend the Dell R200, go find an R210 II (not the model I) or Dell 410 if you really want a 1u

Initial Setup of a Dell Powerconnect 2724 switch on home network - Part I

With all the servers I've picked up recently I needed a way to network them all together.  Being outside of my normal work scope this took a bit of effort on my part to research.  Not only do I have an existing network currently running in my household, I have a goal to not interrupt others in the household from their own internet usage.  This is the process I needed to setup my surplus Dell PowerConnect 2724 switch using the web interface for use with an existing network.

Upgrading and testing HP RAM in a Dell server

I recently went to try some 16GB 10600R RAM from a HP DL360 in my Dell r510/r710 which had 6x2gbs/12x2gbs respectively of UDIMMs.  I have a Dell T130 which specifically requires UDIMMs and not RDIMMs, so I was wondering if it was possible for r510s/r710s to have the same problem. I've seen enough Dell servers on eBay listing RDIMMs but wanted to empirically double check. A quick check of the owner's manual probably would answer this question.  After installing the 16 GB 10600R, I went into my iDRAC system to start up the system and noticed something weird.  I was downgraded from my Enterprise license to Express on my iDRAC system in my Dell r510. My goal is to see if 10600R ram works in this device . I need to see if Dell locks your system specs to the iDRAC license.

TL;DR:
Result: The end result was that the iDRAC snap-in card came loose and I didn't lose Enterprise.  This error shows if you don’t have an iDRAC Enterprise card installed, change hardware and had "Collect System Inventory on Restart" enabled.  Dell did not implement a separate license for Enterprise iDRACs until generation 12 and generation 13 servers, ie r720 and r730 servers.  It is only hardware module to enable Enterprise before 12g which runs about 10-20dollars on ebay.

See this document for details: http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4491-00-00-20-17-06-31/Lifecyclecontroller_5F00_PartReplacement_5F00_lc.pdf?forcedownload=true

HP RDIMM RAM worked fine in my Dell r510 and Dell r710.

However, something to note. is that if you fill all of the memory bank channels.  Your RAM will decrease in speed from 1333/1067 MHz to 800 MHz.  There's a note at the bottom of this post talking about which RAM numbers to look for if you're on eBay looking for a used server.   If you'd like more details on that open up the post and skip to the end.  If you've had any experience where HP RAM didn't work correctly in a Dell Server please leave that in the comments so I can try and investigate that.  Especially if it was a mix and match.

Setting up iDRAC from front LCD panel without computer being on.

You are able to set up Dell's iDRAC6 from the front LCD panel without ever turning the computer on from the front LCD panel.

From the home screen you'll see Setup|View
Choose Setup

DRAC|Set Error  and if you hit right once you'll see Set Home
*if you see BMC, you have a system without iDRAC Express or iDRAC Enterprise...sorry
Choose DRAC

After Choosing DRAC, determine if you're using DHCP or Static. Since I'm going with Static
Choose Static
Enter IP address that matches your gateway.  For example, if Gateway(Gtw) is 192.168.1.1, your IP address better start with 192.168.1.xxx.
I've chosen 192.168.1.122
Do this by selecting the character to change, hitting enter so the character blinks, adjust to desired value. Be careful not to set .100. when you are desiring .001., the display always shows 3 digits. When completed scroll right till next option
Enter Subnet(Sun), for a home user 255.255.255.0 works.
When completed scroll right till next option
Enter Gateway(Gtw). for me, i'm using 192.168.1.1
When completed scroll right till next option
Set DNS1.
Change values to 008.008.008.008 if you care, or leave 000.000.000.000
When completed scroll right till next option
Set DNS2
Change values to 008.008.004.004 if you care, or leave 000.000.000.000
*these values are Googles DNS servers
When completed scroll right till next option
You should now be given an option to SAVE
DO SO

Your changes are now in effect but make sure you double check that the values were saved. You can now log into your iDRAC system at whichever IP address you chose.  If you had a problem double check your gateway.

Fixing Error : The event log has overflowed and needs to be cleared.

I recently bought 2 Dell R710 Servers and needed to start testing them to see if they were functional or had issues to be resolved before trying to sell them.  When looking at the bios I confirmed the R710 's had Dual Xeon 5530 's running at 2.4 GHz with 2.66 GHz turbo.  However I didn't see anything mention about hyper-threading within the BIOS.  I remembered an easy way to see how many real cores/HyperThread cores you have is to run the MpMemory test which uses every core to put your RAM through its paces. So I loaded it up, went to custom scan, selected all the short tests, not just the default of 7/8, hit enter and got an error.

The event log has overflowed and needs to be cleared.
No existing memory related problems were found
Test(s) Failed

Here's how I went about fixing it.

Making the Diagnostic Tool Bootdisk for Dell Servers.

Making the Diagnostic Tool Bootdisk for Dell Servers.

I started wondering how I was going to test a dell 2950 (9th generation, 9g) if it didn't have a diagnostic tool built into the boot screen like my dell r510(11th generation, 11g) does.  I started hunting around on google and found an interesting link from their knowledge base

How to Run Hardware Diagnostics on your PowerEdge Server

I expanded all the possible choices, 12g and 13g just to see what improvements have been made
11g To see a reference to my own r510,
and then 9g and 10g to try and figure out my approach to my current ~70lb paperweight 2950s.

Looks pretty simple. (It wasn't.) Lets get started.

Western Digital Enterprise level Hard Drives (hdd)

I recently posted a QA post about Western Digital Hard drives.  However after searching on Ebay for some hard drives that were more moderately priced I realized none of the model numbers matched what I was seeing on ebay.  Took me a bit of research to find some information but I found Western Digital has recently updated their website and all of the google links, (and half of Western Digital's links) associated to Western Digital's .pdfs do not work.  Slowly deleting chunks out of the URL eventually put me at their main page.  Western Digital's Search function worked very well that I eventually found what I was looking for.  WD RE4 direct support page . Each page has alot of resources and forum posts addressing user problems so it's worth a read if you have any Western Digital Hard drive.  I included some more information below incase you're looking at other older WD drives or want a few model numbers.

How to check your system components for errors in a dell r510

How to check you system components for errors in a dell r510

Whether you're buying hardware direct from the manufacturer, from an auction on ebay, or from government surplus website you want to confirm that your purchase is working as intended before you put in some serious work..  Many ebay auctions come with a 14-30 day DOA warranty so you'll need to run as many tests quickly to be able to start the return process if something is wrong.

This blog tries to make zero assumptions as to previous knowledge.  I know when I've done something completely out of my comfort zone I start to wonder, 'Is this really supposed to be taking this long'.  This post is going to read half instructional and half like a twitter feed just so you can see what's possible with my Dell r510 hardware and see how much time this process takes. With that preface lets get this show on the road.

The Dell iDRAC6 - Part 1 - iDRAC6 Configuration Utility

The Dell iDRAC or the integrated Dell Remote Access Controller is the core of the remote management of a Dell Server.  This is one of the main differences and advantages of buying a server over a very well equipped Consumer Motherboard and making it into a server. A remote management tool allows us to do many functionalities that have often been limited to us unless we're right next to the PC, The iDRAC gives us access to everything you see scroll on the computer screen before it loads the Operating System (OS).  Each generation of Dell Servers has come with advancements in the iDRAC system. My r510 has the iDRAC6 which were used with the 11g or 11th generation server, My T130 has an iDRAC8 because it's an 13g server from dell.

However to use the iDRAC you must first configure how to find it through your network.  This is where the iDRAC6 Configuration Utility comes into play that you see on your boot screens when your server first starts up.  It's one of the quickest checks your system does during booting and the last one it does on my system before booting into your OS.  Be prepared to hit Ctrl + E as soon as your virtual drives are found.  Lets see what those choices are.

Q and A Tangents 2 - Dell hard drives and storage

Q and A Tangents 2 - Dell hard drives and storage

Q: What hard drive can I put into a dell r510
A: Apparently 2gb is the maximum you can put into a dell server, not sure if that's with Perc 6i only or also with the H700.  Even then, there are certain harddrives that are preferred and others that won't even be recognized correctly.

Need to find a list of supported drives for r510 etc and why others might work, but not be recommended. Need to find a list of sata3 and sata6 capabilities too. TODO

My favorite reddit  r/homelab/ actually answered this, I copied their answer down below so I can find it faster in the future.  Blogger quotes screws up everything so the rest of this Answer is directly from r/homelab.  But for me and my Perc6/i it basically forces me to find some 2tb WD RE4's YYS which are slowly going out of production. Below doesn't mention what is available with the Perc 6'i though. TODO

Dell Perc6/i

Lots of Dell servers still have Perc6/i Raid cards in them.

These cards will not be able to use a drive over 2Tb and Dell have no plans of ever updating the firmware to support the larger drives we want to use.

If your server comes with a Perc6/i and you want to use larger drives then you will need to upgrade it to a newer card, the 2 most popular options if you want to stay with Dell are the H200 or the H700.

Now which out of the 2 you want will depend on your needs, -if you are going to be running FreenNAS or any such solution for your storage you will want to go with the H200 as this card supports JBOD mode which is what you will want to pass SMART data through to the OS, The H700 does not support this mode so will not be suitable for these types of storage solution. If you are interested in Hardware Raid from the card then both cards support this, the H700 is the better of the 2 cards as it supports both a BBWC and Raid Levels 0,1,5,6,10,50,60 Whereas the H200 does not support BBWC and will only do Raid levels 0,1,10.

If you are not using drives over 2Tb is possible to use the Perc6/i but be aware the card is only a 3Gb/s card where as the other 2 are both 6 Gb/s.

Q: What are the advantages and disadvantages of using (WD) Western Digital RE drives
A: Power Consumption and sound seem to be large problems for WD RE drives and interestingly enough the link below talks about Red Pro drives and how similar they are in power requirements and sound to the RE drives vs the normal REDs.  I've got dual 870w gold power supplies so power shouldn't be an issue, but I'm not looking to go broke paying to run a server either.  I can supply it, but do I want to.

The main problem for me is going to be sound volume.  I need some quiet drives

Found the jackpot...by looking where I'm supposed to look.

Q and A Tangents 1 - Articles I'm reading answering questions i'm questioning

Q and A Tangents 1

I believe You should always be learning.  So I read alot of technical articles to answer questions which usually inspire other questions about the stuff I'm working on.  I thought I might write blog posts about questions I've thought up that I get answered from other's posts about what I'm working on.  Unfortunately this will probably lead to more questions so I'll come back to those topics later on. Hopefully this inspires some conversation in the comments section. Maybe if someone sees my unanswered questions they'll answer in a comment or disagree with an answer I have. I may link a post in the question that inspired the question as well.

This is to mimic the hardware review sites I read that kind of aggregate other peoples reviews at the end of their daily posts.  Need to figure out how they do that.  TODO

Q: Can a dell r515 use an amd Opteron 43xx processor? Half the support files only mention 4300.  All mention 4200 and 4100.  Is this a function of being a release II (2) version?

Installing ESXi 5.5 on a Dell r510 (part 2)

Installing ESXi on a Dell r510 (part 2)

• This is a continuation of installing ESXi 5.5 onto a dell r510 with the ESXi ISO from Dell instead of vmware's website.  This kept us from needing to make an account from VMware for the download.  We'll see what that approach does in a later post or just delayed the inevitable. TODO
• So we're chugging along, it takes a bit to load everything up.  Especially from my USB 2.0 stick.  Acquired for $4.99 at office depot because I wanted to get this started, but really the read times on http://usb.userbenchmark.com/SpeedTest/1349/SanDisk-Cruzer-Blade are fairly terrible.  
• Fantastic news though.  Once we got into the Welcome to VMware ESXi 5.5.0 Installation screen. my router found 'A' nic plugged into it...should have found 2, but baby steps. My guess is it found the idrac shared port. (I previously set up IDrac to share nic 1 instead of using it's own port because I only have an 4 port router right now so router ports are hard to come by as of right now.  However the Idrac should be named idrac-XXXXXX where XXXXXX is my Dell service tag, and it's blank. Baby steps.
• TODO-I need to figure out if there's a set standard for rj45 wire cabling color or if it is basically shop independent.  having wires going all over hell and back would be nice to try and narrow things down by color, and then look for labels.

• I chose to install on my 4 disk raid 1+0 array and then hit my first snafu.  It said Virtualization technology was either not turned on in the bios or not possible on this chip.  Since I'm using a xeon e5503, xeons should all have Virtualization capabilities but I'm using near the lowest rung of  the Xeon e55xx series so I wasn't sure, however I swore I had turned it on in the bios.  So I exited the install before continuing, no reason to continue installing a Hypervisor for virtualization if the chip can't handle it.
• Second shot at installation doesn't produce the error but it gives a warning that my hard drives will be partitioned. Accept the notification, not sure why, but lets hit F11 and install
• Install seemed to hang at 28% for a few minutes, lets hope it continues and its just because of my slow read speed usb drive. 
• Success, need to remove the usb drive before hitting enter for reboot.
• and again...the reboot shoots out my CD Drive.  That'll be annoying.
• Ok ESXi is installed, however I need to manage it remotely and try to make the server useful.  The screen of the server says, download tools to manage this host from: 192.168.1.102. Going to that page brings up a default page on where to download the management tools
• Now to figure out what Vsphere Client is  versus the next download link of VMware vCenter
• The vSphere client luckily has a direct download link....of a 353mb file downloading at 1Mb/s, not too slow but definitely not the fastest.
• The vmware vCenter link doesn't have a direct link and puts you at some huge download page.  I have no idea what all this stuff is, so this is going to be a little more work.
• https://my.vmware.com/web/vmware/downloads
• Its also possible that vCenter is not free past the trial period, this will be annoying.
• Installing Vsphere Client
• installing microsoft J# compiler thing, probably going to need a reboot
• Logged into the Client by knowing it's IP address, and using root/the password I set during ESXi installation.  However its telling me I only have 60days before my vms are destroyed....so I'll need to figure that out soon. TODO
• WHOOOO Went into IDrac and told the Idrac to get its IP address from DHCP instead of me assigning it directly.  I believe that I didn't fill in all the information for a static IP address and it never worked because of it.  I also changed it back from having the Idrac be accessed over a shared LOM 1 back to the Idrac port itself.  I was getting worried because the IDrac wasn't even lightning up when something was plugged in and I wasn't sure if I just had a crap cable. I still cannot get anything from LOM2 so I'm wondering if it's setup as failover and not useful when LOM1 is working.  I still have a 4port gigabit card that's working however that is going to be used for something, just not sure what.
• Proof that it's working. Interesting to see what's being used by default with ESXi without anything else running. And now I'm going to be sad because the r510 I bought came with 2x e5503 's which are some of the only processors without hyperthreading.  However 2x e5520s in a matched pair...are 10 dollars for the set off ebay.  Might be something I'm going to be upgrading soon.  Baby Steps.  Lets get a VM running first.

• It looks like I need a vCenter server running to access all the ESXi hosts that would be running per server.  Probably overkill right now since I only have 1 server running. and since the 2 servers I have are manufactured 7 years apart. I question if i'm going to be able to do much just yet with them moving vms back and forth,  I need to figure out what the long term purposes are.  Where will data be stored and should I change my raid array on r510 with 5x160gbs

Installing ESXi 5.5 on a Dell r510 (Part 1) - PreInstallation

Installing ESXi on a Dell r510 (Part 1) - PreInstallation

A little pre-setup,

• server turns on. Hooray. As a surplus buy with no kind of warranty or testing, this is very good
• Since the homelab is not sitting in a good environment I made an air filter out of a all black knee high nylon stocking. Put the front bezel inside it and snapped it right back in.
• Set up DiskGroup/Virtual Disk/
• I have set up my raid 10 with 4 drives and 1 global hotspare.

I've read about not ever using a hotspare, that the best practice would be when something fails, run a backup, then install the new hard drive, then have the raid array rebuild.  This does have downtime, but if something goes wrong when rebuilding the new array with the hotspare, you basically lose all your data from between now and the last backup

• Setup Idrac...sorta
• I've changed the idrac gateway settings to my actual gateway. The Linksys wrt54gl assigning itself 192.168.1.1 instead of what dell looks for being 192.168.0.1 so things went a little crazy.
• Plugged in ethernet wires to idrac and both ports onboard the r510 and nothing is detected by the router. todo-->fixed Ctrl+E is your friend during boot

• Now it's time to set up ESXi so i can load something on this beast.
• First thing you want to do is see what version you're going to install. Unfortunately when you're being economical there's a chance that ESXi is no longer being supported with your hardware.  Doesn't mean the version won't work, but it might get angry.  I'll let you know what that anger entails later with some testing.
• So lets check if our hardware is supported.
• http://www.virten.net/2015/03/vmware-esxi-6-0-unsupported-hardware/
• pretty good post about what's supported and what isn't, supposedly being updated but I doubt that will last for long
• http://www.vmware.com/resources/compatibility/search.php?
• VMware Official compatibility guide
• http://www.vmware.com/resources/compatibility/search.php?keyword=13819&details=1&deviceCategory=server&productid=13819
• for my purpose with my Dell r510 I searched for Dell and xeon 55xx
• According to this Dell r510 only supported VMware 5.5 u3 , but the dell r710 supports 6.0 with the same chip soooo, lets see what sticks and try vmware 6.0 first
• Need to Download ESXi 
• https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/6_0#custom_iso 
• however I notice that there's a tab for Custom ISO's
• That sounds handy, clearly Dell should have one ready for me....
• and they don't list one vmware's website, clearly they should have one somewhere.  Lets go find it.
• googled "Dell has released an ESXi" yeah, english majors cried here
• the second link however is very interesting for another project I'm going to be working on where ESXi does not work with software raid? http://topics-cdn.dell.com/pdf/vmware-esxi-6.x_Release%20Notes_en-us.pdf page 10
• You can download the Dell-customized ESXi installer image from dell.com/support/drivers. To create the ESXi installer media, complete the following procedure:
  1. Click Get drivers and downloads. For more information, see Downloading drivers and firmware .
  2. Select your operating system as ESXi and download the ISO image.
• Official instructions...seem to miss a few steps
• You need to goto the drivers and downloads of your machine.
• To get there goto the generic support page http://www.dell.com/support/home/us/en/04/ and put in your Dell tag.  This will bring up the page of your machine that has a link to find the drivers and downloads section.
• Changing your OS, however does not give the option to get esxi 6.0 and stops at 5.5.  Ok vmware you win for now.

• EDIT: As an alternative I found this ESXi 6.0 Dell Installer Link This Link however has a section dedicated to what systems are supported.  
• Yeah...the dell r710 and r515 are supported but not the 510. Bite me Dell. We'll try and install this anyway and see what happens later on, not sure which install to use though. todo

• Well, that's alot of files and possible updates I need to deal with todo
• Looking around in the headings finally pans out in Enterprise Solutions and I download the most current ESXi iso image.
• I right clicked download and used save link as because reading a line right above it that says this will install after being downloaded and requires a restart on a hypervisor, or anything on this page is questionable.  
• I also just managed to blow up google chrome, and destroyed this blog post's formatting somehow.  Luckily I'm used to finding bugs and after alot of shift tabs to go backwards with my bullets and copying things into notepad and then back out to remove saved blogger formatting, all is fixed.
• Need to put the ESXi iso bootable image onto something.
• Since the file is only 360 mb this could go directly onto a cd and will probably be my plan in the future once i know that I don't have any other things to put onto this cd.
• First step is to put it on a USB drive and see if anything works.
• So how does one make a USB bootable for ESXi.
• Google lights the way again with http://www.virten.net/2014/12/howto-create-a-bootable-esxi-installer-usb-flash-drive/
• So I download Rufus to make the USB, however on Rufus website there's a good point. If you need to make an ISO yourself of something you own, you can use one of my favorite pieces of software Imgburn.  However I would recommend going to oldversion, the latest version of imgburn had some nonsense bundled with it.
• Image shows a few things you should be clicking on

• Boot from usb
• The server didn't find a place to boot from because under harddrive it listed the boot order as perc first and the usb stick second.
• Press F11 to goto boot menu, goto harddrive and selection the Dell-ESXi-5.5U3-xxxxx-Axx installer
• Holy Hell stuff is loading, next blog post we'll see how well it does.
• Need to figure out how to do this remotely instead of being right next to this server somehow through a console.  TODO

Continued on in part 2: http://archyver.blogspot.com/2016/10/installing-esxi-55-on-dell-r510-part-2.html

Racks and Rack Holes - The differences and why you care.

I bought a Dell T130 server recently after finding a good deal on it, and if 1 is good, more is better. After looking at some auction websites I found a Dell R510 (R stands for Rack mount, T stands for Tower is the dell server ecosystem) and purchased that as well.  Once you start finding server hardware you realize you're probably going to need space to store it for easy access, This brings up Racks.  I'm not really a hardware guy, never been a sysadmin from that perspective but all software computer guys have to deal with their own hardware.

I found a good guide describing one of the main differences between racks.  Rack Holes.

Racks use rails to mount servers and come with a few different flavors for mounting them, Round Holes, and Square Holes.  The following movie talks about the differences and the hardware needed for Dell Rack Systems.  Hopefully this helps you decide on what kind of rack you should be looking for on ebay/craigslist if you start branching out into a more robust layout.