Present Perfect

Last Friday was spent at the data center in Barcelona. Our mission, which we chose to accept, was to install fifteen PowerEdge servers into our new rack, including doing the cabling, operating system, and everything. By my side was the inimitable Jan Schmidt of Jan and Jaime fame.

Predictably, we failed in our endavour. In the end, we managed to physically install eight of the fifteen servers (meaning, cabled up, locked in place and booting), and put most of the other seven on rails. The rack itself is cabled beautifully thanks to Jan, and some of the machines have had their minimal configuration applied to the point where I can access them remotely. Not bad for a seven hour job.

Things would have gone a lot smoother if:

• We had known beforehand that we are supposed to mail the data center before coming. (I wonder what we're going to do when we want to come in quickly because our mail server is down)
• Dell hadn't decided to change their rails slightly, making them a few centimeters longer. After some serious thinking the only solution we could think of was to move the vertical rails in the rack completely, for which we needed a wrench for the IKEA screws (Jan, as always, knows the correct name for those bastards - Allen screws).
• Fedora's CD1 would actually be enough for a minimal install. I am fairly certain this used to be the case in the past, but not anymore - let alone that it's hard to get a minimal install going anyway. The worst part was that it only needed one or two packages from the second CD as far as I could tell, but there was still no way to ignore that problem to get at least grub's boot record written. Annoyance number two with Anaconda - more long-standing - is that when you do an upgrade, you can choose to rewrite the grub configuration, but unless the CD actually ends up updating a kernel package, it will not actually write the grub boot record even if you asked it to. And anyone who has ever had to rewrite their GRUB record manually with grub-install can attest that this only ever works after hours of googling and various failed experiments. In the end, we set up a local http server to install from.
• Dell's PowerEdge servers came with PS/2 ports. I had the foresight to bring a small KVM switch, but I gave up hope using it when noticing there is only USB on the machines - this in contrast to the on-line manuals I for these machines.
• Dell's PowerEdge servers would be a little more like their SC range. The SC's we had at work for the development cluster have two NIC's, and quite sensibly NIC1 maps to eth0 and NIC2 to eth1. The IPMI stuff (which is exposed by a fancy always-on controller card that allows you to get sensor data, reboot machines, and do serial-on-lan) is piggybacked onto the first NIC, and so on our dev platform we use the first NIC for a private network, and the second NIC for the "public" one. The PowerEdge machines however assign eth0 to NIC2 and eth1 to NIC1, and there was no way for me to see which one had the IPMI. After experimenting with DHCP last night remotely, it would seem that the IPMI stuff is in fact piggybacked on NIC2 instead. Weird. The reason this is important to me is that I really do want the IPMI traffic (which isn't adequately encrypted) to go only to the private router.
  Jan was muttering something about Ubuntu and aliases making this easy, but neither of us could find any reference to it on the web. In the past, I've always configured order with alias statements in modules.conf, but of course that only work if you have different NIC's. In the end, fixing the order of eth0/eth1 was easy - Fedora allows you to specify the MAC address in the relevant ifcfg file, and it figures it out from there.

I hardly had a chance to deploy my excellent new Coming Soon tool to manage all the configuration for these machines, even though I had prepared most of the config for the live platform before installing the machines. Anyway, I expect I'll be going back after my week in Belgium to finish up details, and I can spend part of this week working out some of the kinks of my deployment.

Last Friday was spent at the data center in Barcelona. Our mission, which we chose to accept, was to install fifteen PowerEdge servers into our new rack, including doing the...