Last Thursday I flew out to Madrid with Sergio, our sysadmin, to go and install a mini-platform for streaming at a customer's data center. 9 machines, two guys, two days, plenty of time, no
We were supposed to bring some stickers to put on the rack, but they were not ready in time. Of course they would be fed-exed to Madrid as soon as they arrived at the office.
I guess my suggestion to Sergio that "I did not want to get up earlier than usual" didn't quite translate, as he had us booked on a plane leaving at 9 in the morning. That's usually when I wake up. Normally that wouldn't bother me, but for a good deployment I need a focused mind, and I already had a few rough nights before that.
Anyway, get on the plane, get off, take a cab, arrive at the data center. The building looks incredibly old and reminds me of a prison. Security is a little strange - we have to wait at the entrance for fifteen minutes, I get a swipe card that has a corner cut off and fixed with sticky tape. I tell the guard that I doubt it will work, he's confident it will, and we both try to get me through five times before he gives up and has the card replaced. By that time the guy that is going to show us into the data center gives up on us and takes another customer in, obviously.
Anyway, some fumbling later, during which I ended up with two access cards instead of one, and we get taken to our rack. We walk past the first door, and throughout the hall there's a good twenty meter run of cable in a loop lying on the floor. The guy asks Sergio not to roll his trolley over the cable so they don't lose network. O.K.
We get to the rack. We get half a rack. We get the middle part of the rack for that though. Ok, fine, no problem yet. Two power blocks at the bottom, we confirm that they are in fact on different circuits. Usually people put circuit A at the top and B at the bottom, but whatever.
So we ask, innocently, "Where's the network cable ?" "What network cable ?" "You know, the one that gives us access to the rest of the internet." "Nobody told me to pull a network cable to this rack."
...
Sergio starts arguing, says he's arranged everything a month before with his contact, calls his contact, his contact explains that he put in the request more than a week ago but that "things are a little bureaucratic and can take time". Well ok - we have two days of time for you before we have to go back to Barcelona. Apparently the company is structured such that Data Center Maintenance is one company, Network Infrastructure another, and our contact is in a company that pulls the companies together to get stuff done. Or not, as the case may be.
I ask Maintenance guy if they have a lot of customers in their data center that don't request Internet. He shrugs. I consider asking where the hole in the wall is for the pigeons that I'm going to put in my rack since obviously I need *some* way to get packets in and out of our servers, but I bite my tongue instead.
OK, let's put pressure on our contact from above to make sure he drops everything he's doing and gets us some network. Meanwhile, let's install servers. We accompany Maintenance guy to the storage room. Maintenance guy takes first server box and throws it on the transporter from a good meter away. I was on the phone when he did this; I look puzzled at Sergio and nod my head to indicate he should say something to the guy. Sergio asks the guy not to throw our servers around; the guy gets upset.
We transport the servers a good ten meters down the hall. Then we have to take off the boxes off the transporter and to our rack, which is a lot further than ten meters, so I wonder what the transporter was useful for in the first place. From that point on, Sergio spent most of the day on the phone, trying to get people to work on the network, while I install the machines and cable the rack. I install the first machine (which will be the installation server) and leave it at that until I have net.
Our PDU units are nowhere to be found in the storage room. We know they were delivered separately from everything else because it took three additional weeks. We help them take the storage room apart (it contains boxes with delivery dates ranging back to 2004, unopened), but we still don't find them. They don't know who actually received the shipment, they have no record of where the box has gone, and they don't really know where it could be. Sergio tells them to start looking harder.
We get a cable around 14.30; it is 100 MBit, but will be upgraded soon, they assure us. Happily, I plug the cable in, and try to get somewhere. Doesn't work. We call our contact; apparently the class C we requested has been pre-reserved for us, but only 64 addresses assigned. We ask to get that changed to the full range - what does it matter if they are reserved anyway ? It makes it easier on me because I can keep the fourth number consistent with other platforms. In any case, we still can't get anywhere - the first hop after the router is on a 192.168.x.y subnet.
Our contact tells us that, while we now have a cable to the switch, our address range is not properly configured. Of course this is handled by another company, the one that takes care of Routing. Sergio loses the rest of the day calling our contact, his boss, our boss, and back and forth. He explains his contact how traceroute works and how he can use it to confirm that the network guys that claim that everything works are wrong, since he clearly cannot reach the gateway they're asking us to use.
Walking around the data center, half of the grills in the floor are bent out of shape and wobbly. At least 30% of the racks are not closed; some just don't have doors at all, others have doors that are ajar. I can easily take backup tapes from seven companies if I would want to. Sergio tells me that they managed to find the PDU units - they were in some small room that no one uses, and there was nothing in the room except this box of ours.
Somewhere during the afternoon someone drops by; apparently he's the head of projects for the data center room we're in. He asks how it's going; he lets us know that Maintenance guy complained to him that we complained about him throwing our servers around. (I know very few people that complain to their boss when they do their job so badly that the customer complains about it).
I was planning to install a first machine, check out the configuration using savon, and completely set up all the services on the first machine to then PXE-install all the others. Of course, as long as I can't get to the network, that is not going to happen. We leave the data center around 21.30, still no network. Our contact assures us that the request has been processed and will take full effect during the night.
On the way out, we get asked to hand in our papers. What papers ? Apparently we're supposed to fill in a paper that has the serial number for our laptop and stuff like that. The guy at the entrance is terribly surprised that no one asked us to fill one in in the morning - "this never happens", he assures us.
We take a cab back to Madrid, drop our stuff at the hotel, and I let Sergio take me around Madrid (we stayed at the Victoria 4 right around the corner from Puerta del Sol) for some excellent tapas: La casa del abuelo specializing in shrimp tapas (with photos of Danny DeVito visiting on the wall), Casa Lucio (which apparently has the best sunny-side-up-with-fries-and-your-choice-of-bacon/chorizo/cheese/... racion, plus pictures of Bruce Willis and Antonio Banderas visiting), and another place where we settle down with sangria, a good plate of ham, and pimientos. We promise ourselves to not go to the data center in the morning until we can confirm from the hotel room that the network is working. We pass buy this peculiarly themed tapas bar. I go in to ask what Belgian tapas are. The woman behind the counter is totally honest about it: "There are no Belgian tapas, but this is a tourist area, so .... Good for her.
We get up the next morning, try the connection to our server , still not working. We call our contact, call our boss, keep applying pressure. First excuse is "the change has been made but it takes time to propogate to the internet". Uh, we are in your data center, you are on your company's network, at least one of us two should have a correct route after 4 hours. " Their other comeback is "We're surprised that you need network to do the installation, all our customers do remote installations." Uh, sure. "Why don't you install the machines and update them over the network later ?" Not sure if it's a good idea to leave unpatched machines connected to a cable that will have Internet at some random unpredictable point in the future.
Meanwhile we work on some other things. By 12 we need to leave the hotel, so we have no choice but to go to the data center. Through security again.
My bag passes through the X-ray without further inspection. The guard asks me if I have any cameras, and I play dumb and say no. Even though I have my camera in the bag, clearly visible on the X-ray, plus two cameras on my phone and one in my N800. This time we have to fill in a paper. Sergio is asked to not only note the number for his laptop, which the guard saw on the X-ray, but also note the serial number of his mouse. The guard asks me if maybe I have a laptop as well, and if so can I fill in the paper ? I wonder how he did not see my laptop on the X-ray...
We change our plans and start installing all the machines from CD's, just to make sure that we have all the machines accessible over the net so we can do what we want with them later. I have a small IPMI problem that I brought on myself by picking the wrong subnet mask, which took some time to figure out - I can blame no one else but myself. We look for any kind of useful work we can do. By now the stickers have arrived through Fed-Ex (we had to pass on name of the driver, model of the car, and license plate to the data center security guys just to get it delivered) so we put them on a machine and on the door. We get warned that the one on the door will be taken off anyway, but whatever.
| Bling |
 |
By 13.30, we suddenly have working network. Now, we had planned to take our plane back at 18.00, and had already resigned ourselves to having to come back. Suddenly the cards have changed - maybe it's better to keep at it and do everything needed to make sure we don't need to come back later ?
We call Sofie, get our ticket upgraded to business, plan to leave at 21.00, and start working like crazy to get stuff set up. I can finally play around with the configuration and store it directly in the repository, adapting as I go the previous layers from the other platform, splitting them as needed where it makes sense.
We have our "lunch" around six, where we get confronted with the vending machines from hell. Neither the food nor the drink machine accepts coins of 1 or 2 euros. We don't have enough small change to get more than one item. Someone comes in and explains to us that these machines refuse 1 euro coins if they don't have enough small change to possibly give you back money. Never mind that I am trying to get some items that cost exactly 1 euro, or that even if that wasn't the case I would be willing to not get change as long as I get some calories. The woman is nice enough though to swap her change with our coins for a small fee.
This is where we pull a nice real-life hack - we first buy some drinks, filling the machine with some small change and getting our drinks. Then it has enough small change to accept our euro coins, so we put those in, but don't buy - we push the Ben Folds Five button (aka the "give me my money back you bitch" button). Most vending machines will actually give you small change at that point. This one does too. So we then use the small change to buy food from the other machine. Sheesh.
We get back to work, move our flight one more time, to the 22.15 flight - the last one with SpanAir out of Madrid. We finish up around 21.10. Except there's no way I can close our rack. Not at the front, not at the back. Maintenance guy tells me not to worry, that he will close the racks. I take another look at the racks around me, half of them wide open, but I have no time left to argue.
We jump in the cab, get to the airport, and arrive at checkin two minutes after cut-off time. It's the last SpanAir flight out of Madrid. Bummer.
I consider taking a plane to Malaga instead - Kristien is there for the weekend because of work, the radio has taken a bunch of listeners on a trip with some of the radio hosts. But I was warned that no partners were allowed, so I decide against it.
Sergio goes to customer service, they try for fifteen minutes to get us tickets on the Iberia flight back, but in the end they give up because their printer breaks down. Of course, we're in T4, and Iberia leaves from T3. Since there are arrows pointing to T1 and T2, this should be easy, but no - T3 is in somewhere else entirely than T1, T2 and T4 combined, and we need to take a bus there. Nobody is really sure where that bus leaves - apparently we need to go up, though there is no way to go up nearby.
After some searching and cursing we get on the bus, get into the hall, and split up - Sergio goes to buy tickets, I go to checkin. I'll save you the details on how we cut past the queue of people checking in for Santiago de Chile just to not miss our window again.
And after all of that, obviously our flight gets delayed. We arrive back in Barcelona at 0:55, I take the last bus back to the city, and join the guys from work at Belchica for a well-deserved Kriek...
I give us a 75% chance that we've done enough to not have to come back later to finish the installation...
Last Thursday I flew out to Madrid with Sergio, our sysadmin, to go and install a mini-platform for streaming at a customer's data center. 9 machines, two guys, two days,...
![[lang]](/images/eye.png)