Present Perfect


Picture Gallery
Present Perfect


Filed under: Fluendo,GStreamer — Thomas @ 15:39


I have yet to try Second Life myself (it's for work, honest - I need to see whether we can stream out of the box to it!), but the next version of the Linux client will use GStreamer.  The fact that I can't recall seeing any linden labs email addresses asking questions on the list is probably a testament to how far GStreamer has come over the years.  They don't even need to ask questions anymore!

In unrelated news, this afternoon a single one of our customers managed to peak over 1 GBit/sec on our platform.  Kaching.  Next stop, 5 Gbit/sec.


Filed under: Fluendo — Thomas @ 21:33


Woah, it's good to know people care about Fluendo. It didn't take much time after Christian's announcement on his blog for people to deluge me with questions. So let's save me some time by replying publically.

Yes - Christian, Wim and Edward are leaving as full-time employees of Fluendo. The reasons for these are many and varied, as is always the case, and I'm sure you could extract some of them with sufficient application of free beers at GUADEC :) Same goes for us. But beyond that it's between us and them. We wish them the best of luck with striking out on their own. Tim, the GStreamer bugmaster, is joining them - he was contracting part-time for us before.

Are we dropping GStreamer ? No, GStreamer 0.10 has been a success on all accounts from both sides of the fence. I think this is obvious, given that today you can go to Fluendo, Collabora, Opened Hand, and tomorrow a new company, to get consulting done on GStreamer. A lot of big companies don't even tell us anymore that they use GStreamer and want help - they just Do It (which amusingly is the result of your project being Free Software in the first place... The more consulting you do, the smaller the percentage of businesses that need consulting. The hope is that the absolute pie gets bigger because of the success of your project though. But I digress)

GStreamer is not going to suffer because of this. If anything, you should expect to see the community evolve, because sadly it is just too damn easy to not communicate enough with your community if a lot of the core developers are sitting next to each other holding hands all day. Maybe we can finally shut up the conspiracy theorists that claim Fluendo have stolen GStreamer :)

So what about Fluendo ? Our plans haven't changed. We still work on GStreamer to make it better. We still make products on top of it, like our codecs and our DVD player, which we hope to sell to you guys to solve legal issues, and obviously also to get filthily rich in the process. And we're still working on Flumotion and Elisa, and the parts that are free will be as free as they have ever been.

And what about our current customers ? They needn't worry either. Business goes on as usual. Contracts will be completed. We're a serious company and we respect our customers.

Is there anyone left ? This is the funniest question - somehow people think that we're still this small five person company. And admittedly I sometimes have trouble coping with the changes over the last year. But today, Fluendo is a group of three companies, with around 25 people spread across, some of whom you'll probably never get to know (because they're not community people - our office manager keeps complaining about this crap called Linux) and all of us sharing the same office space.

So, what about me ? Well, I'm glad two of you care. On a personal level, it is sad to see friends leave the company. I've been working on GStreamer for almost as long as Christian and Wim - must be seven years now - and any hacker out there knows that this creates a bond. I still have my GStreamer shirt, one of the many high-quality shirts with actual embroidery, that Christian sprung for out of his own pocket, for everyone who wanted one. And one of my best GStreamer moments was still the GStreamer weekend in the Norwegian cabin, shared with a class of physical education students, only reachable after an hour of wading through waist-deep snow.

Before Fluendo, I would introduce myself at conferences hunching my back and saying, with a meek voice, "I hack on GStreamer, I'm sorry, don't hurt me". Now, any of us can proudly say "you're welcome" when people thank us for our work, as we used to thank the hundreds of other hackers whose hard work we use every day.

And for me, learning the difficulties of starting and growing a company as I go, having a group of people leave for the first time is a coming-of-age thing. It is something you never want to happen but cannot avoid at the same time - making the business that you make one step further on the way to being a real business, and no longer a small group of friends working together.

And before I get too sentimental, it should be obvious by now that Fluendo is hiring! If you want to step into the footsteps of giants, help us fix bugs in GStreamer and GStreamer-using applications, help us write plugins for more codecs, help us support and package plugins, drop me a line at the usual place.

And for Flumotion, we are also going to be hiring the next two months - we're looking for Flumotion developers, web developers, and a sysadmin to help manage our platform, as well as a customer support engineer and a QA engineer. (It is probably time to put stuff like this on our website, but until we have time to do so, it doesn't hurt to put this out there and have you contact us directly.)

If you have more questions or are still worried feel free to mail any of us (at Fluendo, or the "leavers") directly.  And best of luck to our three musketeers in the next venture.

Go home; find a wench. Raise fat babies, and live a good, long life.

Pigeon Center

Filed under: Fluendo — Thomas @ 23:37


Last Thursday I flew out to Madrid with Sergio, our sysadmin, to go and install a mini-platform for streaming at a customer's data center. 9 machines, two guys, two days, plenty of time, no

We were supposed to bring some stickers to put on the rack, but they were not ready in time. Of course they would be fed-exed to Madrid as soon as they arrived at the office.

I guess my suggestion to Sergio that "I did not want to get up earlier than usual" didn't quite translate, as he had us booked on a plane leaving at 9 in the morning. That's usually when I wake up. Normally that wouldn't bother me, but for a good deployment I need a focused mind, and I already had a few rough nights before that.

Anyway, get on the plane, get off, take a cab, arrive at the data center. The building looks incredibly old and reminds me of a prison. Security is a little strange - we have to wait at the entrance for fifteen minutes, I get a swipe card that has a corner cut off and fixed with sticky tape. I tell the guard that I doubt it will work, he's confident it will, and we both try to get me through five times before he gives up and has the card replaced. By that time the guy that is going to show us into the data center gives up on us and takes another customer in, obviously.

Anyway, some fumbling later, during which I ended up with two access cards instead of one, and we get taken to our rack. We walk past the first door, and throughout the hall there's a good twenty meter run of cable in a loop lying on the floor. The guy asks Sergio not to roll his trolley over the cable so they don't lose network. O.K.

We get to the rack. We get half a rack. We get the middle part of the rack for that though. Ok, fine, no problem yet. Two power blocks at the bottom, we confirm that they are in fact on different circuits. Usually people put circuit A at the top and B at the bottom, but whatever.

So we ask, innocently, "Where's the network cable ?" "What network cable ?" "You know, the one that gives us access to the rest of the internet." "Nobody told me to pull a network cable to this rack."


Sergio starts arguing, says he's arranged everything a month before with his contact, calls his contact, his contact explains that he put in the request more than a week ago but that "things are a little bureaucratic and can take time". Well ok - we have two days of time for you before we have to go back to Barcelona. Apparently the company is structured such that Data Center Maintenance is one company, Network Infrastructure another, and our contact is in a company that pulls the companies together to get stuff done. Or not, as the case may be.

I ask Maintenance guy if they have a lot of customers in their data center that don't request Internet. He shrugs. I consider asking where the hole in the wall is for the pigeons that I'm going to put in my rack since obviously I need *some* way to get packets in and out of our servers, but I bite my tongue instead.

OK, let's put pressure on our contact from above to make sure he drops everything he's doing and gets us some network. Meanwhile, let's install servers. We accompany Maintenance guy to the storage room. Maintenance guy takes first server box and throws it on the transporter from a good meter away. I was on the phone when he did this; I look puzzled at Sergio and nod my head to indicate he should say something to the guy. Sergio asks the guy not to throw our servers around; the guy gets upset.

We transport the servers a good ten meters down the hall. Then we have to take off the boxes off the transporter and to our rack, which is a lot further than ten meters, so I wonder what the transporter was useful for in the first place. From that point on, Sergio spent most of the day on the phone, trying to get people to work on the network, while I install the machines and cable the rack. I install the first machine (which will be the installation server) and leave it at that until I have net.

Our PDU units are nowhere to be found in the storage room. We know they were delivered separately from everything else because it took three additional weeks. We help them take the storage room apart (it contains boxes with delivery dates ranging back to 2004, unopened), but we still don't find them. They don't know who actually received the shipment, they have no record of where the box has gone, and they don't really know where it could be. Sergio tells them to start looking harder.

We get a cable around 14.30; it is 100 MBit, but will be upgraded soon, they assure us. Happily, I plug the cable in, and try to get somewhere. Doesn't work. We call our contact; apparently the class C we requested has been pre-reserved for us, but only 64 addresses assigned. We ask to get that changed to the full range - what does it matter if they are reserved anyway ? It makes it easier on me because I can keep the fourth number consistent with other platforms. In any case, we still can't get anywhere - the first hop after the router is on a 192.168.x.y subnet.

Our contact tells us that, while we now have a cable to the switch, our address range is not properly configured. Of course this is handled by another company, the one that takes care of Routing. Sergio loses the rest of the day calling our contact, his boss, our boss, and back and forth. He explains his contact how traceroute works and how he can use it to confirm that the network guys that claim that everything works are wrong, since he clearly cannot reach the gateway they're asking us to use.

Walking around the data center, half of the grills in the floor are bent out of shape and wobbly. At least 30% of the racks are not closed; some just don't have doors at all, others have doors that are ajar. I can easily take backup tapes from seven companies if I would want to. Sergio tells me that they managed to find the PDU units - they were in some small room that no one uses, and there was nothing in the room except this box of ours.

Somewhere during the afternoon someone drops by; apparently he's the head of projects for the data center room we're in. He asks how it's going; he lets us know that Maintenance guy complained to him that we complained about him throwing our servers around. (I know very few people that complain to their boss when they do their job so badly that the customer complains about it).

I was planning to install a first machine, check out the configuration using savon, and completely set up all the services on the first machine to then PXE-install all the others. Of course, as long as I can't get to the network, that is not going to happen. We leave the data center around 21.30, still no network. Our contact assures us that the request has been processed and will take full effect during the night.

On the way out, we get asked to hand in our papers. What papers ? Apparently we're supposed to fill in a paper that has the serial number for our laptop and stuff like that. The guy at the entrance is terribly surprised that no one asked us to fill one in in the morning - "this never happens", he assures us.

We take a cab back to Madrid, drop our stuff at the hotel, and I let Sergio take me around Madrid (we stayed at the Victoria 4 right around the corner from Puerta del Sol) for some excellent tapas: La casa del abuelo specializing in shrimp tapas (with photos of Danny DeVito visiting on the wall), Casa Lucio (which apparently has the best sunny-side-up-with-fries-and-your-choice-of-bacon/chorizo/cheese/... racion, plus pictures of Bruce Willis and Antonio Banderas visiting), and another place where we settle down with sangria, a good plate of ham, and pimientos. We promise ourselves to not go to the data center in the morning until we can confirm from the hotel room that the network is working. We pass buy this peculiarly themed tapas bar. I go in to ask what Belgian tapas are. The woman behind the counter is totally honest about it: "There are no Belgian tapas, but this is a tourist area, so .... Good for her.

We get up the next morning, try the connection to our server , still not working. We call our contact, call our boss, keep applying pressure. First excuse is "the change has been made but it takes time to propogate to the internet". Uh, we are in your data center, you are on your company's network, at least one of us two should have a correct route after 4 hours. " Their other comeback is "We're surprised that you need network to do the installation, all our customers do remote installations." Uh, sure. "Why don't you install the machines and update them over the network later ?" Not sure if it's a good idea to leave unpatched machines connected to a cable that will have Internet at some random unpredictable point in the future.

Meanwhile we work on some other things. By 12 we need to leave the hotel, so we have no choice but to go to the data center. Through security again.

My bag passes through the X-ray without further inspection. The guard asks me if I have any cameras, and I play dumb and say no. Even though I have my camera in the bag, clearly visible on the X-ray, plus two cameras on my phone and one in my N800. This time we have to fill in a paper. Sergio is asked to not only note the number for his laptop, which the guard saw on the X-ray, but also note the serial number of his mouse. The guard asks me if maybe I have a laptop as well, and if so can I fill in the paper ? I wonder how he did not see my laptop on the X-ray...
We change our plans and start installing all the machines from CD's, just to make sure that we have all the machines accessible over the net so we can do what we want with them later. I have a small IPMI problem that I brought on myself by picking the wrong subnet mask, which took some time to figure out - I can blame no one else but myself. We look for any kind of useful work we can do. By now the stickers have arrived through Fed-Ex (we had to pass on name of the driver, model of the car, and license plate to the data center security guys just to get it delivered) so we put them on a machine and on the door. We get warned that the one on the door will be taken off anyway, but whatever.


By 13.30, we suddenly have working network. Now, we had planned to take our plane back at 18.00, and had already resigned ourselves to having to come back. Suddenly the cards have changed - maybe it's better to keep at it and do everything needed to make sure we don't need to come back later ?

We call Sofie, get our ticket upgraded to business, plan to leave at 21.00, and start working like crazy to get stuff set up. I can finally play around with the configuration and store it directly in the repository, adapting as I go the previous layers from the other platform, splitting them as needed where it makes sense.

We have our "lunch" around six, where we get confronted with the vending machines from hell. Neither the food nor the drink machine accepts coins of 1 or 2 euros. We don't have enough small change to get more than one item. Someone comes in and explains to us that these machines refuse 1 euro coins if they don't have enough small change to possibly give you back money. Never mind that I am trying to get some items that cost exactly 1 euro, or that even if that wasn't the case I would be willing to not get change as long as I get some calories. The woman is nice enough though to swap her change with our coins for a small fee.

This is where we pull a nice real-life hack - we first buy some drinks, filling the machine with some small change and getting our drinks. Then it has enough small change to accept our euro coins, so we put those in, but don't buy - we push the Ben Folds Five button (aka the "give me my money back you bitch" button). Most vending machines will actually give you small change at that point. This one does too. So we then use the small change to buy food from the other machine. Sheesh.

We get back to work, move our flight one more time, to the 22.15 flight - the last one with SpanAir out of Madrid. We finish up around 21.10. Except there's no way I can close our rack. Not at the front, not at the back. Maintenance guy tells me not to worry, that he will close the racks. I take another look at the racks around me, half of them wide open, but I have no time left to argue.

We jump in the cab, get to the airport, and arrive at checkin two minutes after cut-off time. It's the last SpanAir flight out of Madrid. Bummer.

I consider taking a plane to Malaga instead - Kristien is there for the weekend because of work, the radio has taken a bunch of listeners on a trip with some of the radio hosts.  But I was warned that no partners were allowed, so I decide against it.

Sergio goes to customer service, they try for fifteen minutes to get us tickets on the Iberia flight back, but in the end they give up because their printer breaks down. Of course, we're in T4, and Iberia leaves from T3. Since there are arrows pointing to T1 and T2, this should be easy, but no - T3 is in somewhere else entirely than T1, T2 and T4 combined, and we need to take a bus there. Nobody is really sure where that bus leaves - apparently we need to go up, though there is no way to go up nearby.

After some searching and cursing we get on the bus, get into the hall, and split up - Sergio goes to buy tickets, I go to checkin. I'll save you the details on how we cut past the queue of people checking in for Santiago de Chile just to not miss our window again.

And after all of that, obviously our flight gets delayed. We arrive back in Barcelona at 0:55, I take the last bus back to the city, and join the guys from work at Belchica for a well-deserved Kriek...

I give us a 75% chance that we've done enough to not have to come back later to finish the installation...

one-click terminal window to remote hosts

Filed under: Fluendo,Hacking,sysadmin — Thomas @ 15:54


By now we have three platforms we're managing - one internally for development, and two external ones. It gets a little tedious always typing the right commands to log in to them, and it gets a little dangerous because the hostnames are named relatively consistently across the platforms, and a terminal prompt only shows the first part. So, "myimportantserver" would exist in all three platforms, and it's just no fun at all to discover that you typed "reboot" in the wrong window.

Here's how I set up a gnome-terminal launch icon on my desktop for each platform:

  • Create a new profile in gnome-terminal (Edit>Profiles, New)
  • Give it the name of your platform/group of hosts, for example my.platform
  • Edit the profile, go to the "Title and Command" tab, and select "Run a custom command instead of my shell"
  • As the custom command,enter "ssh firsthost.my.platform"
  • In the Colors tab, deactivate "Use colors from system theme"
  • Pick a background color that will remind you of this platform
  • Right-click on a panel, add a custom launcher
  • Give it a name like "Terminal my.platform"
  • As the command, use "gnome-terminal --window-with-profile=my.platform"
  • Pick an icon (Would it not be nice to have nautilus's emblems for this ?)

Now, clicking the icon should give you:

  • a terminal window
  • with a login on that machine
  • in a different color than your normal terminals, so you really know what platform you are on

Make sure you make log-ins very easy by using ssh keys, and using a desktop running under ssh-agent, so you can ssh-add your key once and then log in to these machines with a single click.  You should make it easy to jump from this entry point host to the other hosts on your platform, by using authentication forwarding for this first host.  This makes going to the next one as easy as typing "ssh nexthost"

Now, for the dessert: open a new tab in this terminal and see what happens!


Filed under: Dave/Dina,Fluendo,Hacking — Thomas @ 23:15


(For the lazy or impatient, eycandy links at the bottom)

Weeks like this are hard. For some reason, I can't get to bed when I want to and end up getting up a few times and trying to go back to sleep. For some reason my head keeps spinning and I keep thinking about stuff that needs doing. This week was supposed to be a week with a little more focus on Elisa, and while I assisted in some discussions, I have not been able to do much yet. Every day it seemed there was some problem with something someone was doing that needed some assistance, or a bunch of meetings to be in.

Days like those that sap my energy and consume my time without really allowing me to make progress on my TODO list are black karma. So I try to offset them by increasing my white karma, trying to help out someone else to do something. I was feeling guilty of never being able to sit down with Phillipe and figure out his needs for the new Elisa website, so when I finally got back to my disk after meeting X I saw the following on IRC:
<philn> MikeS: is it easy to do screencasting with flumotion? because i need to make a new one for elisa

I have been wondering the same thing and thinking it should be easy, but never got round to trying it. And lately I've been wanting to do screencasts of some of my projects, like Flumotion and moap. While I hope Istanbul will work for me sometime in the future, I am not entirely sure it is a good approach for screencasting Elisa, because Istanbul encodes to Theora on the same machine, so it would not leave much CPU to show off the fancy Elisa effects.

So I decided to give it my best shot and stay at work until I got this working.

So, the first step was simple enough. Remember how GStreamer has this crack gst-launch command that we advocate as being useful for prototyping, but that people keep treating as a real player application and spawn it from other programs ? Havoc used to poke fun at blog posts saying "See how easy it is to do stuff with GStreamer ? This pipeline walks your dog and fills up your fridge" and then some random hodgepodge of characters, slashes, numbers and exclamation marks.

Well, good news - flumotion has the same insanity ! There is flumotion-inspect to tell you what components there are, and there is flumotion-launch to launch flumotion flows for rapid prototyping. (And yes, on our side, we still really mean "rapid prototyping".)

So, first stab:

flumotion-launch -d 4 pipeline-producer pipeline="ximagesrc" ! theora-encoder ! ogg-muxer ! disk-consumer directory=`pwd`

Well, that starts up, does stuff, then quickly consumes a lot of CPU. When I stop it, I have a file in my local directory that plays back my desktop. Most players crash on it because XVideo doesn't like 1600x1050 videos. So let's limit the size a little.

flumotion-launch -d 4 pipeline-producer pipeline="ximagesrc endx=800 endy=600" ! theora-encoder ! ogg-muxer ! disk-consumer directory=`pwd`

OK, that gives me a video of the size I want, and works reasonably well. At this point we are still running on only one machine though. That means that our machine is doing the theora encoding, which is a heavy load to carry. flumotion-launch is only for prototyping, and doesn't have a way of distributing components across machines.

So, next step. I take a spare machine in the office which will do theora encoding for me. I start a manager on it, and a worker.
I connect to the spare machine from my own machine using flumotion-admin, and the wizard pops up. I go through the wizard, choosing the videotest producer for video, and the soundcard for audio. I pick Theora bitrate settings, and choose to save the stream to disk instead of streaming it. At the end, I export the configuration to disk, and I start editing it.

There are three things I need to do. For the components that I want to run on my own desktop, I set the worker name to "producer". For the others, I leave the worker name to what it was. The second thing I need to do is replace the videotest-producer section with the pipeline producer component. And the third thing I need to do is to start a worker on my local desktop with the name set to "producer", and make it connect to the remote manager.

After this, I stop the flow in the admin client, clean out all components, and load the new configuration. The flow starts up, and all components turn happy. My memory usage is increasing steadily and then dropping again. Something is probably up, and in the latest version of Flumotion I have some additional things I can look at to help me find problems. I can look at the video-producer component, which is producing a raw video feed, and see if the next component (the encoder) connecting to it over the network is losing any buffers. And yes, as expected, every so often it drops 450 buffers, which is what happens in Flumotion if a component further down the flow doesn't read fast enough.

So, how do I speed things up ? There are three things I can do. I can reduce the framerate, but I would like to show the fluid animations. I can reduce the size a little, so I change to 640x480. But a good reduction in bandwidth can be achieved by already converting the image format in the producer component from raw RGB to I420, a different colorspace that Theora will use anyway and uses less bytes per frame.

Making these changes allowed me to do a reasonably good capture without dropping too much frames. The only problem is, I'm still doing a screencast of the top left part of my desktop. I just want to screencast the Elisa application and nothing else. Here's where I cheated a little - I started offsetting the ximagesrc coordinates to make sure the capture fell inside the Elisa application window. This makes it look like it runs fullscreen.

So, reload config, restart flow, start Elisa, and do some stuff in it. This is my first screencast, and I was focusing on how to make a screencast, not what to show off in a screencast. I'm sure Philippe will do a better job tomorrow if he wants to give this a try.

So without further ado, here is a link to the Elisa screencast (using Cortado to watch it), or a link to a playlist to open in a player, or a link to the Ogg file directly.

And here is the Flumotion config file.

Bug I noticed in Flumotion when trying this: having the manager on the producer machine confused the worker on the encoder machine.

Bugs in Elisa I noticed: Elisa doesn't think my Ogg video files are in fact video, playing a folder of images when playing music makes Elisa stop accepting input events.

Todo for Flumotion: add screencasting to the scenario.

By the way, Philippe will be talking about Elisa at FOSDEM in Brussels, Belgium, this weekend, so if you are there, be sure to catch your dose of eyecandy. I will be there as well, as will Edward, Andy and Florian.

Mission accomplished, 23:07, time to go home.

« Previous PageNext Page »