Present Perfect

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps them into client groups, then makes routing decisions by group to any of our sites based on a policy (least cost, geographically closest, ...) In addition, I wrote a simulator that replays actual request log lines, and is able to then track the bandwidth used by each client group and each site, and even able to dump it to an .rrd file for later graphing. Not bad at all for a quick hack. Python is awesome when it needs to be.

The bad: some binary tree algorithm I use to be able to get a sorted list of simulated events seems to crap out currently at about 1000 request lines with a 'Maximum recursion depth exceeded'. Last time I asked our database guy, I think we had about 10 million lines a day.

I don't think my simulator is ready for prime-time just yet...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps...

So, it's now official: Flumotion has won the 'Best Streaming Innovation' award at last week's Streaming Media Europe conference in London. Thank you for all who voted! I'm surprised to see so many people in total vote, that means that there are quite a few real users out there.

I wasn't there myself to accept the award (our commercial team accepted what basically is a technology award, but we get along well enough :)), but I did go on Friday to replace Julien on a panel about the promise and disruoption about Open Source Video (scroll to the bottom).

This was actually my first panel, but I think we did well, especially considering it was the last panel of the day. There certainly were many more questions from the audience then in the transcoding panel I attended earlier that day. The panel was led by Dr. Tsur from Kaltura, an open source video platform.

After the conference I took the underground with Damien from New Bamboo who was also on the panel. They developed Panda, a cloud transcoding solution with the very catchy tagline of 'video solved'. It uses ffmpeg at its core, which might get them in trouble now that they want people to run it as a cloud service, so I suggested he take a look at GStreamer instead. My first suggestion was that it should be doable to provide a drop-in binary replacement with the same interface, but having looked at the output of ffmpeg -h this weekend, I'm not sure anymore - that binary was already complicated to use way back when and it has only exploded further.

In any case, since it's an open tool, maybe someone at work should take a look and see how easy it would be to teach it GStreamer. Or, maybe instead, we should open our own transcoder solution.

Anyway, back to London and the conference. Since the conference was over after the panel and people didn't hing around, I called Lotte, a friend living there, to go have a drink before taking the Eurostar back. And thus it happened. I felt like a real global cosmopolite, having a drink in London after a conference for my Spanish job coming in from Belgium wearing a Spanish jacket, American shoes bought in London, a Florentine shirt, and New York pants.

So, it's now official: Flumotion has won the 'Best Streaming Innovation' award at last week's Streaming Media Europe conference in London. Thank you for all who voted! I'm surprised to...

The last few months news about streaming to iPhone 3.0 has been making the rounds. I've been holding off commenting on it for a while since I didn't actually look into it much and didn't want to base anything on hearsay. And I don't even have - or want - an iPhone!

Last week I took some time to read the IETF draft and the Apple developer introduction.

On my next plane ride I quickly hacked together a simple segmenter in Python, and tried it the next day at work to see that it sort-of-worked for about a minute.

And yesterday evening, during Nerd Night, I changed my original plans (since Wiebe cancelled, I wasn't going to work on the Spykee robot yet) and decided to go back to the iPhone streaming hacking.

After tweaking mpegtsmux to do something useful with GStreamer's GST_BUFFER_FLAG_DELTA_UNIT and teaching the segmenter to always start a new segment on a non-delta-unit, and after switching to a black videotestrc with a timeoverlay (the normal one seems to trigger a weird encoder bug in our H264 encoder, need some help from our Fluendo codec gurus for that), I started a simple stream last night:
71118

I left it running for the night.

And this morning when I got up, it was still going strong, and I left it pass the 10 hour mark:

71122

So, a good first step.

Hope to finish up some loose ends across the week to make this work inside Flumotion.

I'll leave you with my first impressions on this Apple creation:

• Naming a draft 'HTTP Live Streaming' pretending this is something new after years of Shoutcast - Icecast - Flumotion is either plain ignorance or typical Apple hubris. At least qualify the name with something like 'segmented', 'TS', or 'high-latency', Apple. Come on, play nice for once.
• The streaming system is very different from your typical streaming system. Effectively, this approach creates a live stream by segmenting a live feed into a sequence of MPEG Transport Stream segments at a regular interval. This has some benefits and drawbacks.
• The key concept is now the playlist file, an extension of .m3u called .m3u8. This playlist file is the entry point into the stream, as it lists the segments that make up the stream.
• This playlist file can reference other playlist files. This is what enables adaptive bandwidth streaming.
• One clear benefit that Apple was aiming for is that they effectively managed to separate the preparation part from the streaming part - the actual streaming can be handled by any old web server that can serve up files. I'm sure this is the main benefit they had in mind. The benefit is two-fold: first of all, it's easy and cheap to install web servers, and second, you get all the benefits of using a bog-standard protocol like HTTP: firewall acceptance, proxy and caching support, edge caching, ... Take for example the fact that a company like Akamai charges more for some streaming protocols because they have to deploy specific servers and can't use all their edge infrastructure for it.
• Another benefit is that you are generating the data for your live and ondemand streaming at the same time. The transport segments can be reused as is for ondemand .m3u8 streams. This blending of live and ondemand is something we started thinking about with the developers at Flumotion too.
• A third benefit is how easy this system would make it to do load balancing on a platform. In most streaming services, a connection is long-lived, and hard to migrate between servers. Since in Apple's live HTTP streaming the stream consists of several short files, you can switch servers by updating the playlists, effectively migrating the streaming sessions to another machine within a minute.
• As for drawbacks, the biggest drawback I see is the latency. In this system, the latency is at least the segmentation interval times three. This is because the playlist should only contain finished segments, and the spec mandates that the player have at least three segments loaded (one playing, two preloaded) to work. So, the recommended interval of 10 seconds gives you at best a 30 second latency. I don't really understand why they didn't work around this limitation somehow (for example, by allowing a growing transport stream in the playlist, marked as such, or referencing future files, marked as such), because this is where live iPhone streaming is going to catch the biggest amount of flak, if our customers' opinion about latency in general is anything to go by.
• Another possible drawback is the typical problem with most HTTP streaming systems - no synchronization of server and client clocks. Computer clocks typically don't match in speed, so in practice this usually means that the client's buffer will eventually underrun (causing pauses) or overrun (usually causing players to stop). In practice this is not that big of a deal, and I doubt on the iPhone sessions will be long enough to really make this a problem.

Whether this will become a general-purpose streaming protocol remains to be seen. I would assume that Apple is at least going to make this work in a future update of OSX. For us though it is an exciting development, allowing us to showcase the flexibility of our design to this new protocol. And while I saw some fellow GStreamer developers griping about this new way of streaming, there as well it should be seen as an advantage, since (in theory at least) the flexible GStreamer design should make it possible to write a source element for this protocol that abstracts the streaming implementation and just feeds the re-assembled transport stream much like a dvb or firewire element would do.

The last few months news about streaming to iPhone 3.0 has been making the rounds. I've been holding off commenting on it for a while since I didn't actually look...

Last week Jan Urbanski, one of our developers, released a new stable series of Flumotion. This release comes two years and a few months after the previous stable series' last release, so you can imagine a lot of changes have gone into this new series! Basically, this release contains all the changes that two years of platform customers and a year of box/product testers have made us make.

This is Jan's second release. We've had some discussions before the release because we didn't necessarily agree on the strategies for releasing and branching, but in the end I managed to get mostly out of the way and Jan's strategy survived more or less.

With Jan's release strategy some things are now different:

• Jan does not want to add new features to this stable series; you will get only bug fixes
• To balance that, Jan's made a solemn oath to do a new stable series within the next nine months, instead of the two years we managed this time.

My suspicion is that Jan wants to increase the frequency of stable series because of our Flumotion release tradition: new major/minor releases are named after a restaurant, and we go celebrate the release at that restaurant! Maybe Jan is just a hungry guy. (On the other hand, maybe he wasn't thinking about restaurants after all - he had actually forgotten to plan for a release name until the last minute...)

This will be an interesting release dinner; I think last time we were maybe 8 people, now we'll be probably double...

From my point of view this was the smoothest stable release ever, so smooth that I woke up one day and checked my mail and bang, the release was there!

Listing all new features would be a bit excessive, but let me cherrypick a few highlights (or, failing that, less important bits that I just happened to have a hand in myself):

• A 200 page (at least last time I printed it) manual, mostly (re)written by Murray Cumming of Openismus. Murray did an excellent job of entering our little microverse, biting his tongue on the idiosyncracies in our concepts and distilling it all down to a useful manual that has found its way, ring-bounded and plastified, onto desks of our technical people without any prodding on my part! Ok, so what if half of it is auto-generated ? We still had to write the text that goes into those pages too!
• A bunch of new producer components, like screencast, ivtv and rtsp, and a favorite called playlist-producer, which uses GNonLin to create a fake live stream out of an .XML file of ondemand files. See, using GStreamer was a good idea after all!
• A consumer that lets you stream to an Icecast2 server. Yes, we're friendly to our open source competitors! Contributed by - please come back! - Gergely Nagy.
• a flumotion-nagios command that makes it easy to integrate Flumotion checks into Nagios and similar systems (Guess which monitoring system we use on our platform)
• A switcher component, switching between different inputs, based on a manual trigger, the master going away (watchdog scenarios), or based on an iCalendar file; mostly massaged into shape by the firm but soft-spoken Zaheer Merali
• aliased feeds, allowing multiple components to provide the same feed to downstream components, so that if one of the feeders fails the other one takes over. If memory serves me right this was done by the unshaveable Andrew Patrick Wingo back in the day!
• Support for Dirac, the BBC's open HD codec! Sadly it seems while this feature was added during 0.5 (by yours truly), it broke somewhere between GStreamer and Flumotion along the way. Go fix it team, so we can once again be a multi-open-format streaming server!
• Support for Python 2.6!
• A rewritten wizard/assistant, started by Johan Dahlin (the only person to have been hired by the Fluendo Group twice), and handed off to the unstoppable Xavier Queralt when Johan left to join LITL. As part of our box project, they have been completely reworking the wizard, and adding spiffy features like easy ondemand setup and addition of encoding formats to existing flows.
• Rate control for ondemand serving, added by Michael Smith (possibly the only person who's maintained two open source streaming servers at the same time) before he left for Songbird
• More under the hood tools and debug options, process management, and daft things like 'ssh into a python shell on the manager', all safely tucked away behind a debug toggle
• Tons and tons of fixes, optimizations, memory reductions, tweaks, by the whole team

Some thanks are in order too to Andrew Wingo and Arek Korbik for plodding ahead with the decidedly unsexy work of doing unstable releases over the last two years. Jan's been standing on the shoulder of giants.

Last week Jan Urbanski, one of our developers, released a new stable series of Flumotion. This release comes two years and a few months after the previous stable series' last...

8 years ago I had a lot of fun at, I think, Hacking in Progress. This year it's
Hacking At Random, again in Holland. I was actually planning to go there (for one thing, I really want to learn how to pick locks), but with all my travel plans for this summer already it was just getting too much. Since this is a 4-year event, I'll probably regret not going two years from now...

I still remember the first time I was lying in my sleeping bag in a tent, with an ethernet cable running to it, and being in the bag and online with the laptop.

Anyway, it's a small consolation that there are
live streams up and running from over there using Flumotion, set up by Raphaël.

8 years ago I had a lot of fun at, I think, Hacking in Progress. This year it's Hacking At Random, again in Holland. I was actually planning to go...