Present Perfect

Spent the last few days reworking my really hacky RTSP code into fairly nicely objectified RTSP code. I can create media providers given an RTP source class. The source class can be a test stream, a source coming from a file, whatever. The media provider decides how this class will be presented to clients - "live" for a source that's always running and every client gets added to it, "on demand" for when every client has its own streamer and source, and more complex scenarios.

The media provider can then be tied to the RTSP resource tree so that various URL's can cause the provider to, you know, provide.

It's starting to work very well - after rewriting everything so that I could have a Live provider running, doing an On Demand one took me thirty minutes. That tells me my classes are extracted more or less manageably from the heap of code I had.

After that, the fun started. Trying to get various players to actually play stuff from the server. HelixPlayer/RealPlayer is proving to be the biggest pain in the ass. I've had to turn off all of Real's proprietary/extended network stuff to make it focus on doing RTSP/RTP only. Then I had to implement some non-required header fields because otherwise the player either does something silly (like, you know, no Content-base causes it to request relative URI's starting from / and not from the URI it was at), or complains that the server doesn't support the protocol.

Through the power of Open Source however I was able to unpack almost 14 MB of hxplay source code to search for the reason for this last error. Again, it was a header field that the code comments claim is required - though I haven't actually seen that in the spec yet. We'll see I guess.

At the end of the day, RealPlayer on Linux is continuously buffering/congesting on both the live and on-demand mp3 test stream. RealPlayer on Windows plays the latter fine, but not the former. Will take some time to figure out the differences.

Fedora

Got bitten by one of my favourite Red Hat/Fedora rant topics - ext2 disk labels. I absolutely hate them, and Fedora tries to use them as much as possible, often with disastrous results on machines you try to maintain for longer times.

The issue this time was as follows. I had upgraded my dad's home machine last time I was in Belgium from Red Hat 9 to Fedora Core 4. While doing so, we also gave him a new harddisk to do a fresh install, and then put in the old hard disk so that he could clean up and keep files he wanted, and delete the ones he didn't. All of this could be further done remotely.
As usual I edited grub.conf to remove the stupid device labels and put in real partitions.

Now, recently I did a remote upgrade, and after a reboot, the machine didn't come up again. Called my father. It took some time to figure out, but he was claiming it was booting in Red Hat 9 and it was throwing lots of failures. So after some thinking, what happened was this:
- the kernel upgrade rewrote grub.conf
- it used LABEL=/ to define the root partition, as opposed to /dev/sda2 which the other kernel lines specified
- it boots up the RH9 partition from the OLD drive, which also has one partition labeled /
- after some time, it remounts / as read/write. At this point, it mounts the FC4 partition from the NEW drive. Because, you know, it's also labeled /
- the RH9 kernel barfs all over the place as pieces of FC4 try to start up, and nothing works. Not even the network, since, you know, apparently Red Hat did not have the hindsight to put kernel modules for all previous Red Hat kernels in their Fedora installs (for which I can't blame them).

Now, I'm usually pretty careful with the disk labels, making sure I remove them everywhere. But I could have sworn that before, kernel upgrades looked at your grub.conf and tried to copy settings from other lines. Every other line had the partition specified with the device name. Yet *this* kernel upgrade happily ignored that. Causing me to lose a few days trying to debug this problem remotely with my father as the serial console, which as people who have fathers know, is a lot more lossy than a 300 baud modem. It's the kind of serial link that executes random commands you didn't ask for and returns characters out-of-order and skipping a whole lot of them.

Disk labels - I hates them. I absolutely hates them. I have other reasons, but this one tops the list right now. They haven't ever solved a single problem for me.

Unrelated

Robert is a new pornographer.

Spent the last few days reworking my really hacky RTSP code into fairly nicely objectified RTSP code. I can create media providers given an RTP source class. The source class...

Phew. Released new GStreamer 0.8 tarballs of core and plug-ins over the weekend. That's two. Today I released five GStreamer 0.9 tarballs. That's seven total. Release of both gst-pythons will follow soon, and possibly a Flumotion later on.

Because I'm sure we'll get a bunch of questions about the split-up of plug-ins (and just telling you what movies I watched this weekend without any further info would be rather boring) I'm copying the rationale here.

GStreamer 0.9 development series - Hung by a Thread

Starring

GSTREAMER

The core around which all other modules revolve. Base functionality and libraries, some essential elements, documentation, and testing.

BASE

A well-groomed and well-maintained collection of GStreamer plug-ins and elements, spanning the range of possible types of elements one would want to write for GStreamer.

And introducing, for the first time ever, on the development screen ...

THE GOOD

A collection of plug-ins you'd want to have right next to you on the battlefield. Shooting sharp and making no mistakes, these plug-ins have it all: good looks, good code, and good licensing. Documented and dressed up in tests. If you're looking for a role model to base your own plug-in on, here it is.

If you find a plot hole or a badly lip-synced line of code in them, let us know - it is a matter of honour for us to ensure Blondie doesn't look like he's been walking 100 miles through the desert without water.

THE UGLY

There are times when the world needs a color between black and white. Quality code to match the good's, but two-timing, backstabbing and ready to sell your freedom down the river. These plug-ins might have a patent noose
around their neck, or a lock-up license, or any other problem that makes you think twice about shipping them.

We don't call them ugly because we like them less. Does a mother love her son less because he's not as pretty as the other ones ? No - she commends him on his great personality. These plug-ins are the life of the party. And we'll still step in and set them straight if you report any unacceptable behaviour - because there are two kinds of people in the world, my friend: those with a rope around their neck and the people who do the cutting.

THE BAD

No perfectly groomed moustache or any amount of fine clothing is going to cover up the truth - these plug-ins are Bad with a capital B. They look fine on the outside, and might even appear to get the job done, but at the end of the day they're a black sheep. Without a golden-haired angel to watch over them, they'll probably land in an unmarked grave at the final showdown.

Don't bug us about their quality - exercise your Free Software rights, patch up the offender and send us the patch on the fastest steed you can steal from the Confederates. Because you see, in this world, there's two kinds of people, my friend: those with loaded guns and those who dig. You dig.

The Lowdown

GStreamer Plugins has grown so big that it's hard to separate the wheat from the chaff. Also, distributors have brought up issues about the legal status of some of the plug-ins we ship. To remedy this, we've divided the previous set of available plug-ins into four modules:

gst-plugins-base

a small and fixed set of plug-ins, covering a wide range of possible types of elements; these are continuously kept up-to-date with any core changes during the development series.

• We believe distributors can safely ship these plug-ins
• People writing elements should base their code on these elements
• These elements come with examples, documentation, and regression tests

gst-plugins-good

a set of plug-ins that we consider to have good quality code, correct functionality, our preferred license (LGPL for the plug-in code, LGPL or LGPL-compatible for the supporting library).

• We believe distributors can safely ship these plug-ins
• People writing elements should base their code on these elements

gst-plugins-ugly

a set of plug-ins that have good quality and correct functionality, but distributing them might pose problems. The license on either the plug-ins or the supporting libraries might not be how we'd like. The code might be widely known to present patent problems.

• Distributors should check if they want/can ship these plug-ins
• People writing elements should base their code on these elements

gst-plugins-bad

a set of plug-ins that aren't up to par compared to the rest. They might be close to being good quality, but they're missing something - be it a good code review, some documentation, a set of tests, a real live maintainer, or some actual wide use.
If the blanks are filled in they might be upgraded to become part of either gst-plugins-good or gst-plugins-ugly, depending on the other factors.

• If the plug-ins break, you can't complain - instead, you can fix the problem and send us a patch, or bribe someone into fixing them for you
• New contributors can start here for things to work on

Phew. Released new GStreamer 0.8 tarballs of core and plug-ins over the weekend. That's two. Today I released five GStreamer 0.9 tarballs. That's seven total. Release of both gst-pythons will...

I've been mentally building up a backlog of stuff to write about and the moment where I sit down and just lay it all out isn't happening for some reason. So I've decided to do some reverse engineering and work my way backwards instead. Starting with tonight.

Went to see The Island with part of the gang and our honorary employee of the week, the Welsh Frenchie. It was actually quite enjoyable, and while I'm sure lots of people will slag it for the obvious plot holes and the overblown action ... Well, I've never understood why some people can't judge a film in its own category, anyway. And hey - what a happy ending the movie leaves us with - only 5m$ between you and a clone of Scarlett Johanson ! Finally something worth saving for.

Spent way too big a chunk of my weekend covering up for someone who shall rename nameless who completely dropped the ball on his responsibilities after managing to not prepare at all for the commitment he made; he made up for it by going to the beach instead of actually trying to show a morsel of remorse and a whiff of something that would vaguely smell of duty on a good day.

The only redeeming feature of my weekend was the fact that I had visitors over again - Els and Wiebe passed by on their way back from Malaga. We went on a wild tapas hunt looking for Can Tomas, which is rumoured to have the best bravas in town. But two taxis and a closed door later, we let the cab driver decide where to take us. He took us to La Cerveceria Catalana, which he said was one of the best places in town. Entering was like coming home after a long trip - a feeling of having been there before but stuff is out of place. The menu had some stuff that I knew, the waiters looked sort-of-familiarly-dressed, and the decoration reminded me of something. After a few moments it hit me that this place looked an awful lot like our favourite tapas place, Ciudad Condal. It was spooky. The downside was that I didn't find out about a great new tapas place. The upside is that I had great food and we really do know one of the best tapas places around.

Jukebox

Boy, does it take long to close circles. Some five years ago I did a thesis project in my last year at university. The goal of it was to do real-time mixing of MP3 files on a Pentium 200MMX. I don't know if people remember the good old days, but a machine like that had a hard time already just decoding an MP3. I found a really nice way of doing mixing in the subband sample domain, bypassing the most expensive steps of the decode and encode processes, and was able to mix up to four tracks simultaneously in real-time, outputting to MPEG Layer 2. The jukebox driven by that ran for a very long time before it finally got retired.

Now, this was the only good thing about the program I wrote. The bad thing was that it was a gross hack evolved from the reference encoder/decoder code; that it only worked for MP3 files; that all files were required to be of the same sample rate (otherwise the subband sample mixing is wrong); and that as a project it wasn't very maintainable at all. Now this last point was back then still a bit unknown to me - I was just getting started in Free Software stuff. But regardless - I started looking for stuff that would allow me to do a nicer jukebox program for the radio station, and while googling (or was it still called altavistaing back then ?) I learnt about GStreamer. That was around 0.1.1 I think.

I read the docs, was impressed with the ideas (little did I know back then :)), started poking at it, found some things that didn't work, got involved more and more, all with the idea of doing my mixer as quickly as possible. But pretty soon all things I wanted to do with something like GStreamer were moved to the back of my mind and GStreamer became a goal on its own. Sure, I wrote a bunch of programs with it, and once in a while I took a small stab at some ideas for the mixer, but I never pushed through.

Until a few weeks ago, where we needed a decent radio demo to be online 24/7 and I just didn't want to put in another hack. So I wrote some stuff in Python, found some bugs in the current versions of some of the first GStreamer elements I wrote (why did people let me commit DSP stuff in the first place ?), and now I have something that does a remarkable job at mixing music of *any* type GStreamer can handle, automatically, picking decent mix points and a per-track volume level that makes the average loudness over songs constant.

It's definitely not perfect yet (Bohemian Rhapsody's dynamics trip it up for example), but it's a good first stab that I'll work on refining over the next few months. If you want to give it a listen and tell me what you think, the URL is http://stream.fluendo.com:8831 for a HQ Vorbis stream. (There's an mp3 stream at 8833 and a LQ Vorbis stream at 8832)
You can also run it for yourself; get GStreamer core, plugins and Python bindings for the 0.8 branch from CVS, go into gst-python/gst/extend and run

python jukebox.py playlist.m3u queue max-size-bytes=0 max-size-time=3000000000 ! { osssink }

. The program will eat a *lot* of CPU while it's scanning all the files from your playlist for good mixing points, but once that's done the values are cached.

I'm making a much bigger deal out of having finally done this than it is, but it is weird how these things take on a life of their own and run away with you. It was nice to finally just get something done that I was planning to do when learning about GStreamer.

Memory hunting

The only problem I had was that the jukebox component I wrote for Flumotion was eating too much memory. After a week, it was consuming 50% of the 512MB memory on the machine it was running on. I spent some free time looking into what was going on, fixing various memleaks in the elements (I also finally learnt how to properly valgrind python applications), learning some more about pygtk's refcounting and cycle problems, and doing some more explicit cleanup in my python code. I got to the point where I was sure my program wasn't leaking in the sense that it was dropping references to blocks of memory.

But it was still leaking memory in the sense that memory use was slowly increasing over time. The problem is that that sort of thing is pretty hard to work on in Python. I started adding some debugging that made the program garbage-collect at specific points in the program so I could bring some determinism to the freeing patterns and cross-reference with the GStreamer object refcounting debugging. That allowed me to fix a bunch of elements that were kept around, but not used anymore.

To get a better sense of how bad the problem was, I wanted to run the program outputting to a fakesink, basically making it mix as fast as it could. The only problem is that my home desktop heats up fairly quickly, and starts flooding the console with CPU thermal info around 70C, making the system unusable. This happens after about five minutes of 100% CPU use.

So, because I really wanted to run these tests over the weekend, I finally dove into the wonderful world of case modding. I took Wiebe to some of the computer stores and we bought a whole set of fans - plus a KVM switch that I wanted for some time so I could stop diving under the table whenever the server machine had a problem. I took the opportunity to completely rework the engine room layout, separate the two machines (they were standing right next to each other), install the fans, work out airflows, and re-paste the CPU's. And now the machine is running at 40C idle, and no more than 54C at 100% CPU. With that, I could run the jukebox at 100% CPU, and at this point it's mixed about 12000 songs - which is over a month of output. The Data size is around 44 MB, or a good 10% of my memory. So it's still increasing slowly, taking about 3.7K per song.

Not bad, but could be better. At this point I'm stuck. If anyone has ideas on how I could further figure out what could be causing this, let me know. I've tried massif, but both in 2.4.0 and 3.0.0 it failed on the first song. I've tried memprof, but I can't even get that to run anymore. My next hack will be something that LD_PRELOAD's to track allocation and freeing, and has a python hook so I can ask for the allocated memory at specific points (for example, each time a song has played completely) that are comparable to each other, to figure out what I'm not getting rid of each cycle. Not sure if it's possible to do a python binding for a lib you LD_preload, but it's worth a shot...

Anyway, upwards and onwards. I can't imagine why I've went in such detail on so little given that I was planning to flush out a whole backlog, but folks back home were getting worried about what I was up to...

I've been mentally building up a backlog of stuff to write about and the moment where I sit down and just lay it all out isn't happening for some reason....

Spent way too long not writing anything, should catch up on that soon.

But meanwhile, a quick shot of our office stream on a phone

RTSP/RTP H263 streaming using GStreamer and Python.

Spent way too long not writing anything, should catch up on that soon. But meanwhile, a quick shot of our office stream on a phone RTSP/RTP H263 streaming using GStreamer...

Pfff. A whole day of fighting against a mesmerizing bug in either of python, gnonlin, gst-python, or pygtk. A simple program that did nothing more than

import gst
gst.element_factory_make("gnltimeline")

was throwing an assertion in python's garbage collector. I had this problem on FC2, FC3, and FC4. Edward didn't have this particular problem on his Gentoo machine, and Jan wasn't having it on his Breezy install (which has pygtk 2.7.something, by the way), even though his breezy was broken in other places.

Of course, I didn't have a need for this particularly short program, but for some reason gnltimeline triggers creation of some object that gets garbage-collected in my application where I do want to use it, even if I keep the element around. In Edward's examples, it only asserts when leaving the application; but when I use it, it asserts as soon as Python garbage-collects after creating.

We tried a lot of different approaches, and in the end I just gave up. Days like these are lost forever. It probably didn't help that I was still a bit tired after last night's Xiph meeting at 2 AM, but still - compared to last week's excellent flow in dire circumstances (more later) this day was an absolute low point.

Pfff. A whole day of fighting against a mesmerizing bug in either of python, gnonlin, gst-python, or pygtk. A simple program that did nothing more than import gst gst.element_factory_make("gnltimeline") was...