Present Perfect

Didn't post much of late - various factors conspiring against me, the biggest one being me.
Being poked repeatedly to do something about it, so here goes. A quick rundown of the essentials, possibly followed by separate posts.

Life

Stuff has happened. I became a godfather. His name is Arthur, a solid Flemish name. I'm going over next week to see him for the first time. No pictures online yet, the parents are old-school :)

Congratulations to Jeroen and Annelies - they had their second son, Finn. The pictures look good, but I'm not sharing on account of the father's paranoia towards bad people.

Sad when friends break up - my thoughts are with you, you know who you are.

Lunya had her ovaries removed. She was very well-behaved when she had the cone on her head - maybe we should keep it around for when she's too rough.

Music
After my lamenting my own ignorance of fresh music, I found a bunch of new bands I like. I'm not yet sure if I should be sad that I'm getting most of my musical recommendations from this brat, but what can I say, the lad knows what's good.

Among other bands, I've picked up on Art Brut, the Departure, the Editors, Go Team. I've enjoyed new albums by Greg Dulli, Elbow (simply astounding), Sigur Ros, Nada Surf (to a lesser extent), dEUS (Belgium's back).

In the mean time, one of my favourites, Six By Seven, has split up. Bastards.

Lots of concerts to look forward to over the next few weeks - Bloc Party, dEUS, Coldplay (shut up if you have an opinion), Art Brut, Nada Surf, Editors, with possible seconds of White Stripes, Rufus Wainwright, Daniel and the Johnsons, Backstreet Boys (?!), Black Eyed Peas (why not), and a whole bunch of others I'm forgetting.
It got to the point where I had to make a list for myself and I freaked out my co-workers with it.

Work

Good stuff is happening all around me at work. But meanwhile I was getting the feeling of just not being able to complete what I was working on. Andy's image of the "big ball of mud" is what you need to keep in mind here. You climb the ball you had left yesterday and something underneath slips out from you that was broken by some other change. You try to add something to the ball and you find that there's nothing supporting it and you need to dig all the way down to add it and then go back up. It brings you to the point where you cannot fairly estimate how long finishing your thing is going to take.

I had several days where, to write the next two lines of Python code, I had to spend a few hours debugging a problem, some more hours adding/fixing stuff in C, then checking and writing the two lines of Python - only to run into the next problem.

While personally I do not mind this approach - it ensures that the layers you're using are getting proper testing and fixing and are only improving in quality - it is very painful if you want to complete something. Every day you take half an hour to update all of the underlying code, check some changes, run the testsuite (thank God for me seeing the light on testsuites the past year), figure out if anything broke, and try to add to the project.

But that's not even the worst part. The worst part is having to test this stuff with an actual phone that

• doesn't have any logging
• has a crappy user interface
• gives one out of three possible error messages - so it's useless for feedback
• takes a long time to do anything at all
• fails to get a stream even when the stream is fine

Seriously, no debugging, no logging, no way to tell what's going on, not even being able to be sure that if the phone doesn't show anything, I broke something ...

I'll be glad when we have a first working version.

It got to the point where I was starting to doubt myself and felt like I couldn't achieve anything useful anymore, with everyone around me hacking up excellent stuff. So I took a weekend for myself (sorry Kristien) to hack on something of my own choosing to see if I could still "do it". The thing in question was something I want to work on to test synchronization over the network. I wanted to combine GStreamer and Twisted to expose GstPipelines over the network to one application. As a first run, I wanted to get the excellent seek example running over the network, so I could control the application running on my desktop from the laptop.

With some fixes to the python code for the seek example, some deep digging into Twisted's serialization layer to be able to serialize objects with both methods and state, and a bunch of hacking and abstracting, I had something that nicely seeked across the network. Phew - so it wasn't just me.

As a second challenge for myself, I've started looking at GStreamer on Windows. Sebastien (Julien's brother) had been working on it for a while and had passed by the office to help me set up a buildslave and explain how stuff works. It was actually challenging and, in a weird way, fun to get GStreamer to build on Windows. I'm lucky I had help from Sebastien, and we could use the work of some very smart people before us - Steve Lhomme, Tor Lillqvist, the MingW people, ...

The reason I want to do this is to put up a buildbot so we figure out platform issues immediately. In the past, we've had some people throw some patches or build files at us for Windows, and then they would just bitrot until the next person came along. They also always had strange requests or problems that seemed to be hacked around in the end- I wanted to see if I could go over all of that and fix stuff properly. Also - there are various ways of building stuff on Windows, and everyone had their own preference, and nobody knew what the "best" way was. You can build stuff with MinGW, CygWin, VS6, VS7, the last two of which with or without solution files, or with Makefiles, ...

I've also set up a clean VMWare install at home to try out stuff, taking notes as I go along (which I'll probably post once I've gone through it all). At the moment, I have a VS6 build working from fresh CVS, and my MinGW build is close to working. There's lots of silly little things that caused GStreamer to not work (the most obvious one being that any printf-style function with a %s crashes when passed a NULL string on Windows) or have issues, and it's fun to apply the stuff I've learned to a new platform.

The other reason it was fun is because it proved a nice counterbalance to my daily anal-retentive self when it comes to software development. Windows is like the wild wild west. The rules are completely different for developing. There is no package management system. You add crap all over the place until you have a huge ball of stuff with bits poking out and you can't be sure what goes where. But there's no other way - that's how it works. It feels like being an LFS user, and it's ok because that's the only way to make it work ! You install a version of libtool in one environment, but it has some bugs, so you add some other version for some other environment to your path, and see what happens... Basically, you just throw random stuff at the problem, and poke at things with a sharp stick, until it works.

Of course, after two days of anarchy, I made sure I cleanly separated several different tools on my clean home install, started writing batch files instead of tweaking environment variables, and so on. But the mess we made on the work machine was fun to make.

So, with these two challenges under my belt, I can safely say that it's not just me. My actual work project is *hard*. I'll just have to poke at it harder, and refrain from projecting a finish line.

Didn't post much of late - various factors conspiring against me, the biggest one being me. Being poked repeatedly to do something about it, so here goes. A quick rundown...

Went out to one of my favourite restaurants last friday with Kristien and Nat: Can Margarit. It's a restaurant where the menu hasn't changed for the last 25 years, and there's only a small menu. But the rabbit is *ek-sel-lent* there, completely different from how mom makes it - lots of garlic, rosemary, thyme, and salt.

After that, we went to a bar I had never been before which I thought was great - nice music, not too smoky, atmospheric, decent cocktails though a little expensive.

All in all, an uneventful evening, except for one little detail. Right before getting the rabbit, I got a little fishbone stuck in my throat. I got a little teary-eyed and went to the bathroom to try and get it out. That didn't work very well, although I felt it move a little until it got lodged into the back of my throat.

We got home around 2 and I was starting to get a little worried. Stuck my fingers down my throat to get it out, I managed to throw up just fine but no fishbone came out. Tried to look at the back of my mouth in the mirror, didn't work, too dark. Took a smaller mirror and stood opposite of a light, trying to find the correct angle at which I could illuminate the back of my throat - pretty tricky, but finally worked. There was a little blood down the back and something white sticking out of the flesh. Tried to reach it with my finger but couldn't reach it reliably because it triggered my gag reflex. Took a pair of tweezers and tried to maneuver it but it was too hard trying to hold two things at once and look in the mirror.

Dragged Kristien in the bathroom, who freaked out, but managed to convince her that she really needed to try and get that white thing out of my throat. We tried in various positions. She gave me a knife to push down my tongue because my reflexes were curling up my tongue each time she got to the back of my mouth with the tweezers. After that, the tweezers weren't long enough, but luckily we had longer in the first aid kit. Kristien tried again, and this time it felt like she dislodged something, though it was hard to tell. Looking at it again, there was a little more blood, and the white bit sticking out was hard to find, so we assumed it was gone. It still felt like something was stuck in my throat further down, though. At this point I was getting really desperate.

Google to the rescue ! Amazing how many dimwits like me write about their incredible adventures with the stuck fishbone. While taking medical advice from the internet might not be a good idea, it was 2.30 AM and I really wanted to try some new ideas and was hoping they'd make sense. The most common advice seemed to be to eat rice without chewing, so Kristien put on some rice. But my biggest worry was "how important is it to get this bit of fish out of my throat right now ?" I mean, would it get horribly infected if I left it where it was, or could I die in my sleep while it perforated my lung, killing me softly with each breath ?

So I tried to call my dad (who's a doctor), but he was probably asleep. I called my sister (who's studying to become a doctor) and she started laughing when I explained the situation. I told her about the throwing up - an "uh oh" escaped her lips. I told her about the tweezers - "are you crazy ???" was her reply. So if any of you ever have this problem, don't try this at home.

She agreed with the rice strategy, though. She also put my mind to rest saying that the worst that could happen would already have happened by now - that would be the case where the fishbone manages to get in your windpipe and wind up in your lungs.

So, there I was, 3.00 AM, swallowing gobs of rice without chewing, trying to relax and not think about imminent death. After I had all the rice I could eat, I still felt like something was there, but it might as well just be the wound I was feeling. I decided I had done all I could do, and after some reading went back to bed. And I'm still alive today.

It was an interesting experiment, but even on this very Talk Like A Pirate Day, I don't think I will be eating fish again soon.

In related news, Lunya was taken to the vet to be itted/spayed/deovarized. She now sulks around the house with a silly cone round her head and a stupid fish net over her body. We hope we're getting the feisty Lunya back soon.

Went out to one of my favourite restaurants last friday with Kristien and Nat: Can Margarit. It's a restaurant where the menu hasn't changed for the last 25 years, and...

Spent the last few days reworking my really hacky RTSP code into fairly nicely objectified RTSP code. I can create media providers given an RTP source class. The source class can be a test stream, a source coming from a file, whatever. The media provider decides how this class will be presented to clients - "live" for a source that's always running and every client gets added to it, "on demand" for when every client has its own streamer and source, and more complex scenarios.

The media provider can then be tied to the RTSP resource tree so that various URL's can cause the provider to, you know, provide.

It's starting to work very well - after rewriting everything so that I could have a Live provider running, doing an On Demand one took me thirty minutes. That tells me my classes are extracted more or less manageably from the heap of code I had.

After that, the fun started. Trying to get various players to actually play stuff from the server. HelixPlayer/RealPlayer is proving to be the biggest pain in the ass. I've had to turn off all of Real's proprietary/extended network stuff to make it focus on doing RTSP/RTP only. Then I had to implement some non-required header fields because otherwise the player either does something silly (like, you know, no Content-base causes it to request relative URI's starting from / and not from the URI it was at), or complains that the server doesn't support the protocol.

Through the power of Open Source however I was able to unpack almost 14 MB of hxplay source code to search for the reason for this last error. Again, it was a header field that the code comments claim is required - though I haven't actually seen that in the spec yet. We'll see I guess.

At the end of the day, RealPlayer on Linux is continuously buffering/congesting on both the live and on-demand mp3 test stream. RealPlayer on Windows plays the latter fine, but not the former. Will take some time to figure out the differences.

Fedora

Got bitten by one of my favourite Red Hat/Fedora rant topics - ext2 disk labels. I absolutely hate them, and Fedora tries to use them as much as possible, often with disastrous results on machines you try to maintain for longer times.

The issue this time was as follows. I had upgraded my dad's home machine last time I was in Belgium from Red Hat 9 to Fedora Core 4. While doing so, we also gave him a new harddisk to do a fresh install, and then put in the old hard disk so that he could clean up and keep files he wanted, and delete the ones he didn't. All of this could be further done remotely.
As usual I edited grub.conf to remove the stupid device labels and put in real partitions.

Now, recently I did a remote upgrade, and after a reboot, the machine didn't come up again. Called my father. It took some time to figure out, but he was claiming it was booting in Red Hat 9 and it was throwing lots of failures. So after some thinking, what happened was this:
- the kernel upgrade rewrote grub.conf
- it used LABEL=/ to define the root partition, as opposed to /dev/sda2 which the other kernel lines specified
- it boots up the RH9 partition from the OLD drive, which also has one partition labeled /
- after some time, it remounts / as read/write. At this point, it mounts the FC4 partition from the NEW drive. Because, you know, it's also labeled /
- the RH9 kernel barfs all over the place as pieces of FC4 try to start up, and nothing works. Not even the network, since, you know, apparently Red Hat did not have the hindsight to put kernel modules for all previous Red Hat kernels in their Fedora installs (for which I can't blame them).

Now, I'm usually pretty careful with the disk labels, making sure I remove them everywhere. But I could have sworn that before, kernel upgrades looked at your grub.conf and tried to copy settings from other lines. Every other line had the partition specified with the device name. Yet *this* kernel upgrade happily ignored that. Causing me to lose a few days trying to debug this problem remotely with my father as the serial console, which as people who have fathers know, is a lot more lossy than a 300 baud modem. It's the kind of serial link that executes random commands you didn't ask for and returns characters out-of-order and skipping a whole lot of them.

Disk labels - I hates them. I absolutely hates them. I have other reasons, but this one tops the list right now. They haven't ever solved a single problem for me.

Unrelated

Robert is a new pornographer.

Spent the last few days reworking my really hacky RTSP code into fairly nicely objectified RTSP code. I can create media providers given an RTP source class. The source class...

Phew. Released new GStreamer 0.8 tarballs of core and plug-ins over the weekend. That's two. Today I released five GStreamer 0.9 tarballs. That's seven total. Release of both gst-pythons will follow soon, and possibly a Flumotion later on.

Because I'm sure we'll get a bunch of questions about the split-up of plug-ins (and just telling you what movies I watched this weekend without any further info would be rather boring) I'm copying the rationale here.

GStreamer 0.9 development series - Hung by a Thread

Starring

GSTREAMER

The core around which all other modules revolve. Base functionality and libraries, some essential elements, documentation, and testing.

BASE

A well-groomed and well-maintained collection of GStreamer plug-ins and elements, spanning the range of possible types of elements one would want to write for GStreamer.

And introducing, for the first time ever, on the development screen ...

THE GOOD

A collection of plug-ins you'd want to have right next to you on the battlefield. Shooting sharp and making no mistakes, these plug-ins have it all: good looks, good code, and good licensing. Documented and dressed up in tests. If you're looking for a role model to base your own plug-in on, here it is.

If you find a plot hole or a badly lip-synced line of code in them, let us know - it is a matter of honour for us to ensure Blondie doesn't look like he's been walking 100 miles through the desert without water.

THE UGLY

There are times when the world needs a color between black and white. Quality code to match the good's, but two-timing, backstabbing and ready to sell your freedom down the river. These plug-ins might have a patent noose
around their neck, or a lock-up license, or any other problem that makes you think twice about shipping them.

We don't call them ugly because we like them less. Does a mother love her son less because he's not as pretty as the other ones ? No - she commends him on his great personality. These plug-ins are the life of the party. And we'll still step in and set them straight if you report any unacceptable behaviour - because there are two kinds of people in the world, my friend: those with a rope around their neck and the people who do the cutting.

THE BAD

No perfectly groomed moustache or any amount of fine clothing is going to cover up the truth - these plug-ins are Bad with a capital B. They look fine on the outside, and might even appear to get the job done, but at the end of the day they're a black sheep. Without a golden-haired angel to watch over them, they'll probably land in an unmarked grave at the final showdown.

Don't bug us about their quality - exercise your Free Software rights, patch up the offender and send us the patch on the fastest steed you can steal from the Confederates. Because you see, in this world, there's two kinds of people, my friend: those with loaded guns and those who dig. You dig.

The Lowdown

GStreamer Plugins has grown so big that it's hard to separate the wheat from the chaff. Also, distributors have brought up issues about the legal status of some of the plug-ins we ship. To remedy this, we've divided the previous set of available plug-ins into four modules:

gst-plugins-base

a small and fixed set of plug-ins, covering a wide range of possible types of elements; these are continuously kept up-to-date with any core changes during the development series.

• We believe distributors can safely ship these plug-ins
• People writing elements should base their code on these elements
• These elements come with examples, documentation, and regression tests

gst-plugins-good

a set of plug-ins that we consider to have good quality code, correct functionality, our preferred license (LGPL for the plug-in code, LGPL or LGPL-compatible for the supporting library).

• We believe distributors can safely ship these plug-ins
• People writing elements should base their code on these elements

gst-plugins-ugly

a set of plug-ins that have good quality and correct functionality, but distributing them might pose problems. The license on either the plug-ins or the supporting libraries might not be how we'd like. The code might be widely known to present patent problems.

• Distributors should check if they want/can ship these plug-ins
• People writing elements should base their code on these elements

gst-plugins-bad

a set of plug-ins that aren't up to par compared to the rest. They might be close to being good quality, but they're missing something - be it a good code review, some documentation, a set of tests, a real live maintainer, or some actual wide use.
If the blanks are filled in they might be upgraded to become part of either gst-plugins-good or gst-plugins-ugly, depending on the other factors.

• If the plug-ins break, you can't complain - instead, you can fix the problem and send us a patch, or bribe someone into fixing them for you
• New contributors can start here for things to work on

Phew. Released new GStreamer 0.8 tarballs of core and plug-ins over the weekend. That's two. Today I released five GStreamer 0.9 tarballs. That's seven total. Release of both gst-pythons will...

I've been mentally building up a backlog of stuff to write about and the moment where I sit down and just lay it all out isn't happening for some reason. So I've decided to do some reverse engineering and work my way backwards instead. Starting with tonight.

Went to see The Island with part of the gang and our honorary employee of the week, the Welsh Frenchie. It was actually quite enjoyable, and while I'm sure lots of people will slag it for the obvious plot holes and the overblown action ... Well, I've never understood why some people can't judge a film in its own category, anyway. And hey - what a happy ending the movie leaves us with - only 5m$ between you and a clone of Scarlett Johanson ! Finally something worth saving for.

Spent way too big a chunk of my weekend covering up for someone who shall rename nameless who completely dropped the ball on his responsibilities after managing to not prepare at all for the commitment he made; he made up for it by going to the beach instead of actually trying to show a morsel of remorse and a whiff of something that would vaguely smell of duty on a good day.

The only redeeming feature of my weekend was the fact that I had visitors over again - Els and Wiebe passed by on their way back from Malaga. We went on a wild tapas hunt looking for Can Tomas, which is rumoured to have the best bravas in town. But two taxis and a closed door later, we let the cab driver decide where to take us. He took us to La Cerveceria Catalana, which he said was one of the best places in town. Entering was like coming home after a long trip - a feeling of having been there before but stuff is out of place. The menu had some stuff that I knew, the waiters looked sort-of-familiarly-dressed, and the decoration reminded me of something. After a few moments it hit me that this place looked an awful lot like our favourite tapas place, Ciudad Condal. It was spooky. The downside was that I didn't find out about a great new tapas place. The upside is that I had great food and we really do know one of the best tapas places around.

Jukebox

Boy, does it take long to close circles. Some five years ago I did a thesis project in my last year at university. The goal of it was to do real-time mixing of MP3 files on a Pentium 200MMX. I don't know if people remember the good old days, but a machine like that had a hard time already just decoding an MP3. I found a really nice way of doing mixing in the subband sample domain, bypassing the most expensive steps of the decode and encode processes, and was able to mix up to four tracks simultaneously in real-time, outputting to MPEG Layer 2. The jukebox driven by that ran for a very long time before it finally got retired.

Now, this was the only good thing about the program I wrote. The bad thing was that it was a gross hack evolved from the reference encoder/decoder code; that it only worked for MP3 files; that all files were required to be of the same sample rate (otherwise the subband sample mixing is wrong); and that as a project it wasn't very maintainable at all. Now this last point was back then still a bit unknown to me - I was just getting started in Free Software stuff. But regardless - I started looking for stuff that would allow me to do a nicer jukebox program for the radio station, and while googling (or was it still called altavistaing back then ?) I learnt about GStreamer. That was around 0.1.1 I think.

I read the docs, was impressed with the ideas (little did I know back then :)), started poking at it, found some things that didn't work, got involved more and more, all with the idea of doing my mixer as quickly as possible. But pretty soon all things I wanted to do with something like GStreamer were moved to the back of my mind and GStreamer became a goal on its own. Sure, I wrote a bunch of programs with it, and once in a while I took a small stab at some ideas for the mixer, but I never pushed through.

Until a few weeks ago, where we needed a decent radio demo to be online 24/7 and I just didn't want to put in another hack. So I wrote some stuff in Python, found some bugs in the current versions of some of the first GStreamer elements I wrote (why did people let me commit DSP stuff in the first place ?), and now I have something that does a remarkable job at mixing music of *any* type GStreamer can handle, automatically, picking decent mix points and a per-track volume level that makes the average loudness over songs constant.

It's definitely not perfect yet (Bohemian Rhapsody's dynamics trip it up for example), but it's a good first stab that I'll work on refining over the next few months. If you want to give it a listen and tell me what you think, the URL is http://stream.fluendo.com:8831 for a HQ Vorbis stream. (There's an mp3 stream at 8833 and a LQ Vorbis stream at 8832)
You can also run it for yourself; get GStreamer core, plugins and Python bindings for the 0.8 branch from CVS, go into gst-python/gst/extend and run

python jukebox.py playlist.m3u queue max-size-bytes=0 max-size-time=3000000000 ! { osssink }

. The program will eat a *lot* of CPU while it's scanning all the files from your playlist for good mixing points, but once that's done the values are cached.

I'm making a much bigger deal out of having finally done this than it is, but it is weird how these things take on a life of their own and run away with you. It was nice to finally just get something done that I was planning to do when learning about GStreamer.

Memory hunting

The only problem I had was that the jukebox component I wrote for Flumotion was eating too much memory. After a week, it was consuming 50% of the 512MB memory on the machine it was running on. I spent some free time looking into what was going on, fixing various memleaks in the elements (I also finally learnt how to properly valgrind python applications), learning some more about pygtk's refcounting and cycle problems, and doing some more explicit cleanup in my python code. I got to the point where I was sure my program wasn't leaking in the sense that it was dropping references to blocks of memory.

But it was still leaking memory in the sense that memory use was slowly increasing over time. The problem is that that sort of thing is pretty hard to work on in Python. I started adding some debugging that made the program garbage-collect at specific points in the program so I could bring some determinism to the freeing patterns and cross-reference with the GStreamer object refcounting debugging. That allowed me to fix a bunch of elements that were kept around, but not used anymore.

To get a better sense of how bad the problem was, I wanted to run the program outputting to a fakesink, basically making it mix as fast as it could. The only problem is that my home desktop heats up fairly quickly, and starts flooding the console with CPU thermal info around 70C, making the system unusable. This happens after about five minutes of 100% CPU use.

So, because I really wanted to run these tests over the weekend, I finally dove into the wonderful world of case modding. I took Wiebe to some of the computer stores and we bought a whole set of fans - plus a KVM switch that I wanted for some time so I could stop diving under the table whenever the server machine had a problem. I took the opportunity to completely rework the engine room layout, separate the two machines (they were standing right next to each other), install the fans, work out airflows, and re-paste the CPU's. And now the machine is running at 40C idle, and no more than 54C at 100% CPU. With that, I could run the jukebox at 100% CPU, and at this point it's mixed about 12000 songs - which is over a month of output. The Data size is around 44 MB, or a good 10% of my memory. So it's still increasing slowly, taking about 3.7K per song.

Not bad, but could be better. At this point I'm stuck. If anyone has ideas on how I could further figure out what could be causing this, let me know. I've tried massif, but both in 2.4.0 and 3.0.0 it failed on the first song. I've tried memprof, but I can't even get that to run anymore. My next hack will be something that LD_PRELOAD's to track allocation and freeing, and has a python hook so I can ask for the allocated memory at specific points (for example, each time a song has played completely) that are comparable to each other, to figure out what I'm not getting rid of each cycle. Not sure if it's possible to do a python binding for a lib you LD_preload, but it's worth a shot...

Anyway, upwards and onwards. I can't imagine why I've went in such detail on so little given that I was planning to flush out a whole backlog, but folks back home were getting worried about what I was up to...

I've been mentally building up a backlog of stuff to write about and the moment where I sit down and just lay it all out isn't happening for some reason....