Present Perfect

Back from holidays. More on that later. For now ...

Peter

finally found out that where his fancy power block is. But he's insinuating that I'm a thief. Now, if I was a thief, would I really be so dumb as to link so obviously to a picture of his belongings ? No, of course not. The truth of the matter is that this power block was taken with me to Barcelona on the last day in our house in Gent. The day where Peter said "I'm sure I took everything that's mine, if you find anything else you can throw it in the trash." But would a friend throw one of another friend's most prized possessions in the trash ?

Well maybe another friend would. But not I.

Perl

The good thing about moving to Python is that I've completely forgotten Perl. I've tried to read some code recently from someone in Perl and it was *hard*. I think that's great.

The bad thing is that there's still Perl out there. Someone recently asked me about a small project I did at the radio station, libuecp. This is a library that speaks the UECP protocol to send commands to RDS transmitters. RDS is the system that sends info your (car) radio like text, station ID, traffic announcements, ...

He had a small issue with it so I decided to clean it up and possibly hand it off. Call it hobbyist pride if you will - this is code I wrote three years ago and really, I should at least make sure it works and can be used, even if I won't rewrite the perl bits just now. I was trying to fix make distcheck, and I noticed I had to put in a hack because perl-generated makefiles do have an install command. But the uninstall command says that uninstall is "unsafe and broken", so it doesn't actually *do* make uninstall. That makes it pretty annoying to integrate with autotools.

I'm sure other people have tried doing this, but google didn't find anything and I couldn't be bothered solving an integration problem for a language I'm pretty sure I'll never use anymore myself anyway in the future, so I put in an uninstall-local rule with lots of finds. Ugly, but works.

Friends

An and Mike passed by after we got back from holiday. And suddenly I'm a godfather-to-be.
Found a new restaurant close by our place yesterday, it's only been open for two weeks. Had a very good tuna fish tartar.
Andy feels I should really write about every little thing I do, like tying my shoe laces, not just this overview stuff. For now I'm showing remarkable restraint.

Pet peeves

#19028475: people referring to Windows using any variant such as windoze, winblows, winslow, window$, ... Seriously, stop that crap. It makes you so unmoney. It wasn't funny the first time and the joke hasn't aged well from that point on.

#19281741: autoconf making configure check for all sorts of versions of fortran. There must be some fortran lobby out there sitting on INSANE chunks of money ! Seriously ! WHY ??? Can someone tell me how to make configure not check for all this fortran crap ?

Back from holidays. More on that later. For now ... Peter finally found out that where his fancy power block is. But he's insinuating that I'm a thief. Now, if...

dmix

Since dmix is heralded as the solution to desktop audio, I decided to give it another go on my laptop. Sadly, I was incredibly underwhelmed (I know it's not a word cause I looked it up). First of all, it manages to add a little distortion on my left channel. Really weird, I don't know why this would happen, but it does. Second, the mixing is not very good - playing a song and a sine tone at the same time shows obvious bleed that shouldn't be there. Third, and this is much worse, every ten songs something goes wrong. Either it goes into continuous interruption mode, much like a skipping CD player, or it just locks up completely. a pause/play solves it, but it's damn annoying.

I don't know if this is my driver, my sound card, whatever. All I know is that if a fairly standard Thinkpad T40 can't rely on dmix, it's probably not yet something that should be enabled by default...

webcam
Got a new webcam for work handled by the spca5xx driver, since we got a bug report for it. I got a nice little Creative camera that's really small and attaches to your laptop easily. I've built RPMS for this webcam for FC3, so please try these out and let me know. I got a nice mail from someone this week that just said thank you for the PWC kernel packages, and a thank you once in a while makes all the difference :)

GStreamer
Finally finished our buildbot setup. Zaheer was the first volunteer to set up a slave - a nice 64-bit gentoo slave. I added it to the "test" set of builders, so it wouldn't insult anyone while we were fixing all the issues, and then we went through all the compile and run problems yesterday one by one. 24 hours later I added it to our 0.8 buildbot, and now look at all those purty green colors :)

If you have some architecture or platform lying around that you wouldn't mind setting up a buildbot slave up on for us, please contact me. I'm looking for Sparc, S/390, powerpc, Mac OSX, ubuntu, suse, mandrake, ...

Are you listening, Sun ? This is a way for you to feel less like a secondary citizen in the GNOME world and beyond...

A 0.8.10 release of GStreamer that works on 64 bit and for gcc 4 is being cooked up as we speak.

Dreams
I dreamt I was at a dinner party. I had blue furry ears and I had a blue cloud on my belly and a taste for honey. I was feeling anxious because I was supposed to show and tell and I didn't bring anything so I zipped off on a rainbow to my cloud.

Gnome Fun
While there's a grain in truth in pretty much everything everyone is saying, I do have to say that I don't think that it's really as bad as is being said. In any community I've ever been a part of for a longer time, there's always the sense of wonder and fun and discovery at the start - and there is always a sense of disenchantment somewhere along the way when you realize that there's a lot of work to be done, there's a lot of time to invest, and things just aren't as fresh and new anymore as they used to be. And people change along the way and become interested in new things.

It has been this way for me in scouts, at the college radio, in GStreamer (though it comes in waves), and in GNOME. So I tend to think it's not necessarily something that changed in GNOME that has stirred this feeling. It seems to me that a lot of people who have stopped having fun hacking GNOME are precisely people that have been around for a while. When I see new people coming in on any of these projects, they still seem to be very enthusiastic and full of energy.

Maybe "us" old-timers need to realize just how much is the lack of fun because the project changed, and how much is due to the lack of fun because we ourselves have changed. I can't blame GNOME for my lack of hacking on the platform - I am just too deeply immersed in other stuff right now. And sure, valid points have been raised. But at the end of the day, the only person who is responsible for making a project fun to hack on is you.

As for some points raised by people in particular:

• John, why the hell do I have to find out about Snagglepuss like this ? :) So you're a cool Multimedia Hacker and none of us ever knew ?
• Mikael, I agree in part with what you say. It is true that pieces of our development platform could be better. My number one source of irritation is my innate desire to want to follow the HIG (because this is an aspect where I want to be told what to do because I acknowledge my incompetence in the field), but the huge disconnect between GTK+ and the development tools, and the HIG itself (As an example, here is what your slider should look like, but if you really want to put in tick marks, you have to hand-code everything). I guess that either working on a project to translate the HIG to widgets, or working on a project that uses HIGGY widgets would be heaps of fun, but working on a project that tries to respect the HIG today is not that much fun. However, it needs people to pull together and define a direction.
• Beside that, I'm particularly interested in your opinion on how much of your discomfort is because of your personal changes. You've been around for quite some time, you've started a company related to GNOME, you got married and now are going to have a kid ... I would expect your priorities have shifted somewhat, and you've probably changed as well ?
• Benjamin, the Sullivan thing is a prime example of cubicle humor - something that only happens in a corporate environment :)
• Questions about D-BUS ? Poke J5 or davidz on IRC. They're almost always around and two of the most helpful people you'll ever encounter on IRC
• The comment about GStreamer is understandable but particularly unfair - but I'll move that to below since it's going out of scope
• If you think that not being allowed to break ABI makes things less fun, you have a wide open 0.9 branch where ABI doesn't matter. Nothing is stopping you.
• People in companies come and go. When you started on GNOME there were already people quitting their jobs over it.
• I can see where Edd is coming from. Hacking out your place on a very specific subject that not a lot of people deal with - in his case, Bluetooth - can be a lot of fun at the start, but inevitably you need some help and feedback about the choices you make and sometimes that just does not happen. I wish there was something I could do to help GNOME hackers in this situation, but it's just not there. Same for all of those SUN hackers out there wishing people would care more about Solaris - we're just not using it. Reach out that small bit, like James said, and provide access to your hardware, and you will see people fix all the problems.

GStreamer & Fluendo
So coming back to Benjamin's point, I want to expand a little. It's a bit of a stretch to go from a temporary lack of motivation and a discussion about GNOME Fun to blame The Evil Company.

No point in going into details, but people should consider that we invited the core developers from GStreamer to come over in February and discuss ideas and design stuff for 0.9 to resolve the standstill. We did that the year before as well and I feel it was very motivating. Benjamin, you were there too, and you told us you were working on reverse-engineering stuff and weren't motivated enough to hack at the time. Also, you always have said you don't like designs and don't want to waste time on it, preferring to let the code speak for itself. We discussed the plans, everyone got their say, we came to a fair consensus, and we went on. Wim chunked up his changes so everyone could follow, and we're still waiting for comments and discussion on his design documents, be it on IRC or by mail. At no point during this whole process did we lock you out. If anything, not taking the chance to participate in this process makes me feel that at some points you locked yourself out.

The point is simple - I think Fluendo as a company puts quite some effort into being very cooperative with the rest of the community. Sometimes I wonder if that's worth it at all. Companies are evil by definition and some people always reason from that starting point. Nothing you do as a company will ever be pure and unsuspect again - some people will always complain no matter how much effort you make. I see very few companies out there doing this at all. Sometimes I get disheartened by that and wonder why we bother. On the other hand, there are enough people that seem to see us for what I think we are: a company hacking on GStreamer and building stuff on top of it, and genuinely trying to make it better for everyone involved. If you disagree with this, feel free to drop me a line and tell me why, because I'm interested in knowing what we can do better.

You know, even if Wim was really doing all his designing behind closed doors and no one was allowed in - I'd still prefer a design I can trust over a heap of code that's thrown together and starts breaking as soon as people silently disagree on the way to fix things. A problem GStreamer surely has suffered from in the past. I, for one, welcome our new threadsafe overlords.

Holidays

So, tomorrow morning (way too morning) Kristien and I are leaving to go to the south of Spain for ten days. I'm not in an urgent need for holidays, but I think I'll enjoy this one regardless... See you all in ten days.

SPAM
One type of SPAM I really don't get at all is the "Want your sex life the way it used to be" type. When I was fourteen my sex life was lonely and self-involved. Of course I don't want it the way it used to be - what makes you think you will get me to buy your product this way ?

dmix Since dmix is heralded as the solution to desktop audio, I decided to give it another go on my laptop. Sadly, I was incredibly underwhelmed (I know it's not...

Weekend

My ex-boss and his wife came over, together with a couple they knew. The guy was going to ask the girl to marry her and Kristien had arranged a nice table on the beach for them. But before that, we went to the park and drank six liters of Caipirinha between the six of us. And I was pretty much on an empty stomach...

BuildBot

Warren did a new release this week, and I'm happy to see a bunch of my patches have gone in ! Also, I think our current buildbots look very spiffy.

I had started learning arch (using baz) for developing on BuildBot. What I did was create a branch off the last release point, and then create a branch per feature I was hacking on. Then I made one big branch that collected all of those features into a fluendo branch which I used for deployment. That worked out pretty well.

However, now with the new release I wanted to again branch the release version and finish up other features against the latest release. And for easier upstream merging, I also want to merge in these patches against a branch that tracks buildbot head. So I guess I'll have to redo my whole tree locally.

I'm enjoying hacking on Buildbot a lot. It's a nice project, it has fairly nice code, it just misses some documentation. There was however the initial problem of getting Warren to look at what I was doing. In the beginning I was describing what I wanted to do, I sent a few patches, and then I was waiting for comments because I didn't feel like going off very far on my own copy and then not be able to merge back because he wanted it done differently.

But meeting up in person at PyCon obviously helped. I got to explain some of the problems we deal with for GStreamer (stacked dependencies, conditional builds, ...), and get some ideas from him for those. After PyCon, Warren took some time and started looking at what I was doing. And I guess me not giving up and doing documentation work must have helped too :)

And Twisted is just sooo damn nice. The last time I had this sort of click with a project, where stuff just felt into place, and I got the feeling that here was a project that had some really good ideas, and some excellent people, was with GStreamer.

Discovering Twisted is like having that sense of wonder all over again.

Flumotion

We're looking at roping in some potential customers so I get to spend some time setting up tests and stuff. The upside of that is that we have some more stuff to play with and some nice ideas we can execute. The downside is running into the bugs you were trying to avoid in your own software :)

So today I got to buy a barebone 64bit machine to use for one of the demos, and that means I can finally start doing some of the 64 bit fixing myself. I was particularly embarassed a few months ago when Zaheer couldn't get his streaming up and running because of a bug in my code that I hadn't encountered myself in GStreamer.

As for Flumotion, currently trying to spend most of my time hacking on the RTSP stuff. It gives me ample opportunity to dig a little deeper into the internals of Twisted. Implementing from RFC is boring and rewarding at the same time. It gives you a chance to work on some other parts of the set of development skills one tries to build, because the hard work of designing is mostly done already.

Holiday
Next week Kristien and I are going to go down to the south of Spain to visit Malaga, Cordoba, Sevilla and Granada. If any of the Spanish people want to offer me some tips on what to see when I'm there, feel free !

Weekend My ex-boss and his wife came over, together with a couple they knew. The guy was going to ask the girl to marry her and Kristien had arranged a...

James, I didn't know bittorrent used rar internally. Or, let me rephrase - are you sure it does ? I've downloaded single files, and multiple files, and the multiples end up in a directory, one by one, and the first one is usable as soon as it starts writing the second one. You sure you didn't just get a torrent for a .rar file ? I mean, it could be that person hosting the torrent rar'd up his .avi files and serves it up that way. Which would be dumb, but possible.

Also, most video files are actually not streamable if you haven't downloaded them all. Unless you force index rebuilding which is really slow.

As for LCA, could anyone who's physically there please drop me a line ? I'm trying to figure out where I can follow the broadcasts, and apparently jdub thinks getting married is a reasonable excuse for being offline. Slacker !

James, I didn't know bittorrent used rar internally. Or, let me rephrase - are you sure it does ? I've downloaded single files, and multiple files, and the multiples end...

Algernon recently posted a reply to Christian's claim that flumotion is the best choice for streaming. (This is of course an example of Christian's youthful enthusiasm, crystallized in his "Promote early, promote often" motto.) He did make some good points, so I wanted to address them. I just wanted to take the time to give the anwer the time it deserves, so I'm a bit late.

Let me start by saying something that needs to be said: Icecast is *GREAT*. It's been around for quite some time, and back in the day when I was a student (my personal version of "When I was in the army...."), we put our university radio station on the Net with it as soon as it worked well enough. We were definitely the first in Belgium to stream live radio on the net, and probably the first in Europe, though it's hard to verify such a claim because there is very little info on this. But this was around 1995, I think (geez, where has the time gone :)) Also, one of the most active current hackers, Karl Heyes, is a very nice guy, always willing to help, and has given me some useful insight. Icecast has a good community.

I'll say some more about icecast later, but let me first address algernon's points and offer suggestions on how they can be done.

• Flumotion does not currently support more than one stream on the same port.

  This is correct, but a pure implementation choice because we were lazy and knew this is easy to fix. It's just that there are so many easy ways to choose from to fix this :) The main limitation here is that only one process can listen on a port. As soon as I get round to this, I'll do the most straightforward fix: have the streamer component take feeds from multiple other components, have the main thread listen to the port, and have Twisted serve one Resource per feed, on the same port, but different URL's. It's not a hard thing to do, it just needs doing. Estimated developer time: 4 hours.

• This also means - as far as I understand it - that if I want multiple streams, I need multiple flumotion instances.

  Not really - you start just one flumotion-worker binary on that server. This worker binary can fork multiple components all by itself. So you just put some more streamer components in the flow, as many as you like. Only one binary to manage. Unless by "instance" you meant to include these forks as well, but this is standard practice - look at Apache for example.

• There's no easy way to send a stream to flumotion, from a random GStreamer application. (...) For this, I would need to learn python, which I don't intend to anytime soon.
  Actually, there is. Here's what you can do to achieve this:
  1. start up Flumotion and the GUI
  2. set up a stream with only a video test producer
  3. verify that runs correctly and export the XML config
  4. edit the XML config, replace the videotest producer with a base producer type that takes pipelines
  5. use tcpclientsink for the pipeline with a given port and the GDP protocol
  6. restart flumotion with this new config
  7. producer component being hungry - nothing is feeding it data yet
  8. start up the following GStreamer pipeline: gst-launch videotestsrc ! video/x-raw-yuv,format=(fourcc)I420, framerate=5.0,width=320,height=240 ! tcpclientsink port=3333 protocol=gdp

  And there you go ! Total implementation time: 13 minutes, including feeding the cat (and at 9 am in the morning). Number of lines of Python code hacked: 0.

  I was surprised myself actually, I would have expected to have to fix at least one bug in the code. Thanks for prodding me to actually try this out. Now obviously this is not the "best" solution. If you stop the producing client, you need to restart the producer component as well, because the simple pipeline component type has no way to recover from the broken connection. And that's where you move from "quickly test something out with what's available" to "turn it into a 15-line component". Exercise for the reader.

• Flumotion does not have fallback mounts yet..

  IIRC, the fallback mounts can only be whatever the same icecast server is serving. So with the solution to the "multiple URL" problem, this can be done immediately as well. Twisted doesn't care *what* thing is providing a Resource tied to a path. So we'll probably get to this soon.

• With icecast, I can easily make relays.

  IIRC, Icecast has two types of relay. One is where you import a stream from another server that can be any HTTP streaming server: Icecast, Shoutcast, ... This one is easy to implement: you use a pipeline producer that uses gnomevfssrc to get the stream. There's only one catch at the moment - there needs to be a GStreamer element to parse the Ogg stream properly so that it can figure out what the streamheaders are. oggdemux ! oggmux should do this, but it has a bug, which I noticed while trying this just now. So, that's the fix needed. (Estimated developer time: 6 hours)

  The other mode is icecast-specific: icecast pulls in an .xml file from the other one that describes all streams, and then relays them all automatically. This is also doable, but would mean including icecast-specific support. This is something we'd really like to do - make Flumotion work really well with icecast. I see icecast more as an ally than as competion. The competition we see is Windows Media Server, RealServer, Quicktime Server (and HelixServer and Darwin Streaming Server). I'd have to look at exactly how this relay info is pulled though to say how long it would take to do this.

  (Zaheer reminds me that in Flumotion itself, it's of course even easier to do relays. You can create as many streamer components in your flow as you want, connecting to any other producing component, and the streamer can be on any machine in your network. And all of that is managed from the GUI. So in Flumotion, it's easy to do relays. We still need to make something nice though to go through firewalls transparantly - an advantage Icecast currently has because the clients are push-based).

• Icecast has this nice stuff with me being able to apply random XSLs to its XML output.

  If you tell me what this means, I'll cook this up for you. Estimated developer time: 1.5 hours

• Icecast has a usable C library that can be used to feed it

  As shown above, GStreamer is that library.

• I couldn't, for the life of me, get flumotion into a usable state under FreeBSD. It's not flumotions fault, per se, but of its dependencies.

  Ok, this is a point where I might get defensive and ranty, because it's not a fair point IMO. We all have limited resources and time. Most focus is spent on Linux, and this is the same with many projects. BSD users know this and are typically more used to - and apt to - fix problems themselves and get stuff installed. We know GStreamer and gst-python runs on FreeBSD. So does Twisted. We haven't tested it ourselves, we don't run FreeBSD. But it can't be too hard. You can't get a lot more portable than Python.

  Of course, if you don't tell us exactly where the problem is, there's nothing we can do to help. And we'd love to help it get set up. But it's not fair to blame a vague "I can't install this on FreeBSD" problem on Flumotion :) Talk to us and we'll help you get it fixed.

• While I think flumotion has a damn good design, I think icecast follows my ideals better. As the UNIX paradigm goes: it does one thing, and does that well. It doesn't try to be a complete streaming solution. It is simply a streaming server. And that I love. I know that flumotion is usable this way too (or will be, as soon as there is a flumotionsend element), but it does stuff I do not need at all. (...)That makes it harder to use for what I need, and makes the developers focus on things I will never ever use, so it lacks key features I do need.

  There are pros and cons to going either way. We think in the end that a product that can carry the task through from beginning to end is better. It's better integrated, it has a unified feel, and it allows you to do more powerful stuff. If one of my icecast streams is down, I need to leave the Realm of Icecast and go see what happened to my application producing the data. In Flumotion you just see in your GUI (which soon will have a notification area applet) what's up. However, you still can use it in "be only a server" mode, and yes, we want to make that work correctly. Saying that we focus on things you will never use, does not automatically mean we should lack key features you do need. You don't seem to be streaming AAC, yet icecast has spent time on getting that working.

• Oh, and you might probably ask why I didn't report these to the flumotion developers. That's a good question, and I admit I should have done that. However, I had deadlines, and that didn't allow me to properly report issues.

  This is indeed a crucial point. We're all strapped for time. You had a deadline so you didn't inform us of these problems you had. We have deadlines too. Yet here I am making the time to address your problems, even if it's after the fact. We care about our project. We care about people using the project. If they have a problem, we want to help them. It doesn't take much to get us to go out of our way to do so. Here I am replying to your points. Someone recently wanted to try it on SuSE and he was missing deps and built stuff by hand and something below the stack segfaulted. So I started looking for someone to make SuSE packages and now we have them (I still need to test them though, but VMWare seems to lock up when trying to install SuSE).

  You're in our IRC channel, you see us discuss. Feel free to bring up these points when you run into them, even if it's just cursory or you don't have time to go in details.

So here's the simple point: these issues you have you can let us know on IRC, on the mailing list, in our issue tracker, and we'll comment on them. When faced with our own deadlines, we don't spend the time to psionically probe all people for problems they might have with Flumotion. But we still spend time addressing valid points about it when we know what they are.

But don't take my word for it ! Try it out. Give us some stuff you want to see happen. And if you prefer to wait until we're done on our own, that's fine too. Let's meet back in a year and see where Flumotion is, feature-wise.
Flumotion is almost a year old. In software land, that's young. For a normal use case (which you indeed aren't, since you prefer to hack yourself and code up your own streaming source), I do think it's a better match than Icecast.

You don't really need relaying, you can have multiple streamers streaming the same stream, for the same effect, except that you have one less failure point. You have a nice GUI and wizard that makes it easy to set up. And the design underneath is flexible enough to handle more complex tasks, and the code is simple enough to get started with it quickly.

Now, back to icecast. I haven't used it much lately because I do not work for radio stations anymore since I started for Fluendo. The friend I originally did the streams with now works for an ISP, where they have 5 round-robin icecast servers with about 1200 streams each. He says the machines are easily pulling 60% CPU and more, which is why he needs to round-robin in the first place. One of our customers is using Flumotion to stream to 4000+ clients, pushing 80 Mbit. The CPU usage is below 5%. I wish I could test icecast in these conditions myself to see where the difference lies. Maybe they use an inefficient polling method ? Maybe something else is completely different ? But when he asked about it, they told him that they don't really have many users that serve that many streams, and they don't focus on those conditions.

One other thing that annoys me is that, while Icecast is closely affiliated with Xiph.org, it has always supported mp3 streaming, and now has added AAC streaming. You can sort of make a point for mp3 - it was around before Vorbis, it was the first thing icecast supported, ... But adding AAC does not gel well with Xiph.org's mission. Flumotion from the start has made it clear that we do not only focus on free formats. We can't do that, business-wise. But we actively support Xiph.org and the codecs they make, and we make sure they are first-class citizens in our world. And that this support is available for free. If you want non-free codecs support, you'll get non-free code. I think that's fair, and promotes free codecs better than Icecast does. We make people aware that there is a cost involved, both in money and in freedom, when using non-free codecs. And we go out of our way to make it easy to avoid that cost by using free codecs.

Flumotion is an active and young project, and I hope this shows that it's a very flexible project, easy to extend, and that as developers we are very open to other people trying out ideas and suggesting stuff we haven't thought about.

Algernon recently posted a reply to Christian's claim that flumotion is the best choice for streaming. (This is of course an example of Christian's youthful enthusiasm, crystallized in his "Promote...