Present Perfect

You all could have told me I had been hacked, or rather, rooted, last week. What good is a weblog among the stars if the stars don't talk ?

Anyway, I should have known right from the start that the system acting funny would indicate something bigger. Everything has been re-installed now and my boss has seen the need for good intrusion denial and detection. So I installed and took the time to know some decent intrusion tools. I'd tell you which if my Hacking Linux Exposed book would allow it. But it would open me up to social hacking. Even mentioning the book is bad because then people know I probably installed tools mentioned in that book(*)

So, year++. Let's get GStreamer officially accepted world-wide. At the moment we're deciding on the right canonical name for packages and tarballs after the module split. Things start to build nicely again, I have a build script running on a few machines which even builds fresh RPM's daily if possible. We might actually be doing something resembling quality control real soon. But let's decide on the names, it's stalling us.

As for Dave/Dina, this is the year I want to see it happen. I'm going to fix up some stuff then release some form of a distribution and try my best to attract new developers. I've laid down the groundworks ok, now it's time to get it rolling.

(*)Except I didn't. Heh.

You all could have told me I had been hacked, or rather, rooted, last week. What good is a weblog among the stars if the stars don't talk ? Anyway,...

My boss is on holiday. He had set up my schedule so that I could program on one thing for three days. Ah, the cruel hand of fate !

I arrived this morning to find one of our servers with a cryptic Unable to load interpreter /lib/ld-linux.so.2. And you thought stuff like this only happened on Windows (which I refuse to spell incorrectly, btw).

Now, I've seen this error before, and mostly it happened when too many processes were running at the same time and memory space was exhausted - which shouldn't be happening anyway, but it does sometimes. Mostly when I try out a new monitoring tool which goes into a frenzy because of an NFS or samba mount error. But it has always happened when I was still logged in on a terminal somewhere so it was, with some effort, fixable. But no such luck in this case. Since I've made an effort to log out any root terminals left open on the servers I now find myself stuck.

The server is in heavy continuous use : all of our journalists write their pieces in their Explorers, and news is read every half an hour during the day. So I can't afford much downtime. But a quick reboot might do the trick, right ? I can't hack into my own system, so it's my last option. I will not learn what caused it, but at least it'll get on.

Well, no, actually. On a fresh reboot, I couldn't log in either. This was getting worrysome. I quickly tried booting in single user mode and that worked. I downloaded newer glibc packages (which include this problematic library) and rebooted, but still the same. I was getting nervous.

What's a guy to do in a situation like this ? Well, either you panic and start fixing it like mad, or you look and see how any possible solution can be fitted into the bigger plan. Let's see. First of all, the server still works. Only we cannot log in, which is problematic for the import of playlists, as well as for the newsletter I spent enough time on already lately (about which you can also read in this diary ;) ). So people are sure to complain.

Well, the long-term plan was to upgrade a few servers here anyway. I installed them when I was young and fresh at the job, and I might have dropped the ball here and there. I know for sure I installed a tarball too many and a few alien RPMs as well. People sometimes complain about rpm, but here too, it is a case of poor workmen blaming their tools. I used to blame it as well, but now I know that if you heed the warnings rpm gives you, and if you stick to RPM's on an rpm-based system, you're safe.

The thing is, I'll never be able to shut it down for the hour or so I'll need to upgrade or reinstall it, so how do I fix that ? Well, the answer should be easy. Upgrade one of the other, less critical servers (that dual-processor machine with RH70 and a 2.2 kernel that keeps crashing anyway for example) and let that take over for a day or so. So I started upgrading that one. And being the reckless idiot that I am, I thought it was a good idea to finally find a solution for the opened cases problem.

This problem being the one where both the server case and the external storage case need to be left open because I could only get a connection to work between them using an internal twisted SCSI cable. Yes, you are allowed to laugh. This would actually be a good idea since last year during winter it started snowing inside the building through a hole in the roof and the next morning I arrived to find a small puddle of water on the carpet ten centimeters next to the drive array. The opened drive array.

Anyways, this wasn't such a good idea : I tested various SCSI cable connections, but my terminator still lit up green while my hunch was it was supposed to lite red, judging from the other two terminators on the tape streamers.

I spent three hours working on that - my boss will love that when he returns ;) - and gave up. I did run the internal cable through holes at the back of both the server and the drive case.

So I considered if I should install fresh or upgrade. Someone convinced me to upgrade, even though I wasn't too happy with the drive layout. So I did, and of course the new kernel didn't boot. Probably because we can't expect Red Hat to compile in each and every raid controller into the kernel. So here I am, booting from another older kernel (in that respect, GRUB is great), downloading the new kernel updates and recompiling a kernel to match my system.

Let's see if this time I at least get my kernel configuration right the first time around. That's what experience should do for a person, right ? *SIGH*

My boss is on holiday. He had set up my schedule so that I could program on one thing for three days. Ah, the cruel hand of fate ! I...

Lots of stuff has been happening as of late. I stopped working on Dave/Dina for a bit, I wanted to help out on GStreamer.

After agreeing on splitting plugins from the core (finally), we had a last-minute disagreement on how it should be done. Do we use a given directory structure for the plugins, or do we lump them all together ? We're talking about close to 100 plugins by the way.

I had an idea that would make the build maintenance a lot easier, IMO, after having spent quite a few days on fixing build issues and removing cruft everywhere. Some others thought that the advantage here was minimal, and thus it wasn't worth it to use a directory structure.

I'm glad I stood up to it, even though I might have less experience with these issues. IMO, if you have taken the time to get really intimate with autotools and build issues, and got some first-hand experience on how it works and how it's supposed to work, then your voice should count for something.

In the end we decided to vote on it together. It might not be the best solution, but at least it allowed us to move on and get it done. Which we promptly did, after the vote.

GStreamer CVS is in pieces at the moment, but things are starting to work out fine. It builds, it even plays stuff again, and it is easier to maintain at the least !

Now we're working out stuff for the next release which is scheduled in a week.

Using my work's bonus, I bought something totally irrational : a Lego Mindstorms developer set ! It looks cool. I haven't opened it yet though because I know I won't get anything else done anymore as soon as I start ;)

Lots of stuff has been happening as of late. I stopped working on Dave/Dina for a bit, I wanted to help out on GStreamer. After agreeing on splitting plugins from...

The good news is that GStreamer (your future) has a new release out. It took a lot of sweat working on the autotools stuff to get it to build, compile, install and check ok. GStreamer should release more often and hopefully it will.

Whacking the GStreamer ball around all weekend made me realize a few things :

• I don't know enough autotools
• People always blame autotools, when the tools are probably not to blame
• Autotools are hard to debug when you're actually trying to debug them in the project you're trying to get running. Most of the time, you don't want to spend time forcing autotools to do what you want, you want to get them out of the way.

To solve all these problems at once I decided to create a new project : Autostars, the Autostar Sandbox. The idea is delightfully simple, and I suppose I'm not the first one to think it up.

The main problem with debugging autotools stuff is that the fix/run/test cycle takes too long when you do it in a project. While fixing GStreamer builds this weekend, I often tried fixing checks. Running autogen.sh after that again took a few minutes, because GStreamer checks about fourty libraries. So by the time this is finished you have long left your window of concentration and you end up getting frustrated. You're also not learning anything about autotools. You're just throwing cement at little holes all over the place and hope it sticks.

So, the solution is simple : strip down the files involved (configure.ac, Makefile.am, ...) to the bare essentials for that test only, and you'll have a much quicker result of your changes. This is what the Autostar Sandbox is for : simple, atomic tests, showing you how to implement particular tests. Not every autotools problem can be solved this way of course, but most of the basic ones can.

There's not much in there as of yet (five checks at the moment), and it's only in CVS, but I invite you to try it out anyway if you're currently having problems getting something to build. You'll actually fix your build problem quicker if you do it this way (I know, it's paid off for me in the last week as I implemented GStreamer check bits in autostars), you'll end up less frustrated and maybe with a bit more knowledge, and best of all : you get to share that knowledge with other developers ! Enough to make you feel fuzzy and warm all over. My boss would say it's a win-win situation.

The good news is that GStreamer (your future) has a new release out. It took a lot of sweat working on the autotools stuff to get it to build, compile,...

Here's a reminder of what insomnia can do to you. Like I said before, the GStreamer team is getting a new release ready.

One thing that might need to be done after this new release is to split off the plugins from the core to make it more manageable. To this end I had proprosed a division of the plugins among directories. One of the developers (who shall remain nameless to protect the innocent ;) ) spoke out against the idea of directory classification, and argued that we might as well make a flat tree.

Well, yesterday night I couldn't get to sleep and got up again, and on IRC someone commented on the plugins, and he said that "I might as well put all of the .c and .h headers in a separate .tar.gz".

Hah flipping hah. So I thought, sure, why not ?

Ladies and gentlemen, you can download an unofficial pre-release of GStreamer 0.2.99. Grab the tarball (which contains more than 1700 tar.gz's) and the script to unpack them, configure the thing and make it. Send us mail as feedback on whether or not gstreamer compiles for you while you're at it, because all this fun stuff actually serves a purpose. Or drop by in #gstreamer on irc.openprojects.net

Needless to say, I got a good night's rest out of it, though a bit short.

Here's a reminder of what insomnia can do to you. Like I said before, the GStreamer team is getting a new release ready. One thing that might need to be...