Present Perfect

First of all, congratulations to Joe Shaw on getting hitched! I can't believe that, only a few months ago, I was having drinks with him, and when I asked him how he was he didn't even mentioned getting married! I guess it was already a plain simple fact of life for him by the time.

But to keep it technical, I read through his reply to some Beagle comments, which resonated with me:

And lastly, having worked with Trow on a reasonably big desktop Python app, we wanted a strongly typed language. Writing real applications in Python requires a discipline that unfortunately most people (including myself) are unwilling to adhere to, and this easily leads to buggy and hard to maintain programs. You have to be very diligent about unit tests and code coverage for every line of code, because you can’t rely on the compiler to catch errors for you. We had been burned by this a bit, and wanted to get back to a strongly typed, but still easy to use language that integrated well with the desktop.

The core of this message is 100% true - you need discipline to write real apps in Python; you have to be very diligent about unit tests and code coverage.

The crux of the matter though is that this is true for *any* real application in *any* language. However old and boring it makes me feel to say so, doing these things are part of good software engineering practice, and for good reason. Since realizing this, all the projects I've started working on, as soon as they leave the single-day-hack-prototype phase, first get converted into a simple bare-bones project, and the first things I add is unit test, code coverage and API documentation infrastructure. This is very hard to retrofit on an existing codebase; you teach yourself the habit of doing so at the start so you can reap the rewards later.

Unit tests and code coverage support two separate things. The first is that they give you confidence, if done right, that overall your project code is working well. It allows you to decide to release because you have some way of measuring that the quality of your project is at least as high as on the previous release.

The second is that you can refactor correctly. Some of our hackers in the office incorrectly use the word refactor when they really mean "I don't like or understand this big section of code and it's buggy and I am going to rewrite it completely and see what happens." Refactoring is much more well-defined than that. It is the act of rewriting the internal implementation of a section of code, without changing the external behaviour at all. (A logical consequence is that refactoring code does not fix bugs - you need to produce the same bugs in your refactored code!)

How do you know if you changed the external behaviour ? You can only know so if you have unit tests which cover the code you're refactoring.

As a side note, a distinct bonus of your project having unit tests is that it makes it easier for others to hack on. Just this weekend, I found a simple bug in pychecker. Now, pychecker has a unit test setup, so that means I can add the unit test that is failing, figure out where in the code it's failing, fix the code, verify that it works with my unit test, then verify that I did not break any other functionality by running the complete unit test. The result is 15 minutes of patching work, and it's a joy to go bug fixing this way.
Getting back to Joe's post. I'm not sure Joe intended it to sound that way, but I felt the conclusion he drew from his experience with Python was "I don't use Python anymore because it forces me to think about unit tests and code coverage."

I don't know what kind of tests Beagle has as part of its codebase and release procedure. I wrote about my experience with Beagle at some point
and Joe commented to the effect that the kernel gives you 8192 inotify watches and when you run out, it acts funny.

Well, no shit. Of course I have over 8192 directories under my home directory. If Beagle is, for example (I have no idea if it does), really watching every .svn and every CVS directory, no wonder it is running out. Two questions spring to mind - how did Joe get away with not having over 8192 .svn and CVS directories, and why does the software not deal with this gracefully ? If I would learn about this limit, I would write an integration test (probably not a unit test, in this case) and test graceful handling of this boundary condition. (It's entirely possible that a test like this exists in the Beagle code, I haven't checked - I don't intend this as an attack on Beagle or Joe, for those who assume the worst in me :))

To cast a stone at myself, GStreamer for a long time only had a bunch of small ad-hoc test applications that developers would write as they hacked on some feature, then at best it would be integrated in some make check command, or at worst it would be forgotten about and bitrot. I think one of my biggest contributions to GStreamer was to reorganize the test suite into a coherent whole, and making it easy for developers to integrate into it. It's obviously not the only reason 0.10 is so much better than 0.8, but it has definitely helped a lot.

I'm not saying a program is automatically better for having unit tests and code coverage, that would be silly. A perfect programmer can knock out a perfect program with zero coverage and tests. A crap programmer can knock out a crap program with 100% test coverage. But all things equal, having tests and coverage makes your program better. These days, I definitely trust programs more, and prefer to use them more, if I've seen they come with tests, and the developers care about their quality.

The first project that really convinced me of these simple principles was the amazing Twisted project. In spite of the sometimes vicious social commentary those guys receive from their peers (bloat, overdesigned, badly documented, downright crazy ?), which makes them sound jaded sometimes, these guys have, over several years, created a consistently excellent framework. They care about their quality and their users, and have integrated so many of these good engineering practices in their daily workflow. I'm not sure they themselves realize how rare this was for a FOSS project back then, but for me, Twisted as a project was an eye opener.

Wow, this post is a lot longer than I intended it to be, after waking up way too early and still fighting the aftermath of a stomach flu. I have to get going.

Anyway, here's what I want you to take away from today's post.

I program in Python precisely because it forces me to write unit tests and think about coverage. You should be doing this anyway, regardless of the language you do it in. By choosing a language Python, you will realize why much earlier on in your project, or your project will fail spectacularly :)

First of all, congratulations to Joe Shaw on getting hitched! I can't believe that, only a few months ago, I was having drinks with him, and when I asked him...

Josep had a GStreamer problem last Friday and really really wanted to bug me about it because everyone else he usually bugs about them is gone :) Julien was at IBC, Andy is doing an Andy, Mike is in the US, Zaheer is in London, and Jan is in Ireland. All with varying degrees of longetivity and permanence.

Now, I've hardly done any code with GStreamer for a while, but I helped him trawl through the log file he gave me until we found a likely symptom for his problem. He went away again then came back later asking me to check his theory about the bug.

Not to bore you with details, but it boiled down to a simple question he had: does gst_buffer_create_sub() set the master buffer's caps on the subbuffers it creates or not ?

Well, first of all, I have no idea. Second, why does he ask me, he's been working here for a year now so he should be able to find his way around GStreamer - but that's a post for a different day. Third, it's likely that the answer is in the documentation.

But most of all, why don't you just give it a try ?

He looked at me, puzzled, so I fired up an ipython shell. He said "Well, I don't know Python", and I replied "then it's time you give it a try, because you're about to get the answer to the question you asked me".

I could finish of the post here and lead you to believe that it's really easy to figure out the answer using ipython, but I'm guessing there might be more people out there who need a simple example to realize the power of trying things in ipython. So here goes - I'm pasting my session verbatim, mistakes and all, with comments in italics.


[thomas@level ~]$ ipython
import Python 2.5 (r25:51908, Apr 10 2007, 10:27:40)
Type "copyright", "credits" or "license" for more information.



IPython 0.7.2 -- An enhanced Interactive Python.
? -> Introduction to IPython's features.
%magic -> Information about IPython's 'magic' % functions.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.



In [1]: import gst



In [2]: gst = gst.Bu



gst.Buffer gst.Bus gst.BusSyncReply
gst.BufferFlag gst.BusFlags



In [2]: gst = gst.Buf
gst.Buffer gst.BufferFlag



In [2]: gst = gst.Buffer()



In [3]: caps = gst.caps_from_string('audio/x-raw-int,channels=2')


---------------------------------------------------------------------------
<type 'exceptions.AttributeError'>        Traceback (most recent call last)

/home/thomas/<ipython console> in <module>()

<type 'exceptions.AttributeError'>: 'gst.Buffer' object has no attribute 'caps_from_string'

In [4]: caps = gst.caps_from_string('audio/x-raw-int,channels=2')
---------------------------------------------------------------------------
<type 'exceptions.AttributeError'>        Traceback (most recent call last)

/home/thomas/<ipython console> in <module>()

<type 'exceptions.AttributeError'>: 'gst.Buffer' object has no attribute 'caps_from_string'


In [5]: import gst


In [6]: b = gst.Buffer()


In [7]: caps = gst.caps_from_string('audio/x-raw-int,channels=2')


In [8]: b.set_caps(caps)


In [9]: b
Out[9]: <gst .Buffer 0x9980c0 of size 0>

In [10]: b.caps
Out[10]: <gstcaps at 0x927140>


In [11]: b.caps.to_string()
Out[11]: 'audio/x-raw-int, channels=(int)2'



In [12]: b = gst.Buffer('kakapipi')


In [13]: b
Out[13]: <gst .Buffer 0x998140 of size 8 and data 0x6b616b61>


In [14]: b.set_caps(caps)



In [15]: c = b.create_sub(2, 4)


In [16]: c
Out[16]: <gst .Buffer 0x9981c0 of size 4 and data 0x6b617069>


In [17]: c.caps


In [18]: c.caps.to_string()
---------------------------------------------------------------------------
<type 'exceptions.AttributeError'>        Traceback (most recent call last)

/home/thomas/<ipython console> in <module>()

<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'to_string'

So there's Josep's answer. I hope this makes getting answers from GStreamer easier for some people out there.

On a slightly related note, why is it so hard to copy and paste code or output in WordPress ? I've installed the code markup plug-in, disabled the advanced editor like it asked, and it still insisted on pretending anything in angle brackets was a tag that it then had to close, explicitly, in my text.

Josep had a GStreamer problem last Friday and really really wanted to bug me about it because everyone else he usually bugs about them is gone :) Julien was at...

I type duck all the time.  It makes me feel pythonic.

I type duck all the time.  It makes me feel pythonic.

Europython is turning out to be quite interesting. It takes some effort to start talking to people given that I know relatively few here, but I'm learning lots of new stuff. But I'll save more for some other post.

Right now I want to focus on the more amusing combination of Linux and hotels. So here's the thing. The guys from Tuxdroid are here, selling their little Tux robots. For those who don't know, this is a small robot with text-to-speech that can flap its wings, make its eyes glow blue, and run around, as well as serve as speaker. I'm sure there's other stuff I'm missing. They're selling these babies for 79 euros, and the company is Belgian. They have a bunch of code available to program the thing (radio-controlled with a USB dongle), and it has a bunch of stuff in python. So far, so cool.

So here's the thing. I plug Tux into the power socket to charge him. When you start charging it, it says "Hello" and flaps its wings. In the hotel room, when I leave the room, I have to take the card out of that socket that controls whether there is power at all in the room (which is annoying in hotel rooms, because it means you can't leave your laptop charging while you are away.)

So, what happens when I come back to the room at night, get in, and put the card in the socket ?

A little bird flapping its wings and shouting "Hello" because it's getting fresh power juice. Too cute.

I'm wondering how I'm going to lug this thing around in my backpack for the next set of flights...

Europython is turning out to be quite interesting. It takes some effort to start talking to people given that I know relatively few here, but I'm learning lots of new...

The last package part of the Twisted splitup in Fedora Extras has finally hit devel and FC6: the python-twisted umbrella package that pulls in all the others so that the upgrade from previous python-twisted (Sumo) packages is seamless.

Of course, in the long time it took to siphon all packages through, the Twisted team has released a new version. So my work is not done yet - but I will put off updating them to next week when I'm in a more comfortable place to do the work.

With that, it is finally getting time to take a look at how we can split up Flumotion so that people can install manager and worker pieces completely without any GTK/X dependencies. Twisted has split in pieces, pygtk has split off pygobject, so everything is ready now.

Flumotion 0.4.0 is right around the corner, Zaheer has done the hard testing work and Andy's been fixing robustness-related issues all over for the last month. I should get off my ass and start documenting some more.

The last package part of the Twisted splitup in Fedora Extras has finally hit devel and FC6: the python-twisted umbrella package that pulls in all the others so that the...