Present Perfect

On this week's plane trip I finally took some time to port two of my trac plugins to 0.11 because work's migration to 0.11 seems blocked on them. I did so because I started needing more and more 0.11 features, and the migration didn't seem to be moving.

Funnily or sadly (pick your poison) after landing I checked our trac and it seems that while I was doing this port one of our developers had just done the same. Oh well.

In any case, I took the opportunity to clean up my Trac 0.11 hacking notes a little. These instructions completely remove the mental friction I always have to getting started doing any trac hacking.

I also updated the trac-hacks presence for these plug-ins: TracDupPlugin and TracKeywordsPlugin.

On this week's plane trip I finally took some time to port two of my trac plugins to 0.11 because work's migration to 0.11 seems blocked on them. I did...

Recently I've been looking at Buildbot again. It's been a while, bu it's nice to poke at it again. There's various things I want to do (like maybe finally finish my IRC praise/blame patch ?), but for starters I wanted to get some buildbots set up for pychecker, moap and morituri.

So that involved finding my old buildbot hacking notes so I can jumpstart a little better. And I started thinking about documentation.

I've always felt I'd like to write more documentation because it's a critical piece of success for a project. But at the same time, there's a simple problem when I want to write documentation: you need the mindset of a beginner to write good documentation. So it becomes a chicken-and-egg problem: in the beginning you don't know the project well enough to write the documentation; and as soon as you learn more about the project, you lose the capacity to ask the right questions the documentation needs to answer.

It is a special kind of skill to get into that mindset and be able to write from that perspective of a beginner. I don't think I have it.

But today I realized that I am now in the perfect situation when it comes to Buildbot. I know enough of how buildbot works conceptually from the past when I hacked on it, but it's been changed enough and I've forgotten enough that I actually need to go through the motions again to get my knowledge back. So, from a user point of view, I'm back to being a newbie.

So, I decided I'd take advantage of the situation and write some tutorials on how to use Buildbot. I've got two sections written up now and I'm ready to have people give me feedback.
The first tutorial focuses on the quickest 'this works' experience you can have, without touching the default configuration at all.
The second tutorial gives a quick tour of some of the features of buildbot by customizing the configuration, but without writing any code.

I plan to write further chunks that show how to set up a real build master for a project and integrate with the change notification, how to set up depending builds, and how to do some actual coding for specific build steps.

If you have some spare time and you're interested in continuous integration, you could do worse than provide me some feedback on these tutorials.

Recently I've been looking at Buildbot again. It's been a while, bu it's nice to poke at it again. There's various things I want to do (like maybe finally finish...

The last few weeks I spent most of my spare time learning a little Django and rewriting the old DAD (Digital Audio Database) code (which was written in PHP) to something a little more modern and malleable.

All these topics deserve separate posts, but for now I'm just content hooking up the various pieces of infrastructure for the overall project.

So, I was happy as a child when I was able to do a query for 'all tracks that contain a hidden track' (which I look up by doing level analysis on the files, and figuring out where there are sections with a reasonably large section of silence).

The full list for my collection is below, but it definately picked out the ones I expected, like 'All Apologies' (I can't count the times I was annoyed when a music player played it then shut up for 10 minutes), Lift To Experience's "Into the Storm", the Auteurs track, and the beautiful beautiful hidden track at the end of Placebo's first album.

Pretty soon this player will play these hidden tracks correctly, and DAD will allow you to rate tracks separately.

Here's the current list (some audio files are repeated because they contain more separate slices):

In [8]: for s in models.Slice.objects.filter(start__gt=0): print s.audiofile
...:
Alanis Morissette - You Oughta Know (Extended).ogg
Auteurs - Home Again.ogg
Andrew Dorff - Angel Puppets.ogg
At The Close Of Every Day - Lower World.ogg
Autour De Lucie - Chanson Sans Issue (Remix).ogg
Ash - Darkside Lightside.ogg
Astrid - Say What You Mean.ogg
And You Will Know Us By The Trail Of Dead - Sigh Your Children.ogg
And You Will Know Us By The Trail Of Dead - Source Tags And Codes.ogg
Better Than Ezra - Coyote.ogg
Ben Folds Five - Evaporated.ogg
Beck - Blackhole.ogg
Blur - To The End (La Comedie).ogg
Blur - Essex Dogs.ogg
Beck - Debra.ogg
Counting Crows - St Robinson In His Cadillac Dream.ogg
Chris Whitley - Ultraglide.ogg
Cowboy Junkies - Those Final Feet.ogg
Chocolate Genius - It's All Good.ogg
Coldplay - Everything's Not Lost.ogg
Drugstore - The Funeral (But Most of All).ogg
Damien Rice - Eskimo.ogg
Damien Rice - Eskimo.ogg
Deftones - Mx.ogg
Deftones - Mx.ogg
Denis Leary - Lock 'N Load.ogg
Drugstore - Flying Down To Rio.ogg
Ed Harcourt - Like Only Lovers Can.ogg
Elvis Presley - Rip It Up.ogg
Everclear - Like A California King.ogg
Fun Lovin' Criminals - Little Song.ogg
Green Day - FOD.ogg
Gabriel Rios - Badman.ogg
Geneva - Have You Seen The Horizon Lately.ogg
Grandaddy - Lawn And So On.ogg
Janet Jackson - Special.ogg
Kyuss - Spaceship Landing.ogg
Kyuss - Spaceship Landing.ogg
Krezip - Fine.ogg
Korn - My Gift To You.ogg
K's Choice - All.ogg
Lemonheads - The Jello Fund.ogg
Lemonheads - The Jello Fund.ogg
Lemonheads - The Jello Fund.ogg
Lemonheads - The Jello Fund.ogg
Lamb - Feela.ogg
Lamb - Just Is.ogg
Lo Fidelity Allstars - Dark Is Easy.ogg
Lift To Experience - Into The Storm.ogg
Magnapop - Voice Without A Sound.ogg
Monster Magnet - Vertigo.ogg
Mansun - Dark Mavis.ogg
Manic Street Preachers - Freedom Of Speech Won't Feed My Children.ogg
Metallica - The More I See.ogg
Nirvana - Something In The Way.ogg
Nemo - The Headphone Song.ogg
Nirvana - All Apologies.ogg
Offspring - Smash.ogg
Placebo - Swallow.ogg
Poe - Fly Away.ogg
Polar - Kill My Fears.ogg
Phoenix - Alphabetical.ogg
Patti Smith - Notes To The Future (live).ogg
Placebo - Burger Queen.ogg
Pearl Jam - All Those Yesterdays.ogg
Queens Of The Stone Age - A Song For The Deaf.ogg
Robbie Williams - Suprême.ogg
Radiohead - Motion Picture Soundtrack.ogg
Robbie Williams - Beyond The Sea.ogg
Starsailor - Coming Down.ogg
Sense Field - Haunted.ogg
Smashing Pumpkins - Daydream.ogg
Sunzoo Manley - Taxidriver.ogg
Sparklehorse - Babies On The Sun.ogg
Sonic Youth - Sweet Shine.ogg
Soulwax - Acapulco Gold.ogg
Therapy - Sister.ogg
Tool - Flood.ogg
Tool - Opiate.ogg
Throwing Muses - Shark.ogg
Tanya Donelly - The Shadow.ogg
Throwing Muses - Fever Few.ogg
Travis - Walking Down The Hill.ogg
Turin Brakes - The Optimist.ogg
Tracy Chapman - I'm Ready.ogg
Thrills - The Irish Keep Gate-crashing.ogg
U2 - All I Want Is You.ogg
U2 - The Wanderer.ogg
Teddy Thompson - Missing Children.ogg
Verve - Come On.ogg
Jesus And Mary Chain - Just Like Honey.ogg
Nits - Meisje Van 16.ogg
Whiskeytown - Bar Lights.ogg
Whipping Boy - Morning Rise-A Natural.ogg
Yeah Yeah Yeahs - Modern Romance.ogg
Beyoncé - Bonnie And Clyde '03.ogg
Sigur Rós - Untitled 4.ogg

The last few weeks I spent most of my spare time learning a little Django and rewriting the old DAD (Digital Audio Database) code (which was written in PHP) to...

I've always had this nagging feeling any time I write XML parsing code in Python that it just ought to be simpler.

Conceptually, I tend to always think that an XML document is really just a big nested python dictionary, and I should be able to walk it as such, or even better, as a composition of objects. That's probably a gross oversimplification and probably not correct according to spec or some corner use cases, but really, that's just how my brain works. Show me an XML document and I'll show you a big dictionary tree.

So I've always been disappointed with the experience of having to write actual XML parsing code. In Flumotion I think we've gone through three different XML parsers as well, depending on which developer was supposed to rewrite or add some piece of code that had to deal with XML config files.

So each time I have to write some XML parsing code again, I tend to go look if there's something better out there. This time I stumbled across a blog post that echoed my sentiments exactly. And it came with a solution: xml.etree.ElementTree. Finally ! A library that more or less maps to my mental model. Reading through the tutorial then writing the code I needed to parse my file took a lot less time than handrolling yet another set of functions to parse tags would have.

I'm posting this to increase the google juice of xml.etree.ElementTree. It simply doesn't show up when you google for python xml, and it should be hit number 1!

I've always had this nagging feeling any time I write XML parsing code in Python that it just ought to be simpler. Conceptually, I tend to always think that an...

Next step on this weekend's yakshave: a first implementation of moap vcs bisect!

The interface is lifted from git, obviously, since that's where most people will know the feature from.

I implemented it first with CVS, so I could fix this pychecker bug which was blocking Fedora from bumping the pychecker version from 0.8.17 (3 years old) to 0.8.18. And sure enough, it picked out the commit I broke.

While implementing and while dealing with CVS's idea of how it stores CVS revisions and dates and so on, I googled and was amused to find this first hit on google for the words cvs and bisect. Clever Andy! And he cleverly sidestepped the problem I wrestled with by making the user specify two dates at the start instead of trying to figure it out from the checkout. And all in lisp too!

Then, to test that my VCS interface was sane, I implemented it for Subversion as well. That took about 15 minutes, since Subversion is much more sane than CVS. I tried the following command on a flumotion checkout:

moap vcs bisect reset; moap vcs bisect start; svn up; moap vcs bisect good; svn up -r 3000; moap vcs bisect bad; MOAP_DEBUG=4 moap vcs bisect run ./test.sh


With test.sh containing

test -e flumotion/component/consumers/gdp/gdp.py


(In other words, look for the commit that added this file.)

Sure enough, it picked out this commit:

[moap-trunk] [thomas@ana flumotion]$ moap vcs bisect diff
Index: /home/thomas/tmp/flumotion/configure.ac
===================================================================
--- /home/thomas/tmp/flumotion/configure.ac (revision 6909)
+++ /home/thomas/tmp/flumotion/configure.ac (revision 6908)
@@ -212,7 +212,6 @@
flumotion/component/combiners/switch/Makefile
flumotion/component/consumers/Makefile
flumotion/component/consumers/disker/Makefile
-flumotion/component/consumers/gdp/Makefile
flumotion/component/consumers/httpstreamer/Makefile
flumotion/component/consumers/preview/Makefile
flumotion/component/consumers/shout2/Makefile
Index: /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am
===================================================================
--- /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6909)
+++ /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6908)
@@ -11,7 +11,6 @@



SUBDIRS = \
disker \
- gdp \
httpstreamer \
preview \
shout2
Index: /home/thomas/tmp/flumotion/ChangeLog
===================================================================
--- /home/thomas/tmp/flumotion/ChangeLog (revision 6909)
+++ /home/thomas/tmp/flumotion/ChangeLog (revision 6908)
@@ -1,16 +1,5 @@
2008-06-20 Thomas Vander Stichele


- * configure.ac:
- * flumotion/component/consumers/Makefile.am:
- * flumotion/component/consumers/gdp (added):
- * flumotion/component/consumers/gdp/gdp.py (added):
- * flumotion/component/consumers/gdp/__init__.py (added):
- * flumotion/component/consumers/gdp/Makefile.am (added):
- * flumotion/component/consumers/gdp/gdp.xml (added):
- Add a GDP consumer.
-
-2008-06-20 Thomas Vander Stichele -
* flumotion/component/producers/gdp/gdp.py:
Add error for http://bugzilla.gnome.org/show_bug.cgi?id=532364

So, the feature is ready for testing. It could use some more documenting, and some additional goodies like accepting arguments to moap vcs bisect start for example.

Feedback appreciated!

Next step on this weekend's yakshave: a first implementation of moap vcs bisect! The interface is lifted from git, obviously, since that's where most people will know the feature from....