Present Perfect

Six years ago, before I moved to Barcelona, my digital music collection was well-organized and simple.

I had my CD's. And I had a copy of all of them, in Ogg, on my Dave/Dina box. I had very few downloaded tracks, and I didn't really listen to those much. All my music on Dave/Dina was tracked in DAD, a project I did with my former housemate Kristof.

Life was great. Every new CD was ripped directly in Dave/Dina, imported in DAD, and from there it could be rated. So each track was immediately available for the dynamic playlists DAD generated. Those playlists were then played directly on the Dave/Dina box attached to the living room stereo, my desktop, or the kitchen or bathroom computers (small Compaq IA1 machines from the Golden Bubble days).

Today, it's all a mess.

I have music (ripped, bought online, downloaded, or copied) on the following devices:

• My elisa machine (which holds all the old Dave/Dina content)
• My home desktop
• My laptop
• My work desktop
• My Nokia N800
• My Cowon A3 media player
• Kristien's iPod

I haven't ripped a single CD in the last 4 years since I stopped working on Dave/Dina, so these days I also listen to CD's on either our small portable stereo or the PS3. I have 200+ CD's still waiting to be ripped.

So my music listening has become erratic, listening to either the old 'good' playlist from Dave/Dina that hasn't changed for the last 6 years, and, while good, is getting stale; or whatever I recently ended up downloading, for a specific album, but random play is terrible when going through those directories, and of course each album is on some different machine or device.

The last two years, I've grown more and more annoyed at this situation. So one of my goals for 2009 was to finally *do* something about it. I realize that music is one of the things I love most in life, and my life would be better with the music I buy and find in it as soon as I have it. So what little hacking time I have left before real life begins (you know, kids and stuff) is going to go in code that is going to make my music experience better.

Having goals is a good way to direct your hacking. I've come up with five major projects I need to work on to get my music where I want it to be. All of these are projects I've had thoughts on in the past, but never really gotten to. Over the last few years a lot of new ideas and technologies have arrived that would help a lot now however.

1. Re-rip all my CD's in a lossless format, with perfect quality, according to a certain website's standard
2. Find a replacement for DAD, or make one. It should be able to track rips, tracks, different encodings of the same recording, different versions of tracks, parts of tracks (hidden tracks for example), and different collections across devices that it should be able to synchronize. Think 'put 10 GB of the best songs on my N800', and each time I'd connect my N800 it would automatically add new ones and remove old ones.
3. Improve on the rating system DAD used to have, reusing ideas from a project at the radio stations all those years ago. Have a bunch of fuzzy parameters on each track which would allow much richer controls for song selection. Possibly turn it into a collaborative website if it makes sense. This hinges on uniquely identifying each track, for which acoustic fingerprinting would be a good solution. Basically, I want to use the power of the web and the music lovers to improve song selection. last.fm and pandora are going in the right direction, but don't really satisfy me.
4. Write a player that does the automatic mixing the way Dave/Dina used to, or better.
5. Make a LEGO Mindstorms robot to automatically rip all my CD's again

I've been tackling each of these separately, which turned out to actually be a good thing. Each time I'm stuck on one of them, I can work on any of the others. For example, I've been stuck on (4) for a long time, waiting for Edward to fix some bugs in gnonlin, so I switched to (1), writing code to parse .CUE files, implement CDDB disc id calculation, and AccurateRip verification of ripped images.
I also waited on a friend who I worked with at said radio station to confirm that he doesn't have any backups either of the database for (3).

Since this is what I'll be hacking on in my spare time in the forseeable future, I'll probably blog about the different pieces as well. I'll start with some more technical information on (1), the ripping part, which I've been working on the last month, in a separate post.

But man, I look forward to ripping my CD's from the last four years and actually listening to those songs regularly, rather than once in a while.

Isn't it amazing how our parents actually had to get up from their desk, go over to the turntable, and actually flip an LP over if they wanted to hear something else for 20 minutes ?

Six years ago, before I moved to Barcelona, my digital music collection was well-organized and simple. I had my CD's. And I had a copy of all of them, in...

Is it just me, or does struct.unpack seem mostly useless to unpack any real-world binary data ? Real-world binary data usually has a fixed amount of bytes for each value to read out of a binary file. struct.unpack does not allow you to specify how many bytes to read for each value to unpack; rather, it has format specifiers that map to the C type declarations.

The only way I found that works to read, for example, sets of 32 bit signed integers, is to read them with one call per variable, and slicing the data, like so:

self.trackCount = struct.unpack("B", data[0])[0]
self.discId1 = "%08x" % struct.unpack("

NOTE: for some reason WordPress does not allow me to have the first <L format string to show up as just that; instead it corrects it to '<I ' (different letter, and a space). FML.

Is it just me, or does struct.unpack seem mostly useless to unpack any real-world binary data ? Real-world binary data usually has a fixed amount of bytes for each value...

Here's an innocent little line of code that caused me to lose an hour of debugging and svn bisecting time yesterday:

gobject.timeout_add(0L, self._pipeline.set_state, gst.STATE_PLAYING)


That change was buried inbetween some other cosmetic changes, and caused the UI of my application to be frozen. It took me a while before I nailed it down to this line, and a little bit more to figure out exactly why.

The goal was to set the state of the pipeline to PLAYING 'as soon as possible', from the main thread.

Of course the line of code does something subtly different. timeout_add schedules a call, so that part's fine. But, the call gets rescheduled automatically unless the call returns False. Since that returns a GST_STATE_CHANGE enum, as long as the state change doesn't fail, it keeps getting rescheduled, over and over. So, no time for the UI to update.

I figure this is something I"m bound to run into again, so a little blog will increase my chances of figuring it out again next time.

Here's an innocent little line of code that caused me to lose an hour of debugging and svn bisecting time yesterday: gobject.timeout_add(0L, self._pipeline.set_state, gst.STATE_PLAYING) That change was buried inbetween some...

Two weeks ago for some reason I didn't even know that Joss Whedon had a new TV series out.

I just finished watching Dollhouse, Episode 6 just now. It was all kinds of awesome. I have purposefully not read too much about the series because it seems Joss is getting a lot of backlash. And now I'm looking for reviews of episode 6 and it seems this is the episode where Joss really puts things into motion.

Well, it certainly clicked with me. The first 5 episodes are slow buildups, introducing new elements all over the places, and having little of the trademark JW dialogue but enough of the trademark JW setups and subtle hints. And then bam - 6 delivers in spades.

Well ... I'm ...
sure I'm in need of some serious moral spankitude, but ...
uhm... guess who's not qualified to be a rabbi.

Man, I love the internet. Thanks for letting me know through Planet Twisted, Itamar!

Two weeks ago for some reason I didn't even know that Joss Whedon had a new TV series out. I just finished watching Dollhouse, Episode 6 just now. It was...

Yesterday, I was working some more on my jukebox code. It is now in the state where I can actually have it play a playlist and do the mixing properly. So I'm finally into dog food mode if you will, or in the mode where I can actually enjoy the result of the code I've written.

But I still had random failures that I couldn't make sense of that I thought were due to gnonlin. I was getting random backtraces on startup when setting up my gnl objects, like so:


Traceback (most recent call last):
File "examples/jukebox.py", line 123, in
main()
File "examples/jukebox.py", line 115, in main
main = Main(loop, tracks, playlist)
File "examples/jukebox.py", line 67, in __init__
self._jukebox.add_track(path, self._tracks[path][0])
File "/home/thomas/svn/src/dad/trunk/dad/gstreamer/jukebox.py", line 81, in add_track
self._process()
File "/home/thomas/svn/src/dad/trunk/dad/gstreamer/jukebox.py", line 137, in _process
operation.props.duration = mix.duration
TypeError: could not convert argument to correct param type


I had been noticing them before but always thought "I'll get to that later." But yesterday it was getting on my nerves.

The problem was, there was nothing wrong with the duration I was setting. 'duration' is an INT64 property on a gnloperation, and the duration I was setting was a normal long. I spent hours sprinkling debugging code and diving deeper into the stack to figure out where it was going wrong, but couldn't turn up anything.

I did see that in pygobject's gobject/pygtype.c code, it was checking PyErr_Occurred, and just resetting the error and the GValue. I added a PyErr_Print to see why the conversion was failing, and it told me:

TypeError: can't convert negative long to unsigned


This made even less sense - I wasn't setting a negative long, and I wasn't setting it to an unsigned property either. It's too bad the pygtype code wasn't bubbling up this clear error up, instead obfuscating it with the generic error, but it still didn't explain my bug.

We had invited friends over for dinner, so in the afternoon I was forced to take my mind off the keyboard, but as you all know that doesn't mean your mind is off the problem. I'm sure you all know that feeling where you have lots of different pieces of the puzzle lying in front of your mind's eye, and none of your theories made sense and you still see a bunch of different pieces.

Dinner was prepared, friends came, a good time was had by all, and my mind was off the problem. My only distraction was listening to the mixing points at the end of each song. And then when they left around 1 AM, Kristien and I sat on the couch, were watching some TV, it suddenly hit me. A classic bug pitfall.

What if the error was actually triggered before the line of code I was looking at ?

I'm sure you also all know that feeling where suddenly, a different way of looking at the problem makes all the pieces of the puzzle fall into place. It's probably the most gratifying moment when doing bug hunting - after hours of frustrating undirected prodding, suddenly you get that one theory that explains both the problem and the lack of result in the prodding.

That's when, instead of flying to the computer and trying it out, I prefer to savor the moment a bit. Because there's still the possibility that I might be wrong, and just uncover another layer of fur on the yak you're shaving. But no, everything seemed to add up, and I knew in my mind exactly the one line of code I needed to add to either prove or disprove your theory. And I also knew that this one line of code was the last thing I needed to try that day. Nothing else. Before, every 'just 2 more minutes' I asked for where valiant efforts borne from bug hunting optimism. Now, I see the problem crystal clear.

If the one line proves the theory, I can go to bed safely, be happy that I found the problem, that my stack ws unwound, and I was holding my juggling balls of my mental state in my hands. (Of course, if it disproved the theory, I was in for a night of bad sleep, going over the problem again and again in my head).

So, while Kristien got ready for bed, I moved back to the keyboard, and before the call that I thought was failing in the bindings, added another PyErr_Occurred check, and ran it. And there it went - and it confirmed that the error was actually generated before the line I was looking at, and coming from somewhere else.

So, off to bed, knowing that in the morning I knew exactly where to look.

After some futzing around, I realized that the call before was setting a negative value on a UINT64 property, and it was not erroring out. I was able to reproduce it from a python shell:


>>> import gst
>>> e = gst.element_factory_make('gnlsource')
>>> e.props.start = -1L
>>> import gobject
Traceback (most recent call last):
File "", line 1, in
TypeError: can't convert negative long to unsigned


And after digging through the history of that code, I found the revision where a change was made that removed the error checking for this particular case.

I filed a bug, and that was it. Hours of frustration boiling down to a simple commit labeled 'guint64 property fix'. The patch is trivial, so I'm sure they will do a better job at it than I would.

Of course, the actual bug is still in my code - I need to figure out why I'm setting a negative start value.

But again I am reminded of a fundamental bug fixing rule that I broke again this time: before trying to fix the bug where it appears to be, first make sure that the bug is where it appears to be. Sometimes the bug is created before where it is detected, and it's a lot easier to verify that first before anything else.

Yesterday, I was working some more on my jukebox code. It is now in the state where I can actually have it play a playlist and do the mixing properly....