Present Perfect

The short version is 'I can finally live with the favicons in my delicious extension thanks to a script I wrote and after upgrading to beta 2.1.005.

The long version is, well, longer.

This morning's yak shave story begins a bit over 5 years ago. That was the time when I was inbetween jobs, and already had my laptop for the new job, which was going to evolve into being 'employee number 1 at Fluendo'. I had decided I wanted to try the 'living life from only one computer' approach to working and hacking. Some people do this instinctively, some people don't choose and end up with crap on all their machines, but me, I like inventing processes where there is no need to and sticking to it.

So, for a long time, I mostly worked only from my laptop, and this served me fine in a lot of cases. My main motivations were 'I want to be able to work anywhere I am' and 'If I'm used to a laptop I will always be able to be productive'. The first proved very useful whenever I was travelling, and the second is what made me not want or need a second screen at work when all developers around me were making it a political point.

The biggest problem, however, is that often the work you do really does need the higher CPU, the more RAM, or the bigger disk that your average desktop easily has over your laptop as soon as your laptop becomes a year old. Also, your laptop wears out a lot faster if you have it running all the time. So, at some point, after knowing for sure that I am able to work constructively on my laptop, I decided that I should also make it possible to work from a home desktop and a work desktop.

Make no mistake, to me this is an incredible paradigm shift. While on its own there is no real problem to habitually work on three computers, to someone like me it is incredibly annoying to have my flow broken by various small details that made up my previous 'work environment' on my laptop that I have not yet successfully migrated to the other machines. If you have only one machine, adapting it to your workflow continuously is easy. If you have three, you constantly bump into the things you forgot to synchronize across all three at the moment where you least want it.

From day 1 the plan was thus to make sure that everything that is important to me would be synchronized across all three machines as often as I needed it. Additionally, I would make sure not to start using any new tools that weren't able to work this way, and move my current set of applications in a direction where they would be easy to sync. It would also be important to make sure I can work offline as much as possible - I spend time in travel and on planes that I want to put to good use.

So, over the years, I've:
- created an svn repository for all my private notes, documents, tests, and code
- created a public svn repository for the same stuff that can be public
- picked a Getting Things Done application called yagtdthat uses a text backend so that I can put the data in subversion to keep it synced across all machines but still work offline
- tried to move my contacts to an LDAP server so that Evolution can use it (which I still consider a 'fail' because Evolution's LDAP backend is still crashy, hangy, and practically unusable for everyday use)
- moved my calendar to various Google Calendars (up to having a public calendar called 'Where is Thomas' so friends and collagues know what country to expect me in)
- actually paid for an application on my phone - the wonderful GooSync Zaheer recommended me which allows me to synchronize all my Google calendars to my phone, in two directions
- moved all of my bookmarks to Delicious for obvious reasons
- used the Delicious firefox extension to replace my bookmarks folder toolbar - Instead, I use the 'Favorite tags' view and use delicious tags like '@daily' (borrowing the at symbol from GTD's contexts) to have quick access to all the sites I use on a daily basis. This works great because as soon as you drag a bookmark to one of those favorite tags on the bookmark bar, it is instantly available on all three computers where I have this set up.
- put all of my passwords into a revelation database that I put in my subversion repo (not 100% ideal, since managing conflicts is impossible, but hey...)
- wrote two scripts, called 'start' and 'stop', which I execute when I start or stop working on a machine, and which sync stuff from subversion and commit stuff to subversion that is important to my workflow (my GTD list, the password database, ...)

None of this is perfect or ideal, but most of it is about as good as it gets for my usual workflow. Lately, I've started focusing on improving these individual elements.

For example, contacts. I really wanted all my contacts to be available in Evolution on all my computers, and on my phone, syncing every way. And bonus points for anything that would finally let me know when my friends have their birthday, a pet peeve for like the last 10 years.

I read some Evolution code and figured out a lot of reasons why the LDAP code is bad, but most of my inquiries (over IRC, on the mailing list, in bugzilla, or poking some friendly hackers (who will remain anonymous)) gave me no idea about why the code was the way it was, who is responsible for maintaining it, and if my ideas for improving were actually any good. I did get a lot of feedback from people agreeing that it sucked or wanting me to work on their pet feature for LDAP support though.

Similarly, calendars. Around the same point, I tried the Google Calendar backend for Evolution as well, because I was getting fed up with not being able to use Evolution for this, and what seemed to be touted as a major feature for a new GNOME release didn't actually work very well. I couldn't figure out at all if it actually supported more calendars than just your primary Google calendar (which I hardly use - I have 10 others), and it didn't work very well at all.

I found a java daemon that is supposed to synchronize your Google calendar with local .ics files which should then work for Evolution, but in practice works very badly for two-way sync and Evolution doesn't seem to notice the file changing anyway. All fixable, but fixing hacky solutions isn't very motivating. And as I mentioned before, while I usually set myself up with a process then stick to it with an almost religious fervor, as I had done for the last years with Evolution - for the first time I was seriously considering completely dropping Evolution.

I saw that Conduit seemed to have some Google support, but after talking to John it seems that the support for Google Calendar got broken and removed in a recent refactoring (I abstained from noting that, by definition, if you broke or removed code then it isn't a refactoring), so it wouldn't work at the moment.

Moving back to contacts, it came to the point where I deactivated my LDAP backends, and tried to go back to using the local one and figuring out other ways to sync. I noticed that GooSync offered the option to sync to Google Contacts, which I had never used before, so I tried that. I found that Conduit did have support for Google Contacts, but I could never get Conduit to sync more than 20 contacts. If I were to hack on anything to fix it, it would probably be Conduit, because I have a feeling it wouldn't be too hard to get it fixed. While my Evolution contacts were being updated on the fly through conduit, I could never get them all. And after a few 3-way syncs I ended up with a bunch of triplicate contacts in my Google Contacts.

But at the same time I noticed something utterly astounding that I would never expect from any Google product. And sure, it's a detail. But somehow I have this opinion of Google that they are so good at getting some of the small details right in their applications. I was so surprised to find out that Google Contacts does not allow you to store birthdays. It's just not there. Even my phone lets me do that and shows the birthdays on its calendar. I was sure it was me who was unable to figure out where they store it, until I found a whole bunch of posts from users sharing the same disbelief. Amazing.

Here's a company with some of the smartest engineers in the world, who've made a successful web mail client, a successful calendaring application, and some technology for storing and using contacts across all of their applications. Yet no one in that company, with all their 20% times, thought it would be a good idea to have birthdays in contacts and have Google Calendar show you a calendar of birthdays automatically all the time ?

I left the whole 'synchronizing contacts' plan on the top shelf right now, while I try and figure out whether I'm going to:
- ignore birthdays and clean up my Google Contacts
- figure out if I can write enough code to make Conduit sync 3-way between my phone, all my Evolutions, and a collection of vcard files in my subversion repository.

So, after that disappointment, there was one more really annoying problem in my 'synchronize everything' approach - since upgrading to Firefox 3 favourite icons have been a huge mess. I had about a 30% chance of getting correct favicons, with the other 70% split between 'not getting a favicon' or 'getting the wrong one, forever'. The delicious extension was even worse at it, usually not able to show more than 5 of them correctly in my toolbar. Seems like a very small detail, but when you have a lot of bookmarks the icons really help in finding the right one consistently.

After various betas and updates of the extension that never actually fixed the problem, a few posts to the yahoo group, little more than promises from the developers that 'things are changing', this week they launched a new beta that takes the first step in a new direction. It wasn't clear to me that it would actually solve the problems, but the notes said that they moved from an RDF file to an sqlite database for storing the synchronized bookmarks. And A developer replied that this new beta should fix my particular problem.

So I updated, tried, and at first nothing changed. I stopped Firefox, opened up places.sqlite and dropped the moz_favicons table. Restarted Firefox, and it refetched a bunch of favicons (some of them wrong - my homepage got the GNOME foot for example), and the delicious extension did get a bunch more favicons, but not all of them by a long shot.

If the extension had been open source I would for sure already have taken the time to look into the code, and fix the bugs - that's how much this irritates me.

But, with the information that now everything was stored in an sqlite backend, I decided to look for the database and see what was wrong. And after 2 hours of hacking (double the time it took to write this overly long blog post) I have a simple python script that adds or fixes the favicons in the ybookmarks.sqlite database.

It's probably dangerous, eats your babies, breaks your database (so back it up first), and takes ages to run as it downloads each page in your delicious account, but for the first time in two years I can finally see that last.fm favicon in my @daily bookmarks folder!

It's Saturday morning going onto 13:15, and finally I can start doing what I was actually planning to do today, and in no way related with the final straw that caused me to fix my favicons.

The short version is 'I can finally live with the favicons in my delicious extension thanks to a script I wrote and after upgrading to beta 2.1.005. The long version...

One of my tasks for Nerd Night was to set up OpenID authentication on my trac so that the other nerds can log in and change the wiki.

Now, there are two OpenID plugins for trac - an unmaintained one for Trac 0.10, and a maintained one for Trac 0.11

In the past, upgrading Trac minor versions has always been a bit of a pain, especially given the number of additional plugins I usually have installed. But I am motivated, because I want to make the same jump on the office Tracs soon, so that we can configure our workflow - one of the new features of Trac 0.11.

After a half-assed attempt at trying the old plugin last week, I decided to try and upgrade to Trac 0.11 for real, and finish this task. And I decided to do it in a more methodological way than the usual try-and-die. Or, in other words - it's yak shaving time, baby.

I started by updating my draft hacking notes to 0.11, and making them work with an uninstalled Trac.

Then, on my F-9 desktop, I downloaded source of Trac 0.11.1 and the authopenid plugin. I followed my hacking instructions, updating them as I went, and ran tracd with the authopenid plugin uninstalled without problems.

But our hosted server still runs Fedora Core 4 (Wiebe, is our new server ordered yet ?) so I had to build packages for that version of Fedora. So, I created a new tao directory in my repository of packages, for packages I will be only offering on my own website. I forked spec files for Trac, python-pygments, python-textile, and python-genshi. I updated the Trac one to 0.11.1, and the others I backported to work on FC4.

Then, I created all packages using mach, and encountered a small bug in it that Julien introduced when he added a fix for the problem with different people in the mach group trying to build packages. I worked around that with Another Small Hack, then rebuilt the packages.

I tried out these new packages inside a chroot of mach, by running:

mach -r f4e setup
mach -r f4e yum -y install trac
mach -r f4e chroot

And I again followed my HACKING instructions, but this time the installed ones.

I re-used my hackish python-openid package from last week, and tried to build an rpm of the OpenID plugin. Using

python setup.py bdist_rpm

failed however - for some reason, the templates and css files do not get packaged. Same when running sdist instead of bdist_rpm. So I settled for bdist_egg, and copied the .egg file in the plugins/ directory.

Now, trying to log in, I got a traceback. Googling for it brought me to this trac ticket.

A

yum install python-sqlite2

solved it, though I have no idea why! The traceback is completely obtuse, and I don't see how installing something would fix it, but hey... Now I could confirm that the OpenID plugin for authentication worked.

Now, final step, do the actual Trac upgrade on my hosted server!

First of all, take a backup. Luckily, I take daily backups with some script, so it's just a matter of executing the script once more. Upgrade the trac RPM. Do a trac-admin (path) upgrade on all my trac installations. Try them out. So far so good - everything seems to be more or less working, bar some macros that I will need to rewrite.

Then, configure the authopenid plugin. It failed with a traceback that said that the table 'oid_nonces' had 2 colums, and the query was trying to insert 3 values. So I whipped out sqlite3 and compared the schemas between my mach Fedora 4 chroot and my online server. I am assuming that the old OpenID trac plugin created two tables with the exact same name (oid_nonces and oid_associations), but with different schemas. And there is nothing in the authopenid plugin that checks if the schemas are right. So, I dropped the tables, imported the schemas from the chroot sqlite db (handy feature of sqlite3, the .schema command), and bam! Login finally worked!

Not the easiest 4 hour task, but finally completed! On to the next Nerd Night.

One of my tasks for Nerd Night was to set up OpenID authentication on my trac so that the other nerds can log in and change the wiki. Now, there...

When we moved into the new apartment, one of the things I definitely wanted to maintain the nerd quotient in the place was a scrolling LED marquee in the kitchen pointing to the living room:

I got the idea when we bought a marquee such as this one as the perfect gift for Peter's 30th birthday some time ago. I remember racking my brain trying to come up with a good gift for us to give him, and at some point it hit me that one of these things would be awesome for him.

We didn't just give it to him either - he had a birthday party at the Charlatan, a bar in Gent, and we installed the marquee there as part of the DJ booth, putting up increasingly weird messages until he finally realized that it was a present for him. Good times!

Anyway, having that thing at home for a few days and playing with it convinced me I should get one for myself for the new apartment and use it for news updates. We installed the thing in the ceiling spanner and, apart from some tedious remote control entering of texts for some occasions, I haven't used it much since. I tried once in a while to control it from the computer but I never really got it to work.

Until last week I had some spare time and decided to figure out what the problem is with the computer control. After a lot of twiddling, I realized that the serial-to-jack cable had a 6P6C jack, instead of the usual 6P4C for RJ-11 use. I hadn't actually ever seen a cable like that, someone suggested it might have been used for ISDN connections, but I have no idea. Hooking it up directly to a computer made the software work under VMWare and Wine to control the device. After that, controlling it directly from Linux was easy.

Of course, I want to use the marquee in its place in the kitchen, not by my computer. After all, that's why I invested in the Abitana network I installed. I went to my favourite electronics store in Barcelona, where they had to custom-make the cable because apparently it's hard to get stock cable like this.

Took it back home, tried it, still didn't work. After inspecting the cable more closely it seems the guy had put one end on the wrong way. Luckily, one end is supposed to go in the 8P8C RJ-45 wall socket, so I crimped a standard connector on that end, and bingo! Finally have the LED marquee in place and controlled from Linux from my computer!

Next step, to write some Twisted-using software to implement the protocol, and write some code to get RSS feeds and display some news! I considered using LCDProc (which I've used on my Dave/Dina box), but it looks like that's mostly geared towards small LCD displays with multiple lines and characters, and some control buttons.

I need to figure out how I am going to prioritize incoming information (RSS feeds, buildbot status, nagios alerts, incoming mail, ...) and create a message queue out of those spread across the 26 pages the marquee offers. If you know any software doing this sort of thing, feel free to comment!

When we moved into the new apartment, one of the things I definitely wanted to maintain the nerd quotient in the place was a scrolling LED marquee in the kitchen...

I was wondering why Arek was arriving later than usual to work, to the point of being late at his own developer meeting where just a week before he instituted a rule that the last person in the meeting is to buy the rest of the team beer on Friday. This week he told me why, and yesterday he blogged about it: an implementation of the Vorbis decoder that works in Flash Player (albeit the newer version 10). It's based on the Jorbis code which we took for Cortado, and it looks like it's more than fast enough too.

I'm sure he wouldn't mind you buying him beer for this so he can make good on his promise to the rest of our team. And he probably accepts pre-orders for a Theora decoder in beer too :)

Congratulations Arek on the release!

I was wondering why Arek was arriving later than usual to work, to the point of being late at his own developer meeting where just a week before he instituted...

Arrived last night, IBC starts on Friday and I'm setting the booth up with Pedro today.

Every time I have a hotel reservation for work for some reason the hotel always thinks the room's not paid for yet and ask me for payment. It's a curse.

On the plane I had a craving to go to the cinema later tonight, and as fate would have it the hotel is right across the street from a cinema. So the choice between a movie and a visit to the Red Light District was easy - I went to see Wanted. Mindless action fun, nicely shot, and ultimately forgettable.

Today, waiting for our cases of stuff to arrive. I'm sure they'll show up around 17.00, an hour before the IBC convention center closes.

In a cruel twist of fate, we have the sewer pipe of Akamai's booth that ends up in our booth and goes in a hole in the floor, so we have one floor tile sticking out. That's where we were planning to put our servers. So it goes.

At IBC we'll be demoing our freshly available commercial version of our Flumotion streaming server (we also have our new website online). A big thanks to Johan Dahlin, who is the only person in the world to have been hired by our company twice, and to Murray Cumming, who wrote our new Flumotion manual, which will be available online for the free version soon.

Maybe we should show some Dirac streaming too, since the BBC, David and Christian seem to be here.

Arrived last night, IBC starts on Friday and I'm setting the booth up with Pedro today. Every time I have a hotel reservation for work for some reason the hotel...