Present Perfect

Drives fail.

Somehow they fail more often around me.

I used to swear by software RAID-1. I loved it because if either disk fails, the other disk still contains all the data and you can get to it and copy it off and so on.

Except that in reality other things get in the way when something fails. Drives may fail at the same time (because of a bad power supply, a power surge, a computer failure, ...). A particular low point was the time when my home server's RAID had a failing disk (which had all my backups on it), and I took out the drive, and added in a new drive to start copying, with both drives hanging out a little outside of the computer case, and at precisely that point a motherboard box dropped out of the shelves two meters higher, and landed with one of its points right on top of the good RAID drive, breaking it. I lost all my backups.

So I learned the hard way that most problems with RAID happen precisely at the time you need to phsyically manipulate the system when one of the drives fail.

Ever since I was wondering how I can do my storage and backup strategy better. I have some other ideas on how I want to back up my stuff from four computers in two different locations (three if you count 'laptop' as one of them) depending on what types of files they are (my pictures I need to make sure I don't lose, while most music I can re-rip if I really have to), which should go in a different post (and if you know of any good descriptions of home backup approaches for all these files we seem to collect, please share!).

But it's clear that part of the solution should involve storing files across more than one computer, preferably in a transparant way. I thought about setting up one drive in one machine, then doing a nightly rsync or dirvish snapshot of that drive, but with lots of big media files moving around it might not be the best solution.

So I was excited when I came across drbd, which implements a distributed file system that mirrors to a secondary disk over the network.

I got two new 2TB drives last week, installed them in my media server and home server (which took quite some fiddling, including blowing out a power supply which exploded in a green flash, sigh), and read through the pretty clear and decent docs, and after 50 hours of syncing empty disks, I now have this:

Sweet! Now to actually copy some files and run some failover tests before trusting my important files to it...

Drives fail. Somehow they fail more often around me. I used to swear by software RAID-1. I loved it because if either disk fails, the other disk still contains all...

Today is an exciting day at the office!

Jean-Paul Calderone (exarkun from Twisted arrived in our office this morning to give our development team an in-depth training on Twisted.

For now, the schedule is two days of Twisted training, one day of code sprinting on Twisted, and two days of consulting on our platform and various issues and projects we have.

For us it's exciting to get a training from one of the top hackers in Twisted, and I hope it is exciting for him to see a commercially successful use of the project he worked on.

But mainly I'm looking forward to a bunch of days of high-level technology talk.

Time to get started!

Today is an exciting day at the office! Jean-Paul Calderone (exarkun from Twisted arrived in our office this morning to give our development team an in-depth training on Twisted. For...

As we are deluding ourselves here into thinking it's snowing in Barcelona, I thought it appropriate to post some videos from the past few snowboarding trips.

Coincidentally, this is my first foray into the HTML5 video world - more on that later.

Let's start with my favorite, the one where I show off how years of gymnastics in my youth help me keep my body in one piece:

(Also notice the cool new orange snowboard pants that I settled on. Snowboard fashion was really boring this year, mostly grey and black only, with some ugly flashy colours as exceptions. I leave it to you to judge whether orange is one of them).

We spent eight full days in Tignes, France, with only about three days of sunny weather, and the rest filled with clouds and snow.

My goal this year was to learn how to do a 180. With the help of an instructor, that's exactly what I did! Here's an admittedly simple one - all the good ones are not caught on video.

Here's a more aggressive one with a bad ending:

A few weeks before our snowboard trip, we also had a business planning weekend which included one day of skiing. Xavier risked life and limb following me around with his iPhone to record this. It's not the most exciting descent in the world, and he ended up missing my one fall in it, but I was surprised to see how short the whole descent really is if you don't take any time to stop!

And here are Xavi and me relaxing over cheese fondue and raclette the day before the skiing:

Some notes about the HTML5 video part:

• there is an enormous difference in colour between playing Ogg in Firefox, MP4 in Safari, and MP4 in Quicktime, on the same MacBook. My pants range from a soft orange to a bright red. Something is obviously up!
• To learn about HTML5, I started with Dive into HTML5 Video, then learned about Video for Everybody, some web code that handles all of the stuff I don't know how to do for me and just makes sure the video can play on Firefox/Chrome/Safari/iPhone/...
• Then I looked for WordPress integration, and found a plugin with a long name that implemented most of Video for Everybody. I modified it a little to do something more sensible for the poster image in case it's external, and to accept .mp4 as an extension instead of .m4v (which is not suggested by Dive into HTML)
• I configured our transcoding platform to generate the three types of output file needed to support HTML5: the thumbnail, Ogg/Theora/Vorbis, and .mp4 with H264 and AAC.

The embedded video should work fine in Firefox/Safari/Chrome/iPhone/Opera (except in Aitor's "I plug mplayer into Opera" case), and work fine in Explorer too where it falls back to Flash.

I couldn't get this to work in Android. 2.0 is rumoured to support the video tag, but so far no dice, and I couldn't find a single HTML5 video page online that the Android phones over here can play. If you can see these videos embedded in Android, or know what I should to fix them, please do let me know!

As we are deluding ourselves here into thinking it's snowing in Barcelona, I thought it appropriate to post some videos from the past few snowboarding trips. Coincidentally, this is my...

One of the things I really love about my N900 is the ease with which I could share photos. You take the picture, click a few buttons, and there it is, your photo on flickr on Facebook. I'm sure other devices offer a similar experience, but this really is the first time I've been able to appreciate

Since a few weeks my Facebook sharing has stopped working. At first it only seemed like a missing icon and broken config. But I had a really really hard time to figure out the problem, much harder than it should have been for a mostly open platform. Of course, for some reason this sharing part is closed, which doesn't make much sense at all. What secrets can Nokia possibly have invested in some code that pushes photos to flickr or Facebook.

So, as part of the debugging process, over various weeks, I've seen and done the following:

• When sharing through the Facebook account, the little 'I'm sharing' icon that pops up in the status area went away almost immediately, compared to sharing through Flickr
• Sharing stores outgoing stuff in $HOME/.sharing/outbox - files were piling up there
• At first I thought I had some login details problems, so I went to Settings > Sharing accounts and clicked on the Facebook account. That didn't do anything at all, no dialog popped up. Since the Delete button is in that popup, I also couldn't use the UI to recreate my Facebook sharing account.
• I started learning about where the N900 stores account info, following some dubious posts until I found out about signond, a daemon running as root that stores login information. This daemon uses the file $HOME/.signon/user_db.xml (You get no points at all for correctly guessing whether this file contains XML data, but please do go look for yourself!)
• I moved this file around to force recreation of data, which seemed to work after a reboot, since my Flickr data was now gone, as well as the Facebook one. I was able to click Flickr in Settings > Sharing accounts, but the Facebook still didn't pop up a dialog. At least this narrows down the problem I should solve for now: I can't even configure the service from scratch even when deleted, so let's focus on that first.
• I reinstalled the package that contained the facebook sharing plugin. This restored the icon, but the popup still didn't work.
• Usually I would just strace the relevant binary, but I considered the friction too high at first since there is no strace in fremantle, and I would have to set up a chroot for this.
• But what binary ? I didn't even know what program handles the settings. After comparing a few ps listings, I figured out the program was controlpanel.
• Interestingly, you can just ssh into the device as the user, and execute controlpanel, and it will pop up the binary on the phone (even with X forwarding), which I didn't expect, but is fine by me. This gave me a message like "sharing-accountstore.c 516 sharing_account_store_add_account Last added id exists: 35
  " which I wrongly took to mean that adding a facebook account was failing because it was still somehow present in the config. This is when I figured out that this code is closed, because I wasn't able to figure out where to get accountstore.c and look at what it is doing. I'm pretty sure it's not storing its settings in gconf (at least I couldn't find anything related to facebook sharing when walking the gconf tree), and it doesn't seem to be in any dot folders in my home dir. Anyone know ?
• Finally, I gave up, set up the chroot, and straced that binary. And I should have known this would have led me to the right solution. Among all the noise, there were some calls trying to open a library. It's normal for a bunch of calls to fail as it searches the whole LD_LIBRARY_PATH, but usually the last one of those in an strace log should succeed. But they weren't; apparently /usr/lib/libfacebookcommon.so.0 was missing!
• Looking for that file on disk confirmed that it wasn't there, and doing dpkg -L feedservice-plugin-fb-common showed that it should in fact be there. This reminded me of a pet peeve I have with those people who claim Debian's packaging system to be far superior to rpm - apparently dpkg doesn't have any equivalent of rpm -qv which allows you to verify that the files that should be installed by a package are indeed on disk. This would have saved me *a lot* of time figuring out this problem, and is typically the first thing I do on an rpm-based system where things act funny.
• With my first real clue in hand, I used ldd to verify that indeed there were libraries missing:
  

  ldd -r /usr/lib/sharing/plugins/libfacebooksharingservice.so
  ...
  undefined symbol: facebook_credentials_free (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: facebook_request_reset (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: network_utils_post_multipart_with_progress (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: facebook_get_email (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: facebook_login (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: generate_signature (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: facebook_store_credentials_to_gconf (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: facebook_request_new (/usr/lib/sharing/plugins/libfacebooksharingservice.so)
  undefined symbol: facebook_request_free (/usr/lib/sharing/plugins/libfacebooksharingservice.so)

• Using dpkg -S to figure out what package this belonged to, I apt-get remove'd that package, then reinstalled it and dependencies.
• Still didn't work, so repeated the process, this time libfeeserviceutils.so.0 was missing, so reinstall feedservice-utils and dependencies.

And finally! The sharing plugin can again be configured, and all is well. But this experience was needlessly painful...

For the record, I didn't tinker with any package files by hand (anyone who knows me a little knows my stance on packages and /usr), and I have no idea what I did to get into a situation where files that should be on disk aren't. And I'm worried about what else is missing, so if anyone can point me to some resources explaining how to verify installed package manifests, that would be awesome.

One of the things I really love about my N900 is the ease with which I could share photos. You take the picture, click a few buttons, and there it...

I've seen other people blog about it as well, so I shouldn't stay behind - obviously it's noteworthy.

In a good seven hours, our platform will be streaming a talk by the eminent Mr. Lessig from Harvard. Apparently the stream is going to be projected in various locations around the world as well where people will gather to follow the speech. While we're only the technological medium and hence a small piece in making this possible, it still makes me proud to be part of this chain. It's moments like these participating in a chain of openness that make me think business and technology can be used for the greater good.

For more info, see our blog. I'll be tuning in after landing in Brussels at midnight tonight!

I've seen other people blog about it as well, so I shouldn't stay behind - obviously it's noteworthy. In a good seven hours, our platform will be streaming a talk...