Present Perfect

One of our customers mailed us to say that they had some database corruption in one of their main tables, and obviously our friend Murphy had forgotten to take backups, and could we take a look at it ?

Obviously it's always way more fun to poke at someone else's train wreck than your own, and you might learn something that could save your bacon in the future, so I decided to give it a go in my spare time. I asked for a full tar of the system so I could chroot into it and do a post-mortem.

It became quite a time-consuming endeavour - not in actual time spent doing stuff, but waiting for things to happen. First waiting to get ftp details, get the password, figure out their iptables rules so I could actually log in, finding a machine with 150 GB free to transfer the image to, actually transferring the image (took the best part of a week), realizing they only gave me the db partition and not the actual system, when I prefer to have the real system to make my debugging easier, making a backup of all the data over our internal network to the new file server, ... Lots of little steps all taking one minute of work and various hours to complete.

But finally I had their system on one of ours, and I was able to chroot into it, start mysql, and run the query that brought down their server.

The table probably has millions of lines, and I was able to query about 20000 before it crashed. I tried varies things, REPAIR doesn't work on InnoDB anyway.

Since the table was InnoDB, I found a utility called innnodbchecksum and tried it. It spat out:

page 535 invalid (fails new style checksum)


So, one of the early pages is invalid, and MySQL just gives up after that.

I found this presentation that explained a bunch of things about the InnoDB database file format, and looked around for file format parsers.

Inspecting the innodbchecksum binary, it seems there are "old style" and "new style" checksums on pages. Reading the code, it seems the old-style matched, but the new-style didn't. So, one of the random ideas that popped into my head was to change the newstyle checksum on that page. I mean, one of the two seems fine, no ? Haven't tried that yet, saving it for later.

I tried a bunch of methods I found all over, including the interesting-looking innodb_force_recovery option which you can dial from 1 to 6, but that didn't help much either.

I checked if hachoir maybe had a tool to parse innodb files, because I've been looking for a good excuse to play with hachoir since forever, but no luck, although it sounds like a good match.

Surely someone must have already written some tool to look at corrupt innodb database files and recover the 99.99% of good pages out of it ?

Activating my network of MySQL-related contacts however brought me to this very interesting post that I hadn't found through Google. Excited, I followed the instructions. I noticed the instructions were for 0.3, and the latest version had moved and now was 0.4.

The tool basically worked, although something has changed, and I couldn't leave a comment on Chris' blog, so I hope trackback works for the people coming after me... Instead of doing

cd innodb-recovery-tool-0.3/
./create_defs.pl --user=root --password=mysql --db=test --table=t1 > table_defs.h

in step 3, I had to do:

cd percona-innodb-recovery-tool
./create_defs.pl --user=root --password=mysql --db=test --table=t1 > include/table_defs.h


If you don't write to the include dir, you end up with recovery results for a table called 'reptest' which is in the default include/table_defs.h, and not what you want.

Now the tool is taking satisfyingly long to complete, the output data looks like it's mostly intact, and hopefully I can make a customer happy for Christmas with a non-standard service that I hope they'll enjoy.

One of our customers mailed us to say that they had some database corruption in one of their main tables, and obviously our friend Murphy had forgotten to take backups,...

Since we are doing our yearly business planning weekend later this week, I had reserved the weekend to do work - mainly, put together eight (strike that, nine, our CEO added one at the 11th hour Sunday evening) presentations. But I wasn't getting into the groove of things, and procrastination hit.

So I wondered, what's the single most important thing that's been on my mind and that I'd do right now if I didn't have anybody else to answer to ?

And the answer was simple - I started another rewrite of DAD (Digital Audio Database) last year, this time based on CouchDB. I was in the middle of splitting up into a core (defining all base classes and simple implementations without any dependencies; for example, a pickle-based storage of the mixing data), a dadgst module (for a GStreamer-based player, since I will also have a pure web-based player), and a dadcouch module (for a CouchDB storage backend).

Before the split-up it was mostly a hardcoded GStreamer player playing from the pickle file, and a bunch of scripts to analyze files and put them into the pickle. I had not properly finished the CouchDB conversion - mostly, a bunch of methods that previously were synchronous now had to be made asynchronous with deferreds, and that was causing some conceptual issues (like, how to a lot of deferreds together - when chaining doesn't work, and parallellizing brings down your computer).

So, that's what I wanted to do this weekend first - get the couchdb backend to a state where it can select tracks slicing the audiofiles and providing the mixing information, and use the data from the old DAD database of now seven years ago. I want to hear those old songs again, according to my preference, and properly mixed. And with that in place, after a few hours of hacking, I could focus myself completely on the presentation preparing.

Well, completely except for the baby visits, the family lunching, and the pregnant friend visiting.

If you like looking at not-completely-finished-code that probably only I can get running usefully anyway, start here.

Since we are doing our yearly business planning weekend later this week, I had reserved the weekend to do work - mainly, put together eight (strike that, nine, our CEO...

After recording a video at work this week, I needed a way to go through it and identify interesting points. So Andoni suggested I gave LongoMatch a try and package it for Fedora.

LongoMatch is a digital coach, useful to tag and analyze sport matches. We didn't actually play sports, but it was still useful for my purpose. I now have thirty short clips that someone else can now edit into a useful video.

Packages are available for Fedora 13 and 14 in my repo.

After recording a video at work this week, I needed a way to go through it and identify interesting points. So Andoni suggested I gave LongoMatch a try and package...

The first PyChecker release done by yours truly is out the door. You can download the source or install the Fedora 12/13/14 package from my package repository. I hope to push it in Fedora as soon as possible.

This release collects two years of bug fixes, and adds support for Python 2.7, which is now the default in Fedora 14.

I already have a few additional local patches that bring out a bunch of new (and correct) warnings in Flumotion, so I'll probably not wait another two years for the next release. My goal is to work towards a clean check of Flumotion.

The first PyChecker release done by yours truly is out the door. You can download the source or install the Fedora 12/13/14 package from my package repository. I hope to...

After more than two years, it's high time another PyChecker release is made. The previous stable release, 0.8.18, does not work at all with Python 2.7. I also implemented a slew of opcodes, and I fixed one particular bug a long time ago where modules with the same name (in different packages) where hiding each other's errors. Especially in Flumotion where most components have an admin_gtk.py, this was very common.

This is going to be my first PyChecker release, so I would appreciate it if you could give this prerelease a whirl and see how it works for you. I'm especially interested in hearing from projects that used to use pychecker but have stopped using it because it hasn't seen releases the past years.

Download the pre-release. The same directory has a src.rpm and a noarch.rpm for Fedora 14.

I plan to release this Saturday.

After more than two years, it's high time another PyChecker release is made. The previous stable release, 0.8.18, does not work at all with Python 2.7. I also implemented a...