Present Perfect

Our team is working on this great new system. It's really cool even though I can't say much about it. They've been surprising me with some inventive stuff. I saw a cool testing tool last month that allowed simulating the whole system on one machine, and it worked, and it detected bugs in the code that they were then able to fix and test again.

But we can't find a good QA engineer to come help us make this thing really rock. Most QA curriculums I get are written in .doc or .docx, show the person having only very vague Unix knowledge, and think Python is a snake best stayed away from.

We contracted a QA consulting company, and after a four day audit they concluded that we were not doing a simple web framework and they did not have the skills to test our system.

I'm sure there's a great QA engineer out there eager to test our multi-datacenter multi-server cluster system doing cool stuff, with some QA experience, some SCRUM experience maybe, some Linux experience please, and some Python experience if possible. But most of all, with a desire to learn, and some capability to start out on his/her own in one project and slowly grow a QA team from the inside out.

Come on, I know you're out there. Send us a mail!

If you're interested, here's the job description.

Our team is working on this great new system. It's really cool even though I can't say much about it. They've been surprising me with some inventive stuff. I saw...

I have two projects on which I will eventually need some kind of file metadata management. One is my still-being-written music application, with one of its core features being distributed - I have three computers and a bunch of devices (like my phone) to put music on, and I want to stop doing that manually.

At its core it will need a distributed database (for example desktop couch) and some rules to decide what files to copy where.

A second application is yet to be started, and I hope it already exists and I can avoid having to write it. But at its core it would help me keep my offline backups up-to-date by indexing online data on all my machines, tracking a backup strategy for each machine and its folders (for example, documents and photos need really good backup, while downloaded iso's for fedora I can lose no problem), and telling me when I should bring an external drive from work and hook it up to the NAS because, according to the records, I have at least 100 MB of changes in important files that I care about.

If I am serious about wanting to avoid rolling my own, I should check what metadata projects exist out there, and the first one on the list is Tracker.

I'm surprised that, for all I've read about it on Planet Gnome I never actually tried to use it.

So, my initial impressions:

• Tracker is a really ungoogleable name. No, you go and try to find instructions on how to start Tracker!
• I installed it on Fedora 14 - the package were there so that's a good start
• I found the search tool. Obviously it doesn't find anything. However, it doesn't actually tell me that. Surely a message telling me that there is nothing indexed yet would make sense.
• Strangely enough it doesn't just start indexing. How do I start using this thing ? I have my laptop running and it can index all night, but it just needs a kick! The documentation is not much help - the header says 'How can I use it' then goes on to mention tools but does not mention the very first step to take to get going.
• Since Google doesn't help much, let's check the package with rpm -qa tracker. The README mentions a tracker-miner-fs in /usr/libexec that should index. Starting that does something. A status icon now shows a looking glass blinking once every two seconds. Applications are 100% indexed, my file system 1%.
• It's hardly using any CPU at all. top doesn't show any tracker-related process doing anything. I want to throw more CPU at tracker. The preferences already have resource use maxed out. I'm sure what I'm asking is ironic to the devs who I'm sure have had gazillions of bug reports asking them to make tracker use less CPU.
• Still not sure how to use the search tool. Typing any letter and searching for it doesn't do anything. Surely one of the files or applications it has indexed has the letter a or e in it?
• Since Applications are indexed, maybe it can find 'terminal' ? Oh great, yes, it finds that.
• Playing around with the command line tools instead then.
  

  $ tracker-search a
  Search term 'a' is a stop word.
  Stop words are common words which may be ignored during the indexing process.

  Results: 0

  Oh, ok, so you're ignoring one letter searches. Maybe your GUI should actually *tell the user*.

• Meanwhile, browsing the documentation to see if tracker is going to suit my two purposes mentioned above, I see it doesn't store any kind of checksums on files. I don't know if that is considered unnecessary overhead (and I'm wrong thinking I will need it for my purposes) or just no added yet.
• asked a question in the IRC channel, did not get a reply so far. Not meant as criticism - IRC help gets taken for granted easily by people, and I know how hard it is to make an IRC channel responsive.

I'm going to let tracker sit and see if indexes something by tomorrow. It'd be great to use a tool that I know will get lots of love and care in GNOME.

I'm pretty sure underneath there's some excellent hacking, but for developers evaluating solutions to their problems the five minute out of the box experience is important. I think Tracker could do a few quick improvements and get some easy wins to get more people convinced.

I have two projects on which I will eventually need some kind of file metadata management. One is my still-being-written music application, with one of its core features being distributed...

When I look at the list folder in my Evolution these days it's just a depressing sight. So manny mailinglists I've accumulated membership to that go unread now.

I think the model of mailing lists is just fundamentally broken. I just subscribed to the cdparanoia mailing list because I had a bug filed against morituri related to the sample-offset option. And you have to subscribe just to be able to post.

So the process to get some question answered is:

• ask to be subscribed
• wait for the request mail to arrive, which arrives somewhere in my admin/list folder which I keep forgetting about because of server-side .procmailrc rules
• wait some more, because the greylisting on our mailserver probably delayed the arrival
• reply to it
• wait some more, you're now subscribed
• send your question
• wait some more
• update my Evolution folder subscription selection, because another .procmailrc rule (that I probably stole from jdub) automatically filters the mail into an unfiltered subdirectory which is a catch-all for all mailing lists; but it only shows up in the list when the mail actually arrived of course
• check the mail in the list, which is usually only mine
• remember to check that folder over the next few days for possible answers
• end up accumulating years of cdparanoia list mails that I won't ever read because the friction of this process is just too much and it's just easier to waste bytes across the net

Of course, I can just change my account settings on a particular list to not receive mails, until I have my next question, but that's still painful.

My garden is in dire need of weeding here, but I don't even know what direction to take it in. Anyone who feels they have a good handle on the mailing list problem want to share their approach with me ?

In particular, I've been googling for a solution to a very particular problem: the not noticing mails on mailing lists that are directly for me - either as a reply to one of my questions, or because I'm being CC'd somehow. My filters file all mails to mailing list in their respective folders, and I tend to miss the mails addressed to me a lot. Surely there is an easy procmail recipe that, when I'm in CC in some other way than just through the mailing list, also leaves a copy of the mail in my INBOX ?
Surely there must

When I look at the list folder in my Evolution these days it's just a depressing sight. So manny mailinglists I've accumulated membership to that go unread now. I think...

Having various machines, some with homedirs passed on across distro versions, I somehow ended up with both Download and Downloads directories. Not to mention that my Firefox and Chromium instances were downloading everywhere, and the silly mental friction of this one letter difference is pissing me off. I was renaming one to the other on one machine, and probably doing the opposite on another.

So, no more. Instead of figuring out what the spec says, I just created a fresh user on my Fedora 14 laptop and did:

$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos


So, that's what it's going to be. Never mind that I hate with a passion something ugly like Music vs Videos (where do I put audio podcasts then ?), I'll just follow like rank and file.

Since I did get a little curious though where the two folders came from, I took a quick look at the FDO page (which as recent blogs have shown is the One Standard Body) and sure enough - version 0.11 renamed Download to Downloads.

This post is just a note to self so that any time I find a computer with both, I fix it properly, instead of the Browsian motion I'm on right now.

Having various machines, some with homedirs passed on across distro versions, I somehow ended up with both Download and Downloads directories. Not to mention that my Firefox and Chromium instances...

One of our customers mailed us to say that they had some database corruption in one of their main tables, and obviously our friend Murphy had forgotten to take backups, and could we take a look at it ?

Obviously it's always way more fun to poke at someone else's train wreck than your own, and you might learn something that could save your bacon in the future, so I decided to give it a go in my spare time. I asked for a full tar of the system so I could chroot into it and do a post-mortem.

It became quite a time-consuming endeavour - not in actual time spent doing stuff, but waiting for things to happen. First waiting to get ftp details, get the password, figure out their iptables rules so I could actually log in, finding a machine with 150 GB free to transfer the image to, actually transferring the image (took the best part of a week), realizing they only gave me the db partition and not the actual system, when I prefer to have the real system to make my debugging easier, making a backup of all the data over our internal network to the new file server, ... Lots of little steps all taking one minute of work and various hours to complete.

But finally I had their system on one of ours, and I was able to chroot into it, start mysql, and run the query that brought down their server.

The table probably has millions of lines, and I was able to query about 20000 before it crashed. I tried varies things, REPAIR doesn't work on InnoDB anyway.

Since the table was InnoDB, I found a utility called innnodbchecksum and tried it. It spat out:

page 535 invalid (fails new style checksum)


So, one of the early pages is invalid, and MySQL just gives up after that.

I found this presentation that explained a bunch of things about the InnoDB database file format, and looked around for file format parsers.

Inspecting the innodbchecksum binary, it seems there are "old style" and "new style" checksums on pages. Reading the code, it seems the old-style matched, but the new-style didn't. So, one of the random ideas that popped into my head was to change the newstyle checksum on that page. I mean, one of the two seems fine, no ? Haven't tried that yet, saving it for later.

I tried a bunch of methods I found all over, including the interesting-looking innodb_force_recovery option which you can dial from 1 to 6, but that didn't help much either.

I checked if hachoir maybe had a tool to parse innodb files, because I've been looking for a good excuse to play with hachoir since forever, but no luck, although it sounds like a good match.

Surely someone must have already written some tool to look at corrupt innodb database files and recover the 99.99% of good pages out of it ?

Activating my network of MySQL-related contacts however brought me to this very interesting post that I hadn't found through Google. Excited, I followed the instructions. I noticed the instructions were for 0.3, and the latest version had moved and now was 0.4.

The tool basically worked, although something has changed, and I couldn't leave a comment on Chris' blog, so I hope trackback works for the people coming after me... Instead of doing

cd innodb-recovery-tool-0.3/
./create_defs.pl --user=root --password=mysql --db=test --table=t1 > table_defs.h

in step 3, I had to do:

cd percona-innodb-recovery-tool
./create_defs.pl --user=root --password=mysql --db=test --table=t1 > include/table_defs.h


If you don't write to the include dir, you end up with recovery results for a table called 'reptest' which is in the default include/table_defs.h, and not what you want.

Now the tool is taking satisfyingly long to complete, the output data looks like it's mostly intact, and hopefully I can make a customer happy for Christmas with a non-standard service that I hope they'll enjoy.

One of our customers mailed us to say that they had some database corruption in one of their main tables, and obviously our friend Murphy had forgotten to take backups,...