Present Perfect

As part of this weekend's yakshave, I'm trying to implement a handler for STORE_MAP in pychecker. STORE_MAP is a new opcode in Python 2.6, which speeds up dict building.

So, for the first time I went under the hood of Python and figured out just enough to understand this problem. It was a lot less scary than I thought it was going to be!

It seems that using dis.dis(), one can easily dissassemble any python function into its opcodes. This shows clearly where the behaviour is different between python 2.5 and python 2.6.

Given the following function:

f = lambda: {'a': 1, 'b': 2}


Python 2.6 gives:

1 0 BUILD_MAP 2
3 LOAD_CONST 0 (1)
6 LOAD_CONST 1 ('a')
9 STORE_MAP
10 LOAD_CONST 2 (2)
13 LOAD_CONST 3 ('b')
16 STORE_MAP
17 RETURN_VALUE


I couldn't find a good description of the output of dis.dis, but in my naiveness I am guessing the following:

• The first 1 maps to the line number in the code object where the function is found.
• The second column is the offset of the opcode and its arguments
• The third column is the opcode name
• The next column is the arguments for the opcode; in the case of CONST, it shows the index as well as the const object indexed

I am assuming each opcode takes one address location, and each argument takes two; that maps with the address pointers in front of the opcodes.

The opcodes are all documented.

So, in human terms:

• we start with BUILD_MAP, saying that we'll create a new dictionary on the stack, with 2 entries.
• we load the constant with index 0 onto the stack (which happens to be the integer object '1', the value of the 'a' key)
• we load the constant with index 1 onto the stack (which happens to be the string object 'a', the key for the '1' value)
• STORE_MAP pops the key and the value off the stack, storing them in the dict. Note that the key was indeed loaded on the stack after the value. Now only the dictionary is left on the stack.
• Repeat LOAD_CONST, LOAD_CONST and STORE_MAP for the next set
• RETURN_VALUE returns the current value on the stack to the caller

Pretty simple, when you look at it twice.

For the same code, python 2.5 gives:

>>> dis.dis(f)
1 0 BUILD_MAP 0
3 DUP_TOP
4 LOAD_CONST 1 ('a')
7 LOAD_CONST 2 (1)
10 ROT_THREE
11 STORE_SUBSCR
12 DUP_TOP
13 LOAD_CONST 3 ('b')
16 LOAD_CONST 4 (2)
19 ROT_THREE
20 STORE_SUBSCR
21 RETURN_VALUE


This code is slightly longer and more complicated. Basically, LOAD_CONST, LOAD_CONST, STORE_MAP was implemented with DUP_TOP, LOAD_CONST, LOAD_CONST, ROT_THREE, STORE_SUBSCR

It looks like DUP_TOP was needed because STORE_SUBSCR consumes the dictionary off the stack, and ROT_THREE is needed because the arguments are pushed on the stack in the wrong order.

Seems like a nice and obvious improvement once you understand it. An exercise for the reader is to profile whether this change actually makes things faster in practice.

So, where does this leave me for pychecker ? It now looks deceptively simple. STORE_MAP simply pops off two items of the stack. There is nothing to check for, since we're in a dictionary context. So all my implementation needs to do is to pop 2 items off the stack, and that's it.

And thus it was commited to pychecker CVS. Popping one item off the yak stack!

As part of this weekend's yakshave, I'm trying to implement a handler for STORE_MAP in pychecker. STORE_MAP is a new opcode in Python 2.6, which speeds up dict building. So,...

The yak shave started yesterday evening. The yak stack is actually a forked one this time, both of the forks involving pychecker.

I might not remember everything in order, but in a nutshell the stack is something like this:

• The original goal for this weekend at the beginning of the week was to release my cd ripper, morituri
• Something in its pychecker run did not run with pychecker 0.8.17 (the version still in Fedora, from 2006), and worked with pychecker 0.8.18 (my own build). Fork point 1.
• While releasing moap this week, I realized that Freshmeat changed their remote API, which I should fix before I do another release of anything. Fork point 2.

Fork point 1 continues here:

• I mailed Fedora's pychecker maintainer, offering my help, and sending a patch to update to 0.8.18, which I built and installed locally. He mailed back informing me about this pychecker bug with anaconda which was blocking the upgrade. Looking at that bug, it looked suspiciously like a bug triggered by code I added to pychecker last year.
• However, I'd like to confirm so in the easiest way. git's got this great feature, git bisect, and wouldn'it be nice if I could do that on pychecker now ? Hey, why not add bisection to moap ?
• CVS is actually not a very manageable VCS if you want to do fancy stuff. It costs me a few hours to figure out how I should get a more-or-less usable date from a CVS checkout. The final solution is similar to the reply to my stackoverflow question
• moap vcs bisect run is now implemented and finds the commit that broke anaconda's pychecking. STACK POINTER IS CURRENTLY HERE.

Fork point 2 continues here:

• moap's make check didn't work because pychecker complained about the following code:
  
ef func():
d = { 'a': 1, 'b': 2}
print d.keys()


  which triggers, in python 2.6, the following warning:
  
Object (d) has no attribute (keys)


• I can't let myself change code in moap without a working make check, so on to figuring out what's wrong in pychecker
• After lots of debugging and print statements, I figure out that pychecker dispatches Python opcodes, and it silently drops the ones it doesn't know about. Python 2.6 added a new opcode, STORE_MAP, and so pychecker doesn't properly handle the stack since it ignores the opcode. I should error out on those opcodes. STACK POINTER IS CURRENTLY HERE
• Before I can fix that though, I decide I should make pychecker's test suite error out on unknown opcodes.
• Of course, this will error out differently for different python versions, so I need different python versions on this machine.
• I could do that by hand, but I'd also like Twisted's trial to run the testsuite, which also needs zope-interface in recent versions, which needs setuptools. So hey, why not set up jhbuild stuff to build all these python versions ?
• Python 2.3 on my 64 bit machine doesn't work with setuptools, so the newest Twisted without it is 1.3.0, and it takes a while to figure out why trial doesn't run my testcases (it ends up being because 1.3.0 trial expects subclasses of twisted.trial.unittest.TestCase, not unittest.TestCase)

I'll blog about the useful products of my yak shave separately, for those who don't enjoy descriptions of yak shavings, only outcomes.

In general, I actually enjoy yak shaves. They're massive treasure hunts, you learn a lot, and you end up fixing a nice bunch of things all over the stack if you persevere. But it's probably more a mentality thing than anything else, and I really only indulge myself in these in my spare time.

The yak shave started yesterday evening. The yak stack is actually a forked one this time, both of the forks involving pychecker. I might not remember everything in order, but...

moap is a swiss army knife for maintainers and developers.

This is MOAP 0.2.7, "MMM...".

Coverage in 0.2.7: 1424 / 1899 (74 %), 109 python tests, 2 bash tests

Features added since 0.2.6:
- Added moap vcs backup, a command to backup a checkout to a tarball that
can be used later to reconstruct the checkout. Implemented for svn.
- Fixes for git-svn, git, svn and darcs.
- Fixes for Python 2.3 and Python 2.6

I've been fixing things left and right for python 2.6, and in the process I noticed that moap hasn't had a release for over a year. This release contains mostly bug fixes collected over the year, and a new feature that isn't implemented yet for all VCS's. Basically it's an automatic replacement for something I was doing manually every time I removed an old GNOME cvs/svn/git checkout: figure out what's in that tree that's not in the repository (diffs, unversioned files, ...), so I can delete everything else and free some disk space.

The only problem with this release is that, after doing the release, I noticed that Freshmeat removed their XML-RPC interface. Apparently they have some new kind of interface they want people to use. Sigh. But that means 0.2.8 is right around the corner!

moap is a swiss army knife for maintainers and developers. This is MOAP 0.2.7, "MMM...". Coverage in 0.2.7: 1424 / 1899 (74 %), 109 python tests, 2 bash tests Features...

Last week, after upgrading my home desktop to F11, I had palimpsest tell me one of my disks was broken on the desktop machine. The desktop is running on two 250 GB drives in software raid. It was time to get new drives.

After a weekend of fiddling with new 1 TB disks for my home desktop, trying failure scenarios, making sure the system can boot from each of the two drives, and waiting for the 4 hour resync of the software RAID in between each step, I finally closed up the desktop machine and cleaned up under my desk again, thinking I was done with my halfyearly messing about with broken disks.

I guess I was tempting faith anyway. Doing a routine operation on my home server after all the configuration stuff I'd done to set up asterisk last week, suddenly an rsync aborted, a journal errored out, a partition changed to being mounted read-only, and the log was full of scary drive errors. Ouch.

Well, that's why I keep around a big box of old drives - for when some drive fails and I want to tempt fate even more by reusing an old drive that's probably going to fail real soon too. And anyway, I had just spent my hard drive piggybank on the new desktop drives.

Luckily, I seemed to have a 400 GB SATA drive lying around that used to belong to my media center. I don't remember why I swapped it out, given that the media center has a 160GB drive for the OS (and two 1.5 TB raid drives for the data, of course), but this was a lucky break. I booted with a rescue cd, and tried copying the root filesystem of my CentOS 5.2 home server partition to this new drive. Which worked fine, except that /var was where I triggered an Input/Output error and some more drive errors in the kernel log.

So, powered off, took out the broken drive, and put it in a USB chassis. The advantage of a USB chassis is that you can easily just replug the drive to try again, instead of locking up your system terribly and having to reboot. Sadly, /var was broken beyond repair. I ran an e2fsck hoping to recover the contents, and that partly worked, but some of the important stuff is missing even from lost+found (apart from the annoying situation where you have to reconstruct file names, which I usually end up not bothering with).

But really, how important can /var be ? Turns out, rather important. As in, you need it to boot in the first place. And also, it holds your rpm database. Crap.

Some Googling gave me some posts on how to reconstruct your rpm database from log files (using --justdb --noscripts --notriggers). But to use those, you actually need those log files. Where are those ? On /var as well. Crap. And they're not in lost+found either.

Ok, so time to get creative. Here's what I ended up doing:

• create /var/lib/rpm, and run rpm --rebuilddb to end up with an empty rpm database
• Based on the contents of /usr/bin, figure out what packages ought to be installed:
  
rpm -qf /etc/* | grep 'not owned' | cut -f2 -d' ' > /tmp/unowned
yum --enablerepo=c5-media --disablerepo=base --disablerepo=updates --disablerepo=addons --disablerepo=extras whatprovides `cat /tmp/unowned` | cut -f1 -d' ' | sort | uniq > /tmp/missing
yum --enablerepo=c5-media --disablerepo=base --disablerepo=updates --disablerepo=addons --disablerepo=extras install `cat /tmp/missing`


  This works by first listing all files that are not owned by rpm (on the first run, that's all of them), figure out what packages can provide these files, then installing those packages.

• Repeat the process for other important directories, like /bin, /sbin, /usr/sbin, /usr/lib, /usr/include, ...
• Clean up .rpmnew files that don't actually contain differences:
  
find / -name *.rpmnew | sed s/.rpmnew//g > /tmp/rpmnew
for c in `cat /tmp/rpmnew`; do echo $c; diff $c $c.rpmnew && mv -f $c.rpmnew $c; done


• Same for *.rpmorig:
  
find / -name *.rpmorig | sed s/.rpmorig//g > /tmp/rpmorig
for c in `cat /tmp/rpmorig`; do echo $c; diff $c $c.rpmorig && mv -f $c.rpmorig $c; done

• Inspect the remaining ones, and merge changes.

While it's not an experience I hope to repeat any time soon, it worked out surprisingly well!

Last week, after upgrading my home desktop to F11, I had palimpsest tell me one of my disks was broken on the desktop machine. The desktop is running on two...

I managed to completely skip updating to F10. All my machines (work desktop, home desktop, laptop, media center) where running F9 without any real problems I worried about.

But of course I was curious. And, especially with the move to python 2.6, things I care about where bound to break.

So, last weekend I took the plunge, and after little over a week here are my first impressions:

• Overall F11 looks slick. Nice work on the artwork! I particularly liked the GDM background, looking like an ancient brushed metal object, reminding me of how I used to love playing Gods by the Bitmap Brothers.
• Apparently anaconda now has bugzilla integration, allowing you to file a bug directly from inside anaconda. Luckily for Jeremy (who I assume still maintains it) it has some code in there to look for existing bug entries with the same backtrace. Very nice!
• Of course, I wouldn't have found out if I hadn't run into exceptions in anaconda. I ran into while setting up two completely new hard disks with 2 software raid partition and LVM on the second one.
• I first installed my work desktop, as usual putting the new installation on a separate partition, keeping my old one around in case the install goes wrong or F11 just isn't stable enough for me. For me, that involves having a /mnt/alpha and /mt/omega partition between which I alternate. At some point I should figure out if other people do this too and if it makes sense for anaconda to support something like this and at least allow me to keep my GRUB configuration for the older installation. For now I do this manually, using a hugeupgrade text file I follow each time I upgrade, accumulating more and more steps each time I perform the procedure.
• On my home machine, when I booted into F11, as usual my second monitor didn't work (I have a Radeon GeCube Pro 2400). My own fault really - I should have tried to get a patch upstream into the default radeon driver the same way I sent a patch for the radeonhd driver that I still use. A rebuild later, I at least had the old radeonhd driver rebuilt to get my second screen working again.
• Having the second screen now made me change my mind completely about the GDM wallpaper. That lion on the right hand side that I didn't see before completely ruins the style for me. Sorry!
• Upon logging in to the work desktop, I had no network. Completely puzzled as to why, until I figured out that I had to actually right-click on NetworkManager's tray icon, choose to configure, and activate eth0 by default. After some browsing it seems that this was a deliberate choice to increase security. While I can possibly sympathize with the motivation for doing so, it really is terrible to change this by default and not provide *any* indication during or after installation. At the very least, the following things could have been done:
  1. provide a clear notice during installation, and allow a user to choose to enable it anyway, assuming the security risk
  2. the same, but during firstboot
  3. after logging in, having the network manager tooltip say 'the network is disabled by default in this new release, here's how you enable it

  I am not entirely sure what the security problems are with enabling the network after installation. The default firewall is pretty locked down, SELinux is enabled by default, and there's no way I can install updates without the network anyway. But I'm sure that I could find huge bikeshedding threads on fedora-devel about this if I really cared why this was decided.

• Upon logging in to the home desktop, I was greeted with a tooltip saying that one of my drives was going bad. That was a nice touch! Really good idea to have something like that be monitored by default. This prompted me to ponder to finally replace my desktop's 250 GB PATA drives with real SATA drives - a story for another post.
• Various deprecation warnings pop up running various Python programs, including my own. Flumotion needed a patch for running against 2.6 (I rebuilt and pushed to F11). So I have some cleanup ahead, and I should revisit pychecker soon.
• The first piece of functionality I checked was Evolution's Google Calendar integration. It still seems a bit shaky, given that I had to restart Evolution a few times as it froze doing stuff with the net, but it does seem to work. That means I will finally be able to accept work invitations done through Outlook and get them on my Google Calendar! Awesome. Now if only I didn't have to manually configure each of the ten calendars I'm interested in...
• At work, when I played a video using XVideo, my machine instantly froze. Seems to be a known bug. The intel drivers are being rewritten. I've never quite understood why rewriting is an excuse for breaking stuff that worked (I should check if Firewire video finally works reliably now when I have the chance, for example), but all in the name of progress I guess.
• I don't know why it's happening, but once in a while my screens blank. Even in the middle of doing stuff. If I were a gamer I'd be hugely annoyed as my character would be shot through the head in that split instant. The closest bug I can find is this one, where I commented. Hugely annoying bug because I don event know how to begin debugging a bug like this that I can't catch in the act.
• PulseAudio integration in GDM seems a bit fragile. I have my pulseaudio configured to send audio to my media center pulseaudio server. Sometimes, after choosing a username in GDM, it doesn't manage to play the audio sample related to that action, and then GDM is stuck there not showing me the password entry dialog. Pretty sure it's due to blocking on pulseaudio, because when I kill it the password dialog appears. Pretty painful bug for new users though.

All in all, not a bad first week experience, and seems like a solid release. Now, off to rebuild bits and pieces, and clean up Python 2.6 deprecation warnings...

I managed to completely skip updating to F10. All my machines (work desktop, home desktop, laptop, media center) where running F9 without any real problems I worried about. But of...