Present Perfect

mach allows you to set up clean roots from scratch for any distribution or distribution variation supported.

This release of mach contains fixes for Python 2.6, and adds Fedora 10 and 11, while fixing the archived Fedora locations.

Get it from the mach project page.

mach allows you to set up clean roots from scratch for any distribution or distribution variation supported. This release of mach contains fixes for Python 2.6, and adds Fedora 10...

Next step on this weekend's yakshave: a first implementation of moap vcs bisect!

The interface is lifted from git, obviously, since that's where most people will know the feature from.

I implemented it first with CVS, so I could fix this pychecker bug which was blocking Fedora from bumping the pychecker version from 0.8.17 (3 years old) to 0.8.18. And sure enough, it picked out the commit I broke.

While implementing and while dealing with CVS's idea of how it stores CVS revisions and dates and so on, I googled and was amused to find this first hit on google for the words cvs and bisect. Clever Andy! And he cleverly sidestepped the problem I wrestled with by making the user specify two dates at the start instead of trying to figure it out from the checkout. And all in lisp too!

Then, to test that my VCS interface was sane, I implemented it for Subversion as well. That took about 15 minutes, since Subversion is much more sane than CVS. I tried the following command on a flumotion checkout:

moap vcs bisect reset; moap vcs bisect start; svn up; moap vcs bisect good; svn up -r 3000; moap vcs bisect bad; MOAP_DEBUG=4 moap vcs bisect run ./test.sh


With test.sh containing

test -e flumotion/component/consumers/gdp/gdp.py


(In other words, look for the commit that added this file.)

Sure enough, it picked out this commit:

[moap-trunk] [thomas@ana flumotion]$ moap vcs bisect diff
Index: /home/thomas/tmp/flumotion/configure.ac
===================================================================
--- /home/thomas/tmp/flumotion/configure.ac (revision 6909)
+++ /home/thomas/tmp/flumotion/configure.ac (revision 6908)
@@ -212,7 +212,6 @@
flumotion/component/combiners/switch/Makefile
flumotion/component/consumers/Makefile
flumotion/component/consumers/disker/Makefile
-flumotion/component/consumers/gdp/Makefile
flumotion/component/consumers/httpstreamer/Makefile
flumotion/component/consumers/preview/Makefile
flumotion/component/consumers/shout2/Makefile
Index: /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am
===================================================================
--- /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6909)
+++ /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6908)
@@ -11,7 +11,6 @@



SUBDIRS = \
disker \
- gdp \
httpstreamer \
preview \
shout2
Index: /home/thomas/tmp/flumotion/ChangeLog
===================================================================
--- /home/thomas/tmp/flumotion/ChangeLog (revision 6909)
+++ /home/thomas/tmp/flumotion/ChangeLog (revision 6908)
@@ -1,16 +1,5 @@
2008-06-20 Thomas Vander Stichele


- * configure.ac:
- * flumotion/component/consumers/Makefile.am:
- * flumotion/component/consumers/gdp (added):
- * flumotion/component/consumers/gdp/gdp.py (added):
- * flumotion/component/consumers/gdp/__init__.py (added):
- * flumotion/component/consumers/gdp/Makefile.am (added):
- * flumotion/component/consumers/gdp/gdp.xml (added):
- Add a GDP consumer.
-
-2008-06-20 Thomas Vander Stichele -
* flumotion/component/producers/gdp/gdp.py:
Add error for http://bugzilla.gnome.org/show_bug.cgi?id=532364

So, the feature is ready for testing. It could use some more documenting, and some additional goodies like accepting arguments to moap vcs bisect start for example.

Feedback appreciated!

Next step on this weekend's yakshave: a first implementation of moap vcs bisect! The interface is lifted from git, obviously, since that's where most people will know the feature from....

Still on the yak shave expedition.

I've written some simple scripts and files to set up and build python 2.3, 2.4, and 2.5 in separate prefixes to be able to test my software against these versions.

If you're interested, in theory it should be really simple:

• Install jhbuild
• Check out my python jhbuild directory: svn co https://thomas.apestaart.org/thomas/svn/jhbuild/python
• run 'make'

As the README says, this should go on to build all versions of python, and install some scripts.

After that, you just run py-2.3 to go into a shell with Python 2.3 on your path.

Don't say I never did anything for you.

Still on the yak shave expedition. I've written some simple scripts and files to set up and build python 2.3, 2.4, and 2.5 in separate prefixes to be able to...

As part of this weekend's yakshave, I'm trying to implement a handler for STORE_MAP in pychecker. STORE_MAP is a new opcode in Python 2.6, which speeds up dict building.

So, for the first time I went under the hood of Python and figured out just enough to understand this problem. It was a lot less scary than I thought it was going to be!

It seems that using dis.dis(), one can easily dissassemble any python function into its opcodes. This shows clearly where the behaviour is different between python 2.5 and python 2.6.

Given the following function:

f = lambda: {'a': 1, 'b': 2}


Python 2.6 gives:

1 0 BUILD_MAP 2
3 LOAD_CONST 0 (1)
6 LOAD_CONST 1 ('a')
9 STORE_MAP
10 LOAD_CONST 2 (2)
13 LOAD_CONST 3 ('b')
16 STORE_MAP
17 RETURN_VALUE


I couldn't find a good description of the output of dis.dis, but in my naiveness I am guessing the following:

• The first 1 maps to the line number in the code object where the function is found.
• The second column is the offset of the opcode and its arguments
• The third column is the opcode name
• The next column is the arguments for the opcode; in the case of CONST, it shows the index as well as the const object indexed

I am assuming each opcode takes one address location, and each argument takes two; that maps with the address pointers in front of the opcodes.

The opcodes are all documented.

So, in human terms:

• we start with BUILD_MAP, saying that we'll create a new dictionary on the stack, with 2 entries.
• we load the constant with index 0 onto the stack (which happens to be the integer object '1', the value of the 'a' key)
• we load the constant with index 1 onto the stack (which happens to be the string object 'a', the key for the '1' value)
• STORE_MAP pops the key and the value off the stack, storing them in the dict. Note that the key was indeed loaded on the stack after the value. Now only the dictionary is left on the stack.
• Repeat LOAD_CONST, LOAD_CONST and STORE_MAP for the next set
• RETURN_VALUE returns the current value on the stack to the caller

Pretty simple, when you look at it twice.

For the same code, python 2.5 gives:

>>> dis.dis(f)
1 0 BUILD_MAP 0
3 DUP_TOP
4 LOAD_CONST 1 ('a')
7 LOAD_CONST 2 (1)
10 ROT_THREE
11 STORE_SUBSCR
12 DUP_TOP
13 LOAD_CONST 3 ('b')
16 LOAD_CONST 4 (2)
19 ROT_THREE
20 STORE_SUBSCR
21 RETURN_VALUE


This code is slightly longer and more complicated. Basically, LOAD_CONST, LOAD_CONST, STORE_MAP was implemented with DUP_TOP, LOAD_CONST, LOAD_CONST, ROT_THREE, STORE_SUBSCR

It looks like DUP_TOP was needed because STORE_SUBSCR consumes the dictionary off the stack, and ROT_THREE is needed because the arguments are pushed on the stack in the wrong order.

Seems like a nice and obvious improvement once you understand it. An exercise for the reader is to profile whether this change actually makes things faster in practice.

So, where does this leave me for pychecker ? It now looks deceptively simple. STORE_MAP simply pops off two items of the stack. There is nothing to check for, since we're in a dictionary context. So all my implementation needs to do is to pop 2 items off the stack, and that's it.

And thus it was commited to pychecker CVS. Popping one item off the yak stack!

As part of this weekend's yakshave, I'm trying to implement a handler for STORE_MAP in pychecker. STORE_MAP is a new opcode in Python 2.6, which speeds up dict building. So,...

The yak shave started yesterday evening. The yak stack is actually a forked one this time, both of the forks involving pychecker.

I might not remember everything in order, but in a nutshell the stack is something like this:

• The original goal for this weekend at the beginning of the week was to release my cd ripper, morituri
• Something in its pychecker run did not run with pychecker 0.8.17 (the version still in Fedora, from 2006), and worked with pychecker 0.8.18 (my own build). Fork point 1.
• While releasing moap this week, I realized that Freshmeat changed their remote API, which I should fix before I do another release of anything. Fork point 2.

Fork point 1 continues here:

• I mailed Fedora's pychecker maintainer, offering my help, and sending a patch to update to 0.8.18, which I built and installed locally. He mailed back informing me about this pychecker bug with anaconda which was blocking the upgrade. Looking at that bug, it looked suspiciously like a bug triggered by code I added to pychecker last year.
• However, I'd like to confirm so in the easiest way. git's got this great feature, git bisect, and wouldn'it be nice if I could do that on pychecker now ? Hey, why not add bisection to moap ?
• CVS is actually not a very manageable VCS if you want to do fancy stuff. It costs me a few hours to figure out how I should get a more-or-less usable date from a CVS checkout. The final solution is similar to the reply to my stackoverflow question
• moap vcs bisect run is now implemented and finds the commit that broke anaconda's pychecking. STACK POINTER IS CURRENTLY HERE.

Fork point 2 continues here:

• moap's make check didn't work because pychecker complained about the following code:
  
ef func():
d = { 'a': 1, 'b': 2}
print d.keys()


  which triggers, in python 2.6, the following warning:
  
Object (d) has no attribute (keys)


• I can't let myself change code in moap without a working make check, so on to figuring out what's wrong in pychecker
• After lots of debugging and print statements, I figure out that pychecker dispatches Python opcodes, and it silently drops the ones it doesn't know about. Python 2.6 added a new opcode, STORE_MAP, and so pychecker doesn't properly handle the stack since it ignores the opcode. I should error out on those opcodes. STACK POINTER IS CURRENTLY HERE
• Before I can fix that though, I decide I should make pychecker's test suite error out on unknown opcodes.
• Of course, this will error out differently for different python versions, so I need different python versions on this machine.
• I could do that by hand, but I'd also like Twisted's trial to run the testsuite, which also needs zope-interface in recent versions, which needs setuptools. So hey, why not set up jhbuild stuff to build all these python versions ?
• Python 2.3 on my 64 bit machine doesn't work with setuptools, so the newest Twisted without it is 1.3.0, and it takes a while to figure out why trial doesn't run my testcases (it ends up being because 1.3.0 trial expects subclasses of twisted.trial.unittest.TestCase, not unittest.TestCase)

I'll blog about the useful products of my yak shave separately, for those who don't enjoy descriptions of yak shavings, only outcomes.

In general, I actually enjoy yak shaves. They're massive treasure hunts, you learn a lot, and you end up fixing a nice bunch of things all over the stack if you persevere. But it's probably more a mentality thing than anything else, and I really only indulge myself in these in my spare time.

The yak shave started yesterday evening. The yak stack is actually a forked one this time, both of the forks involving pychecker. I might not remember everything in order, but...