[lang]

Present Perfect

Personal
Projects
Packages
Patches
Presents
Linux

Picture Gallery
Present Perfect

Programming contest

Filed under: Flumotion,Fun,Work — Thomas @ 5:27 pm

2009-6-29
5:27 pm

Jan and Arek entered this year’s ICFP programming contest. It’s a three day programming contest, so this morning they asked if they could swap their Friday project day for today to finish the contest. They seem to be in the top third at the moment.

Arek’s never been a fan of long meetings, but today’s standup meeting was particularly amusing with Arek urging everyone to keep focused and get out there quickly. They had less than two hours left on the clock.

29062009

Spot the seven differences

Amusingly, today they came to work with almost the same shirt on, by accident! I can only assume there is a big clothes factory in Poland where they have huge stock of the same fabric…

90 minutes left, knock them dead, guys!

mach 0.9.5 ‘MMM…’ released

Filed under: Fedora,Hacking,Releases — Thomas @ 10:57 pm

2009-6-27
10:57 pm

mach allows you to set up clean roots from scratch for any distribution or distribution variation supported.

This release of mach contains fixes for Python 2.6, and adds Fedora 10 and 11, while fixing the archived Fedora locations.

Get it from the mach project page.

moap vcs bisect

Filed under: Flumotion,moap,Python — Thomas @ 8:32 pm

8:32 pm

Next step on this weekend’s yakshave: a first implementation of moap vcs bisect!

The interface is lifted from git, obviously, since that’s where most people will know the feature from.

I implemented it first with CVS, so I could fix this pychecker bug which was blocking Fedora from bumping the pychecker version from 0.8.17 (3 years old) to 0.8.18. And sure enough, it picked out the commit I broke.

While implementing and while dealing with CVS’s idea of how it stores CVS revisions and dates and so on, I googled and was amused to find this first hit on google for the words cvs and bisect. Clever Andy! And he cleverly sidestepped the problem I wrestled with by making the user specify two dates at the start instead of trying to figure it out from the checkout. And all in lisp too!

Then, to test that my VCS interface was sane, I implemented it for Subversion as well. That took about 15 minutes, since Subversion is much more sane than CVS. I tried the following command on a flumotion checkout:

moap vcs bisect reset; moap vcs bisect start; svn up; moap vcs bisect good; svn up -r 3000; moap vcs bisect bad; MOAP_DEBUG=4 moap vcs bisect run ./test.sh

With test.sh containing
test -e flumotion/component/consumers/gdp/gdp.py

(In other words, look for the commit that added this file.)

Sure enough, it picked out this commit:


[moap-trunk] [thomas@ana flumotion]$ moap vcs bisect diff
Index: /home/thomas/tmp/flumotion/configure.ac
===================================================================
--- /home/thomas/tmp/flumotion/configure.ac (revision 6909)
+++ /home/thomas/tmp/flumotion/configure.ac (revision 6908)
@@ -212,7 +212,6 @@
flumotion/component/combiners/switch/Makefile
flumotion/component/consumers/Makefile
flumotion/component/consumers/disker/Makefile
-flumotion/component/consumers/gdp/Makefile
flumotion/component/consumers/httpstreamer/Makefile
flumotion/component/consumers/preview/Makefile
flumotion/component/consumers/shout2/Makefile
Index: /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am
===================================================================
--- /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6909)
+++ /home/thomas/tmp/flumotion/flumotion/component/consumers/Makefile.am (revision 6908)
@@ -11,7 +11,6 @@

SUBDIRS = \
disker \
- gdp \
httpstreamer \
preview \
shout2
Index: /home/thomas/tmp/flumotion/ChangeLog
===================================================================
--- /home/thomas/tmp/flumotion/ChangeLog (revision 6909)
+++ /home/thomas/tmp/flumotion/ChangeLog (revision 6908)
@@ -1,16 +1,5 @@
2008-06-20 Thomas Vander Stichele <thomas at apestaart dot org>
</thomas>

- * configure.ac:
- * flumotion/component/consumers/Makefile.am:
- * flumotion/component/consumers/gdp (added):
- * flumotion/component/consumers/gdp/gdp.py (added):
- * flumotion/component/consumers/gdp/__init__.py (added):
- * flumotion/component/consumers/gdp/Makefile.am (added):
- * flumotion/component/consumers/gdp/gdp.xml (added):
- Add a GDP consumer.
-
-2008-06-20 Thomas Vander Stichele <thomas at apestaart dot org>
-
* flumotion/component/producers/gdp/gdp.py:
Add error for http://bugzilla.gnome.org/show_bug.cgi?id=532364
</thomas>

So, the feature is ready for testing. It could use some more documenting, and some additional goodies like accepting arguments to moap vcs bisect start for example.

Feedback appreciated!

jhbuild for python

Filed under: Python — Thomas @ 7:15 pm

7:15 pm

Still on the yak shave expedition.

I’ve written some simple scripts and files to set up and build python 2.3, 2.4, and 2.5 in separate prefixes to be able to test my software against these versions.

If you’re interested, in theory it should be really simple:

As the README says, this should go on to build all versions of python, and install some scripts.

After that, you just run py-2.3 to go into a shell with Python 2.3 on your path.

Don’t say I never did anything for you.

Python disassembly

Filed under: Hacking,Python — Thomas @ 6:19 pm

6:19 pm

As part of this weekend’s yakshave, I’m trying to implement a handler for STORE_MAP in pychecker. STORE_MAP is a new opcode in Python 2.6, which speeds up dict building.

So, for the first time I went under the hood of Python and figured out just enough to understand this problem. It was a lot less scary than I thought it was going to be!

It seems that using dis.dis(), one can easily dissassemble any python function into its opcodes. This shows clearly where the behaviour is different between python 2.5 and python 2.6.

Given the following function:
f = lambda: {'a': 1, 'b': 2}

Python 2.6 gives:
1 0 BUILD_MAP 2
3 LOAD_CONST 0 (1)
6 LOAD_CONST 1 ('a')
9 STORE_MAP
10 LOAD_CONST 2 (2)
13 LOAD_CONST 3 ('b')
16 STORE_MAP
17 RETURN_VALUE

I couldn’t find a good description of the output of dis.dis, but in my naiveness I am guessing the following:

  • The first 1 maps to the line number in the code object where the function is found.
  • The second column is the offset of the opcode and its arguments
  • The third column is the opcode name
  • The next column is the arguments for the opcode; in the case of CONST, it shows the index as well as the const object indexed

I am assuming each opcode takes one address location, and each argument takes two; that maps with the address pointers in front of the opcodes.

The opcodes are all documented.

So, in human terms:

  • we start with BUILD_MAP, saying that we’ll create a new dictionary on the stack, with 2 entries.
  • we load the constant with index 0 onto the stack (which happens to be the integer object ‘1’, the value of the ‘a’ key)
  • we load the constant with index 1 onto the stack (which happens to be the string object ‘a’, the key for the ‘1’ value)
  • STORE_MAP pops the key and the value off the stack, storing them in the dict. Note that the key was indeed loaded on the stack after the value. Now only the dictionary is left on the stack.
  • Repeat LOAD_CONST, LOAD_CONST and STORE_MAP for the next set
  • RETURN_VALUE returns the current value on the stack to the caller

Pretty simple, when you look at it twice.

For the same code, python 2.5 gives:
>>> dis.dis(f)
1 0 BUILD_MAP 0
3 DUP_TOP
4 LOAD_CONST 1 ('a')
7 LOAD_CONST 2 (1)
10 ROT_THREE
11 STORE_SUBSCR
12 DUP_TOP
13 LOAD_CONST 3 ('b')
16 LOAD_CONST 4 (2)
19 ROT_THREE
20 STORE_SUBSCR
21 RETURN_VALUE

This code is slightly longer and more complicated. Basically, LOAD_CONST, LOAD_CONST, STORE_MAP was implemented with DUP_TOP, LOAD_CONST, LOAD_CONST, ROT_THREE, STORE_SUBSCR

It looks like DUP_TOP was needed because STORE_SUBSCR consumes the dictionary off the stack, and ROT_THREE is needed because the arguments are pushed on the stack in the wrong order.

Seems like a nice and obvious improvement once you understand it. An exercise for the reader is to profile whether this change actually makes things faster in practice.

So, where does this leave me for pychecker ? It now looks deceptively simple. STORE_MAP simply pops off two items of the stack. There is nothing to check for, since we’re in a dictionary context. So all my implementation needs to do is to pop 2 items off the stack, and that’s it.

And thus it was commited to pychecker CVS. Popping one item off the yak stack!

Next Page »
picture