Present Perfect

Now that I want to write a Maemo UI for my GTD application, I need to address the fact that python-couchdb is written using a blocking model.

On Launchpad, I found Paisley a while ago - a CouchDB client library using Twisted. Seemed functional enough, but I was using the 'schema' stuff from python-couchdb which allows you to define a class with typed properties and map CouchDB documents onto them, and Paisley didn't seem to have any of that. In fact, Paisley seemed to have about 10 commits to it, all done over a year ago.

I had two possible approaches. One was to just use python-couchdb together with threads. It's a typical solution to this problem, and one that is an evolutionary dead-end as far as I'm concerned. However, it would have allowed me to focus on the UI today instead.

The other approach was to see how to map python-couchdb concepts onto paisley.

(I actually considered a third approach - using Rodrigo's couchdb-glib library - except that it seems to also be blocking at this point, so I'd have to fix that, and make python bindings or figure out introspection stuff. So I dropped that idea.)

So, after some IRC discussion between me and cmlenz (author of python-couchdb) and dreid (author of paisley), and both of them thinking that it makes sense to merge some of the schema stuff into Paisley, I started to do just that in this branch.

After some plumbing, I was able to re-use my Thing class that was defined against python-couchdb's schema.Document with Paisley. Success!

Three snippets of IRC conversation (that I hope the authors are fine with me sharing):

from private chat:

cmlenz: and if you port, please rename s/schema/mapping or something like that :)
cmlenz: schema was a horrible name choice for what it does
homeasvs: ok, will keep that in mind

from #twisted:

dreid: homeasvs: Thank you for volunteering.
homeasvs: dreid, no problem
homeasvs: dreid, I was discussing with cmlenz who did python-couchdb and he suggested I take some of his document modeling stuff and graft it onto paisley
dreid: homeasvs: No, I mean volunteering to maintain paisley.
homeasvs: dreid, heh :)
homeasvs: dreid, I might be persuaded if this works out
dreid: I'm afraid you don't understand how this works.
dreid: You volunteered.
dreid: If you don't want to do it, you have to fight a bear.
dreid: ;)
homeasvs: I thought the last touched rule only applied to the master branch :)
dreid: That's not what this is.

from #couchdb:

dreid: homeasvs < - FYI this guy has volunteered to maintain paisley.

These are the days I like how open source works. I am ending the day with two more projects than I started it with.

That reminds me, I should apply dreid's approach to one of my projects and figure out how to give someone commit access...

Now that I want to write a Maemo UI for my GTD application, I need to address the fact that python-couchdb is written using a blocking model. On Launchpad, I...

Sweet. After a bunch of hacking, packaging, tweaking, and cheating, I was able to do the following:

• add stuff to my shopping list on my desktop, with the command-line client
• without doing anything manual to sync my phone, take it with me to the store
• at the store, do 'gtd search @shop' (from a terminal, currently) to get the list of things to shop for, including my recent additions
• come back with my stuff from the store, and mark those items as done on my desktop
• do the same search again on my phone, and see those two items gone

Now of course, this needs some improvement, like a dedicated app to view my things on the phone, plus a way to mark them as done. Also, my code needs optimizing, because out of 2500 things in the couchdb database it takes a good 10 seconds to list these ten. But it's such an awesome glimpse of my future.

Now one has to wonder if the balance of doing all of this technology stuff to implement something as simple as an indestructible shopping list is worth it in the end... But as technologists I'm sure we all know that feeling, and delude ourselves into thinking this is sensible.

Sweet. After a bunch of hacking, packaging, tweaking, and cheating, I was able to do the following: add stuff to my shopping list on my desktop, with the command-line client...

So, it's been a long weekend and I'll have to recap the essentials sometime later.

I got my new phone - the n900 - on Friday from Zaheer. I wrote down first impressions but they'll have to wait for now.
I spent most of Thursday night setting up scratchbox a few times and preparing to understand how to Build Stuff.

I was hoping I would be able to build on the source for previous packages but the original site that hosts those packages has been down for two months now. And not down in the usual way - it just leaves the connection attempt hanging! I mailed the guy, but got no reply. So, no other choice than to try it myself.

Over the course of the weekend, I managed to repackage .debs from Debian for erlang, spidermonkey and couchdb to work in the x86 scratchbox. I won't bore you with the details on how much I suffered trying to learn debian packaging enough to build a working xulrunner. In the end, I managed to figure out that there was a very lightweight 'spidermonkey' source tree that builds just the js I need for couchdb (after reading the CouchDB MacOSX build instructions) and I packaged that up from scratch, which was definitely easier.

At some point during the weekend, I managed to run couchdb completely in the x86 scratchbox, so from that point I thought I was home free. Just rebuild the packages in the ARMEL scratchbox and off I go.

Boy was I ever wrong.

No, really.

I basically spent all of today trying to figure out why the build of erlang would hang as soon as it invoked the just-built erlang erlc compiler. Five processes down it seems stuck in a select(0, NULL, NULL, NULL) call, which afaict would have no hope of waking up. It was preceeded by an error about qemu not handling syscall 242. (it took me a while to figure out that this is the syscall for setting scheduler affinity, so probably not that necessary ?)

I got heaps of suggestions from various people, and thought of some things on my own too:

• implement syscall 242 for qemu. Not my idea of a good time just yet.
• patch out use of 242. Doable.
• Set up sbrsh and use cpu transparency. There were no uptodate instructions for fremantle, it took me a good hour to figure out that these instructions are very misleading since those pages do not preserve the formatting, which is essential for the .sbrsh configuration file, besides also having extranuous spacing. And even when that worked to the point I was able to transparently execute hello-world-gtk, it would still hang on running simply quilt when doing the package build.
• Since Erlang compiles to bytecode, I thought I could just copy .beam and .hrl files from my x86 tree into my armel tree each time the compiler hung on a compile. I actually did this for a good 90 minutes, until I gave up, because it was too tedious. I might try it again, but automated.
• It should be possible to create a devkit with just the x86 erlang compiler, then convince the armel scratchbox target to use that compiler to generate all the erlang bytecode, instead of the just-built in-source compiler. Need to investigate further.
• Upload the package to extras-devel, and have it built there. Good idea, but I'm waiting to be approved for now. Also, I don't know if extras-devel will use qemu - it might fail the same way. And I'm not a fan of a build that can only be reproduced in an external service...
• Building it on the phone directly. Apparently this was a working approach with previous devices. I added the sdk repo to the phone, and it proposed to remove some essential umbrella package before installing some dev packages. I didn't want to brick my phone just yet.
• Someone suggested to install scratchbox on the phone. An intriguing thought, but sadly it took only 5 minutes of work to see E: Currently Scratcbox can only run in 32 bit i386 architecture.

I have now settled on a mutation of some of these ideas, and we'll see how that goes. I basically created /opt/chroot, copied essentials off the phone's root into that dir, and chrooted into it. In there, I added the sdk repo, and am now installing the tools to be able to build the packages as if I was doing it directly on the phone - but nicely protected inside a chroot.

I hope there are going to be some packages at the end of this tunnel. And to think some people actually enjoy this lowlevel device hacking!

So, it's been a long weekend and I'll have to recap the essentials sometime later. I got my new phone - the n900 - on Friday from Zaheer. I wrote...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps them into client groups, then makes routing decisions by group to any of our sites based on a policy (least cost, geographically closest, ...) In addition, I wrote a simulator that replays actual request log lines, and is able to then track the bandwidth used by each client group and each site, and even able to dump it to an .rrd file for later graphing. Not bad at all for a quick hack. Python is awesome when it needs to be.

The bad: some binary tree algorithm I use to be able to get a sorted list of simulated events seems to crap out currently at about 1000 request lines with a 'Maximum recursion depth exceeded'. Last time I asked our database guy, I think we had about 10 million lines a day.

I don't think my simulator is ready for prime-time just yet...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps...

I usually tend to think of Python as the discerning gentleman's programming language: well-behaved, well-documented, people take care of the code written. I like the batteries-included approach and assume that the battery code in the standard library is well-written. "import this" is a vision statement directly included in the language - it's hard to get more stylish than that.

I got an eye-opener this weekend however. I was still on my quest to get desktopcouch and ubuntuone working on Fedora. While wresting with this bug and doing things that I usually consider a hanging offense (changing /usr-installed code by adding prints to figure out where the craziness was coming from) I finally drilled down to the exception-raising reason. It all boiled down to a single line of code in httplib.py:

def __init__(self, host, port=None, key_file=None, cert_file=None,
strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):

where socket.py contains

_GLOBAL_DEFAULT_TIMEOUT = object()


So, in a nutshell, httplib2.py tracebacks because of this new object, which isn't a valid argument to sock.settimeout()

Now, I'm pretty sure I'm running into this problem because I'm doing "bad" things to some of the stdlib, pulling in bits I need to make ubuntuone (coded on a 2.6.3 python where my Fedora 11 comes with 2.6) work.

But pulling the cover off like this did point out this one object that:

• seems to be intended to be private, but it gets referenced from other stdlib modules
• comes with no documentation at all
• comes with not a single comment explaining *why* it's there, or *why* it's ok to just create a completely empty and useless "object" that you can't even trace the origin of (I had to override __setattr__ on some class to figure out what the anonymous object was, and where it was being set from, to find it)

Maybe I'm oldfashioned, but this leaves me disappointed. This one line breaks beauty, explicitness, and readability that is included in the Zen of Python.

The only attempt at explaining I found is this.

Any Python guru want to set me straight on why this isn't the incredibly ugly wart on stdlib that I consider it to be ?

Meanwhile I'll go digging in svn to see when it was added to 2.6 and why.

I usually tend to think of Python as the discerning gentleman's programming language: well-behaved, well-documented, people take care of the code written. I like the batteries-included approach and assume that...