Present Perfect

So, it's been a long weekend and I'll have to recap the essentials sometime later.

I got my new phone - the n900 - on Friday from Zaheer. I wrote down first impressions but they'll have to wait for now.
I spent most of Thursday night setting up scratchbox a few times and preparing to understand how to Build Stuff.

I was hoping I would be able to build on the source for previous packages but the original site that hosts those packages has been down for two months now. And not down in the usual way - it just leaves the connection attempt hanging! I mailed the guy, but got no reply. So, no other choice than to try it myself.

Over the course of the weekend, I managed to repackage .debs from Debian for erlang, spidermonkey and couchdb to work in the x86 scratchbox. I won't bore you with the details on how much I suffered trying to learn debian packaging enough to build a working xulrunner. In the end, I managed to figure out that there was a very lightweight 'spidermonkey' source tree that builds just the js I need for couchdb (after reading the CouchDB MacOSX build instructions) and I packaged that up from scratch, which was definitely easier.

At some point during the weekend, I managed to run couchdb completely in the x86 scratchbox, so from that point I thought I was home free. Just rebuild the packages in the ARMEL scratchbox and off I go.

Boy was I ever wrong.

No, really.

I basically spent all of today trying to figure out why the build of erlang would hang as soon as it invoked the just-built erlang erlc compiler. Five processes down it seems stuck in a select(0, NULL, NULL, NULL) call, which afaict would have no hope of waking up. It was preceeded by an error about qemu not handling syscall 242. (it took me a while to figure out that this is the syscall for setting scheduler affinity, so probably not that necessary ?)

I got heaps of suggestions from various people, and thought of some things on my own too:

• implement syscall 242 for qemu. Not my idea of a good time just yet.
• patch out use of 242. Doable.
• Set up sbrsh and use cpu transparency. There were no uptodate instructions for fremantle, it took me a good hour to figure out that these instructions are very misleading since those pages do not preserve the formatting, which is essential for the .sbrsh configuration file, besides also having extranuous spacing. And even when that worked to the point I was able to transparently execute hello-world-gtk, it would still hang on running simply quilt when doing the package build.
• Since Erlang compiles to bytecode, I thought I could just copy .beam and .hrl files from my x86 tree into my armel tree each time the compiler hung on a compile. I actually did this for a good 90 minutes, until I gave up, because it was too tedious. I might try it again, but automated.
• It should be possible to create a devkit with just the x86 erlang compiler, then convince the armel scratchbox target to use that compiler to generate all the erlang bytecode, instead of the just-built in-source compiler. Need to investigate further.
• Upload the package to extras-devel, and have it built there. Good idea, but I'm waiting to be approved for now. Also, I don't know if extras-devel will use qemu - it might fail the same way. And I'm not a fan of a build that can only be reproduced in an external service...
• Building it on the phone directly. Apparently this was a working approach with previous devices. I added the sdk repo to the phone, and it proposed to remove some essential umbrella package before installing some dev packages. I didn't want to brick my phone just yet.
• Someone suggested to install scratchbox on the phone. An intriguing thought, but sadly it took only 5 minutes of work to see E: Currently Scratcbox can only run in 32 bit i386 architecture.

I have now settled on a mutation of some of these ideas, and we'll see how that goes. I basically created /opt/chroot, copied essentials off the phone's root into that dir, and chrooted into it. In there, I added the sdk repo, and am now installing the tools to be able to build the packages as if I was doing it directly on the phone - but nicely protected inside a chroot.

I hope there are going to be some packages at the end of this tunnel. And to think some people actually enjoy this lowlevel device hacking!

So, it's been a long weekend and I'll have to recap the essentials sometime later. I got my new phone - the n900 - on Friday from Zaheer. I wrote...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps them into client groups, then makes routing decisions by group to any of our sites based on a policy (least cost, geographically closest, ...) In addition, I wrote a simulator that replays actual request log lines, and is able to then track the bandwidth used by each client group and each site, and even able to dump it to an .rrd file for later graphing. Not bad at all for a quick hack. Python is awesome when it needs to be.

The bad: some binary tree algorithm I use to be able to get a sorted list of simulated events seems to crap out currently at about 1000 request lines with a 'Maximum recursion depth exceeded'. Last time I asked our database guy, I think we had about 10 million lines a day.

I don't think my simulator is ready for prime-time just yet...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps...