Present Perfect

As mentioned before, I ended up getting a working pystack macro again for use in gdb. Someone commented I should make it public so here it is:


# THOMAS: the test for between Py_Main and Py_GetArgcArgv is because
# code is in that order in the C file; see Modules/main.c and its comment
# print the entire Python call stack
# same for eval in Python/ceval.c

if $pc > PyEval_EvalFrameEx && $pc < PyEval_EvalCodeEx pyframe else # frame end up-silently 1 set $__lastpc = $pc end select-frame 0 end

I also posted it in the upstream bug but I doubt that's going to achieve much in this case...

As mentioned before, I ended up getting a working pystack macro again for use in gdb. Someone commented I should make it public so here it is: # THOMAS: the...

Day two of the internal platform training sessions. Today is hacking and bugfixing day.

I wanted to take a stab at the task of creating an RPC interface to expose our Perspective Broker interface. I have very little experience with RPC systems (apart from PB, and moap's use of Trac's RPC) so this was a good opportunity to get my feet wet.

Smart hacking is lazy hacking, so I started by Googling. After some false positives, I found a Twisted JSON-RPC project, and since it was maintained by Duncan McGreggor it gave me hope that it would work.

And so it did. I wrote a simple adapter object that takes an instance of a PB root and proxies jsonrpc_* calls to remote_* calls.

This is just a proof of concept; obviously there are many caveats. The main thing being that currently you can only use it for remote_ calls with simple objects that txjsonrpc supports (although it looks like for example it supports a deferred, so sweet).

Obviously, one of the attractions of PB is that you can transfer objects. At the least this bridge could be extended to support getting references to objects and then invoking methods on them or passing them as arguments.

In any case, good enough for a one hour hack.

The code is in my tests repository and you can check out with

svn co https://thomas.apestaart.org/thomas/svn/tests/twisted/pb2jr


As for actually using it - after writing it Jan and I discussed how it should be used, and he told me he doesn't actually want to run this in the same process where we run our PB interface. Instead he wants it to acts a proxy, which would at best mean doing code inspection or importing of the PB Root in the proxy to be able to provide the JSON-RPC interface dynamically, and you wouldn't be sure if the running code's PB Root is the same as whatever you have on disk in your proxy.

I'm not very convinced about the coupling argument, because this is just a thin layer on top of the PB server, and everything will end up going through the PB server anyway, so I don't see any gain from decoupling here.

I lost interest while discussing, so I'm going to leave it in its current state for now. Although I would have preferred to just plug ahead, do some of the cooler introspection bits, provide a web UI to look at methods and invoke them, and so on.

Day two of the internal platform training sessions. Today is hacking and bugfixing day. I wanted to take a stab at the task of creating an RPC interface to expose...

Now that I want to write a Maemo UI for my GTD application, I need to address the fact that python-couchdb is written using a blocking model.

On Launchpad, I found Paisley a while ago - a CouchDB client library using Twisted. Seemed functional enough, but I was using the 'schema' stuff from python-couchdb which allows you to define a class with typed properties and map CouchDB documents onto them, and Paisley didn't seem to have any of that. In fact, Paisley seemed to have about 10 commits to it, all done over a year ago.

I had two possible approaches. One was to just use python-couchdb together with threads. It's a typical solution to this problem, and one that is an evolutionary dead-end as far as I'm concerned. However, it would have allowed me to focus on the UI today instead.

The other approach was to see how to map python-couchdb concepts onto paisley.

(I actually considered a third approach - using Rodrigo's couchdb-glib library - except that it seems to also be blocking at this point, so I'd have to fix that, and make python bindings or figure out introspection stuff. So I dropped that idea.)

So, after some IRC discussion between me and cmlenz (author of python-couchdb) and dreid (author of paisley), and both of them thinking that it makes sense to merge some of the schema stuff into Paisley, I started to do just that in this branch.

After some plumbing, I was able to re-use my Thing class that was defined against python-couchdb's schema.Document with Paisley. Success!

Three snippets of IRC conversation (that I hope the authors are fine with me sharing):

from private chat:

cmlenz: and if you port, please rename s/schema/mapping or something like that :)
cmlenz: schema was a horrible name choice for what it does
homeasvs: ok, will keep that in mind

from #twisted:

dreid: homeasvs: Thank you for volunteering.
homeasvs: dreid, no problem
homeasvs: dreid, I was discussing with cmlenz who did python-couchdb and he suggested I take some of his document modeling stuff and graft it onto paisley
dreid: homeasvs: No, I mean volunteering to maintain paisley.
homeasvs: dreid, heh :)
homeasvs: dreid, I might be persuaded if this works out
dreid: I'm afraid you don't understand how this works.
dreid: You volunteered.
dreid: If you don't want to do it, you have to fight a bear.
dreid: ;)
homeasvs: I thought the last touched rule only applied to the master branch :)
dreid: That's not what this is.

from #couchdb:

dreid: homeasvs < - FYI this guy has volunteered to maintain paisley.

These are the days I like how open source works. I am ending the day with two more projects than I started it with.

That reminds me, I should apply dreid's approach to one of my projects and figure out how to give someone commit access...

Now that I want to write a Maemo UI for my GTD application, I need to address the fact that python-couchdb is written using a blocking model. On Launchpad, I...

Sweet. After a bunch of hacking, packaging, tweaking, and cheating, I was able to do the following:

• add stuff to my shopping list on my desktop, with the command-line client
• without doing anything manual to sync my phone, take it with me to the store
• at the store, do 'gtd search @shop' (from a terminal, currently) to get the list of things to shop for, including my recent additions
• come back with my stuff from the store, and mark those items as done on my desktop
• do the same search again on my phone, and see those two items gone

Now of course, this needs some improvement, like a dedicated app to view my things on the phone, plus a way to mark them as done. Also, my code needs optimizing, because out of 2500 things in the couchdb database it takes a good 10 seconds to list these ten. But it's such an awesome glimpse of my future.

Now one has to wonder if the balance of doing all of this technology stuff to implement something as simple as an indestructible shopping list is worth it in the end... But as technologists I'm sure we all know that feeling, and delude ourselves into thinking this is sensible.

Sweet. After a bunch of hacking, packaging, tweaking, and cheating, I was able to do the following: add stuff to my shopping list on my desktop, with the command-line client...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps them into client groups, then makes routing decisions by group to any of our sites based on a policy (least cost, geographically closest, ...) In addition, I wrote a simulator that replays actual request log lines, and is able to then track the bandwidth used by each client group and each site, and even able to dump it to an .rrd file for later graphing. Not bad at all for a quick hack. Python is awesome when it needs to be.

The bad: some binary tree algorithm I use to be able to get a sorted list of simulated events seems to crap out currently at about 1000 request lines with a 'Maximum recursion depth exceeded'. Last time I asked our database guy, I think we had about 10 million lines a day.

I don't think my simulator is ready for prime-time just yet...

The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps...