Present Perfect

Since we are doing our yearly business planning weekend later this week, I had reserved the weekend to do work - mainly, put together eight (strike that, nine, our CEO added one at the 11th hour Sunday evening) presentations. But I wasn't getting into the groove of things, and procrastination hit.

So I wondered, what's the single most important thing that's been on my mind and that I'd do right now if I didn't have anybody else to answer to ?

And the answer was simple - I started another rewrite of DAD (Digital Audio Database) last year, this time based on CouchDB. I was in the middle of splitting up into a core (defining all base classes and simple implementations without any dependencies; for example, a pickle-based storage of the mixing data), a dadgst module (for a GStreamer-based player, since I will also have a pure web-based player), and a dadcouch module (for a CouchDB storage backend).

Before the split-up it was mostly a hardcoded GStreamer player playing from the pickle file, and a bunch of scripts to analyze files and put them into the pickle. I had not properly finished the CouchDB conversion - mostly, a bunch of methods that previously were synchronous now had to be made asynchronous with deferreds, and that was causing some conceptual issues (like, how to a lot of deferreds together - when chaining doesn't work, and parallellizing brings down your computer).

So, that's what I wanted to do this weekend first - get the couchdb backend to a state where it can select tracks slicing the audiofiles and providing the mixing information, and use the data from the old DAD database of now seven years ago. I want to hear those old songs again, according to my preference, and properly mixed. And with that in place, after a few hours of hacking, I could focus myself completely on the presentation preparing.

Well, completely except for the baby visits, the family lunching, and the pregnant friend visiting.

If you like looking at not-completely-finished-code that probably only I can get running usefully anyway, start here.

Since we are doing our yearly business planning weekend later this week, I had reserved the weekend to do work - mainly, put together eight (strike that, nine, our CEO...

I've been working on Paisley some more recently, finishing a first stab
at a document mapping API.

As discussed with Christopher Lenz a long time, I basically took his
mapping code and applied it to Paisley.

In my personal project I also added a caching version of the CouchDB
object, but I'm not yet convinced it is the right approach, so it's not
in Paisley yet. One of the things I think I will need to do to make
that useful is to have it listen to change notifications, so it can
change cached objects when they change in couchdb, and implement
notifications for these changes so that a program can be informed of
them too and react accordingly.

In any case, I'd like to work towards a release, so feel free to take a
look at
the branch I've made to implement this on, give any feedback or
do any code review, and let me know.

I've been working on Paisley some more recently, finishing a first stab at a document mapping API. As discussed with Christopher Lenz a long time, I basically took his mapping...

It took quite a bit of blood, sweat, tears, and dirty hacks, but finally:


Nokia-N900-42-11:~# apt-get install desktopcouch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
couchdb-bin erlang-base erlang-crypto erlang-inets erlang-mnesia
erlang-public-key erlang-runtime-tools erlang-ssl erlang-xmerl libjs
python-desktopcouch python-desktopcouch-records python-keyring
Suggested packages:
couchdb erlang-tools erlang erlang-manpages erlang-doc-html
python-keyring-gnome python-keyring-kwallet
Recommended packages:
erlang-syntax-tools
The following NEW packages will be installed:
couchdb-bin desktopcouch erlang-base erlang-crypto erlang-inets
erlang-mnesia erlang-public-key erlang-runtime-tools erlang-ssl erlang-xmerl
libjs python-desktopcouch python-desktopcouch-records python-keyring
0 upgraded, 14 newly installed, 0 to remove and 9 not upgraded.
Need to get 0B/12,5MB of archives.
After this operation, 27,9MB of additional disk space will be used.
Do you want to continue [Y/n]?
WARNING: The following packages cannot be authenticated!
erlang-base erlang-crypto erlang-mnesia erlang-runtime-tools
erlang-public-key erlang-ssl erlang-inets erlang-xmerl libjs couchdb-bin
python-keyring python-desktopcouch python-desktopcouch-records desktopcouch
Install these packages without verification [y/N]? y
Selecting previously deselected package erlang-base.
(Reading database ... 27651 files and directories currently installed.)
Unpacking erlang-base (from .../erlang-base_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-crypto.
Unpacking erlang-crypto (from .../erlang-crypto_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-mnesia.
Unpacking erlang-mnesia (from .../erlang-mnesia_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-runtime-tools.
Unpacking erlang-runtime-tools (from .../erlang-runtime-tools_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-public-key.
Unpacking erlang-public-key (from .../erlang-public-key_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-ssl.
Unpacking erlang-ssl (from .../erlang-ssl_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-inets.
Unpacking erlang-inets (from .../erlang-inets_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package erlang-xmerl.
Unpacking erlang-xmerl (from .../erlang-xmerl_1%3a13.b.2.1-dfsg-3maemo2_armel.deb) ...
Selecting previously deselected package libjs.
Unpacking libjs (from .../libjs_1.8.0-rc1-1_armel.deb) ...
Selecting previously deselected package couchdb-bin.
Unpacking couchdb-bin (from .../couchdb-bin_0.10.0-0maemo3_armel.deb) ...
Selecting previously deselected package python-keyring.
Unpacking python-keyring (from .../python-keyring_0.2-3_all.deb) ...
Selecting previously deselected package python-desktopcouch.
Unpacking python-desktopcouch (from .../python-desktopcouch_0.5.1-0maemo1_all.deb) ...
Selecting previously deselected package python-desktopcouch-records.
Unpacking python-desktopcouch-records (from .../python-desktopcouch-records_0.5.1-0maemo1_all.deb) ...
Selecting previously deselected package desktopcouch.
Unpacking desktopcouch (from .../desktopcouch_0.5.1-0maemo1_all.deb) ...
Setting up erlang-base (1:13.b.2.1-dfsg-3maemo2) ...
Searching for services which depend on erlang and should be started...none found.
Setting up erlang-crypto (1:13.b.2.1-dfsg-3maemo2) ...
Setting up erlang-mnesia (1:13.b.2.1-dfsg-3maemo2) ...
Setting up erlang-runtime-tools (1:13.b.2.1-dfsg-3maemo2) ...
Setting up erlang-public-key (1:13.b.2.1-dfsg-3maemo2) ...
Setting up erlang-ssl (1:13.b.2.1-dfsg-3maemo2) ...
Setting up erlang-inets (1:13.b.2.1-dfsg-3maemo2) ...
Setting up erlang-xmerl (1:13.b.2.1-dfsg-3maemo2) ...
Setting up libjs (1.8.0-rc1-1) ...
Setting up couchdb-bin (0.10.0-0maemo3) ...
Setting up python-keyring (0.2-3) ...
Setting up python-desktopcouch (0.5.1-0maemo1) ...
Setting up python-desktopcouch-records (0.5.1-0maemo1) ...
Setting up desktopcouch (0.5.1-0maemo1) ...
Nokia-N900-42-11:~# dbus-send --session --dest=org.desktopcouch.CouchDB \
> --print-reply --type=method_call / \
> org.desktopcouch.CouchDB.getPort
method return sender=:1.516 -> dest=:1.515 reply_serial=2
int32 53864
Nokia-N900-42-11:~#


Among other things, I had to:

• port desktopcouch to use python-keyring as suggested by the desktopcouch hackers, although I'm not sure what they now want to do with it since there's no way to make it compatible with their current way of storing in gnome-keyring
• Build the arm parts of erlang and couchdb on my phone in a chroot, since qemu-arm-sb actually hangs when executing erlc. Had to build it in an actual maemo chroot as well; the easy-deb chroot was not a solution since it comes with newer C and ssl libraries.
• figure out how to package spidermonkey, instead of depending on xulrunner the way the debian couchdb does
• package lots of missing packages; for example, while maemo has avahi, it didn't have the python bindings built.

For the adventurous, I think you should be able to download tao.install on your phone, then go to console and run the same apt-get install command as I did. Please let me know if it worked for you !

Now, need to figure out (or get told by the desktopcouch hackers) how I can set up pairing with UbuntuOne for my data without needing to port the whole gnome applet...

It took quite a bit of blood, sweat, tears, and dirty hacks, but finally: Nokia-N900-42-11:~# apt-get install desktopcouch Reading package lists... Done Building dependency tree Reading state information... Done The...

Now that I want to write a Maemo UI for my GTD application, I need to address the fact that python-couchdb is written using a blocking model.

On Launchpad, I found Paisley a while ago - a CouchDB client library using Twisted. Seemed functional enough, but I was using the 'schema' stuff from python-couchdb which allows you to define a class with typed properties and map CouchDB documents onto them, and Paisley didn't seem to have any of that. In fact, Paisley seemed to have about 10 commits to it, all done over a year ago.

I had two possible approaches. One was to just use python-couchdb together with threads. It's a typical solution to this problem, and one that is an evolutionary dead-end as far as I'm concerned. However, it would have allowed me to focus on the UI today instead.

The other approach was to see how to map python-couchdb concepts onto paisley.

(I actually considered a third approach - using Rodrigo's couchdb-glib library - except that it seems to also be blocking at this point, so I'd have to fix that, and make python bindings or figure out introspection stuff. So I dropped that idea.)

So, after some IRC discussion between me and cmlenz (author of python-couchdb) and dreid (author of paisley), and both of them thinking that it makes sense to merge some of the schema stuff into Paisley, I started to do just that in this branch.

After some plumbing, I was able to re-use my Thing class that was defined against python-couchdb's schema.Document with Paisley. Success!

Three snippets of IRC conversation (that I hope the authors are fine with me sharing):

from private chat:

cmlenz: and if you port, please rename s/schema/mapping or something like that :)
cmlenz: schema was a horrible name choice for what it does
homeasvs: ok, will keep that in mind

from #twisted:

dreid: homeasvs: Thank you for volunteering.
homeasvs: dreid, no problem
homeasvs: dreid, I was discussing with cmlenz who did python-couchdb and he suggested I take some of his document modeling stuff and graft it onto paisley
dreid: homeasvs: No, I mean volunteering to maintain paisley.
homeasvs: dreid, heh :)
homeasvs: dreid, I might be persuaded if this works out
dreid: I'm afraid you don't understand how this works.
dreid: You volunteered.
dreid: If you don't want to do it, you have to fight a bear.
dreid: ;)
homeasvs: I thought the last touched rule only applied to the master branch :)
dreid: That's not what this is.

from #couchdb:

dreid: homeasvs < - FYI this guy has volunteered to maintain paisley.

These are the days I like how open source works. I am ending the day with two more projects than I started it with.

That reminds me, I should apply dreid's approach to one of my projects and figure out how to give someone commit access...

Now that I want to write a Maemo UI for my GTD application, I need to address the fact that python-couchdb is written using a blocking model. On Launchpad, I...

Sweet. After a bunch of hacking, packaging, tweaking, and cheating, I was able to do the following:

• add stuff to my shopping list on my desktop, with the command-line client
• without doing anything manual to sync my phone, take it with me to the store
• at the store, do 'gtd search @shop' (from a terminal, currently) to get the list of things to shop for, including my recent additions
• come back with my stuff from the store, and mark those items as done on my desktop
• do the same search again on my phone, and see those two items gone

Now of course, this needs some improvement, like a dedicated app to view my things on the phone, plus a way to mark them as done. Also, my code needs optimizing, because out of 2500 things in the couchdb database it takes a good 10 seconds to list these ten. But it's such an awesome glimpse of my future.

Now one has to wonder if the balance of doing all of this technology stuff to implement something as simple as an indestructible shopping list is worth it in the end... But as technologists I'm sure we all know that feeling, and delude ourselves into thinking this is sensible.

Sweet. After a bunch of hacking, packaging, tweaking, and cheating, I was able to do the following: add stuff to my shopping list on my desktop, with the command-line client...