Present Perfect


Picture Gallery
Present Perfect

asynchronous task interface

Filed under: Hacking,Python — Thomas @ 23:13


In a previous post, I mentioned looking for some code implementing asynchronous operations that you can then attach a progress bar to.

I didn't end up finding anything, so as part of the ripping code I'm writing I prototyped a simple approach to it.

It's working nicely already; I have the basic class for a Task, a class for MultiTasks, and two runners - a command-line blocking one and a GtkProgressBar-using widget.

Here's the code if you're interested. You can run that file to see a simple DummyTask (one that takes 10 seconds to complete) progressing on the command line. Another example you can run (for which you need to check out the whole code) is examples/gtkchecksum.py which checksums decoded audio data using GStreamer and shows you a GTK+ progress bar. An even better example (but you'll need a complete CD rip with matching .cue file) is the ARcue.py program, which calculates AccurateRip checksums and compares them against the online database, and you can choose whether you want to run it in cli or gtk mode.

The cli one uses a GObject main loop to make the asynchronous code seem blocking again. I could easily add a Twisted reactor-using runner, but haven't needed it yet. Also, the one big annoyance I have with the otherwise excellent Twisted is the fact that you can't do much at all without the ugly 'global variable' that is the reactor.

I don't think I'm completely happy with these classes yet. I realized last week that unconsciously I was again moving towards an MVC approach with this, and realizing that also made me realize I'm not yet there. If you consider the Task the Model, then the Runner is currently both the View and Controller. I will probably need to split out the View part again, so I can attach multiple Views to the Model that is the task.

Also, I need to add a way to report on errors while executing the task, I'm considering adding timing statistics and finish estimation, and maybe something that allows doing global progress and subtask progress, the way ripping programs do too (showing current track percentage and complete disc percentage). Not sure how much I want to overengineer it though.

Feel free to comment, suggest improvements, or show me similar concepts and implementations!


  1. You might also be interested in GTask[1] where I implemented Twisted deferreds in C for GOjbect.

    I’m working on the new version of GTask codenamed Iris[2] which is a re-implementation based on Message Passing. It includes a work-stealing scheduler (roughly 8x faster than the thread pool) and Lock-Free data-structures as well.

    You can also have multiple schedulers based on the work sets and the scheduler manager will migrate threads between the schedulers based on demand.

    It’s basically a mix of CCR, NSOperation, Twisted Deferreds, and Parallel C++ extensions for VS10.

    [1] http://audidude.com/blog/?p=118
    [2] http://github.com/chergert/iris

    — Christian

    Comment by Christian Hergert — 2009-04-20 @ 00:13

  2. Have you looked at Christian Hergert’s GTask?


    Seems pretty similar.

    Comment by Joe Shaw — 2009-04-20 @ 01:21

  3. What is gtk module? I thought that for python there is pygtk.

    Comment by Marko Kevac — 2009-04-20 @ 08:17

  4. Thanks for sharing this. It’s definitely something worth looking at :)

    If you want to make the reactor a bit less global, have a look at this ticket: http://twistedmatrix.com/trac/ticket/3205

    However, is there any real need to integrate this at all with Twisted? Any Twisted-using code that wants to update progress on the classes you’ve provided could easily do so. It’s more a question of adding support to specific protocols than to the reactor, I would think.

    Comment by Glyph Lefkowitz — 2009-04-20 @ 09:55

  5. […] today give me, as a followup to yesterday’s task post: […]

    Pingback by thomas.apestaart.org » 90 minutes of hacking — 2009-04-20 @ 23:39

RSS feed for comments on this post. TrackBack URL

Leave a comment