Present Perfect

I finally got around to completing my release, after a week of absence from my apartment.

This is intended as a release for daring and curious people who've had enough
of the fact that Windows has a more accurate CD ripper than Linux.

Here are the features:

- UTF-8/unicode handling fixes
- improved error handling
- ignore tags for alac and wav
- work around GStreamer flacparse bugs
- change how paths get referenced in .cue files
- properly interpret AccurateRip results; no more assertions on unexpected
ordering of results
- add debug command


Get it at morituri or from package repositories if you're on Fedora 14/15

I finally got around to completing my release, after a week of absence from my apartment. This is intended as a release for daring and curious people who've had enough...

After a weekend of extensive bug fixing, and some followup hacking around recent GStreamer regressions that I saw on Fedora 15, everything seems to be hunky dory again, and my F15 rips are just as good and accurate as my F14 rips. Christophe Fergeau as if by magic dropped by IRC on Freenode in #morituri and spent some time figuring out what the problem and the cause was, leaving me with enough energy to actually hack around it.

morituri is a cd ripper that aims for accuracy over speed, implementing functionality not commonly available in Linux rippers, like AccurateRip verification and read offset detection/use.

Get the 0.1.1.2 prerelease, or install the package from my package repositories repositories, and let me know how it goes!

After a weekend of extensive bug fixing, and some followup hacking around recent GStreamer regressions that I saw on Fedora 15, everything seems to be hunky dory again, and my...

Apparently GNOME released a new major version, so I decided to check out what all the fuss was about.

At first my heart sank when I went to the GNOME 3 site and I saw a Youtube video there. Really ? Are we promoting our new release using proprietary technology ?

But fearest you not - it was actually defaulting to the HTML5 player! Neat! I'm guessing it needs a special embed code to default to that. My heart jumped two steps up again. Then I wanted to check what it was using in my case.

I right-clicked, and there was a Save Video option. Huh, that's interesting, never saw that on youtube. Let's try that.

And then I saw Rick Astley dancing.

I never really understood the comic value of rickrolling. 1970 called and they would like you to explain them too.

Apparently GNOME released a new major version, so I decided to check out what all the fuss was about. At first my heart sank when I went to the GNOME...

Well, the cat has been out of the bag for a few days and I have been too busy to blog about it.

But today as I wait for my team to do a final deploy fixing a bug with too-long URL names for Flash Media Encoder, I have some spare time to mention what's going on and make some people an offer they cannot refuse.

So, for the past half year of so we've been hacking away at a new service to solve a very specific problem in streaming. From 2005-2010 the streaming world mostly settled on Flash as a common platform, which was an unstable equilibrium for everyone involved, but it seemed to work. However, with the amount of codecs, devices and platforms there are today, this equilibrium has been falling. The introduction of iPhone, Microsoft's heavy pushing of Silverlight (paying companies to stream in it - and funnily enough those companies usually stop using Silverlight when the money faucet closes), GoogleTV, the introduction of WebM, the arrival of HTML5 (ironically pushed by Apple - yay - even though their HTML5 sites usually only work in Safari - boo)... all these movements served to upset the status quo once again.

To the eye of the casual observer, it would seem that all streaming has standardized on H264, and so transmuxing technologies are popping up - taking the same video encoding and just remux it for different technologies. However, in practice, H264 is a collection of many techniques and profiles, different levels of complexity, and not all devices support the same profiles and techniques. If you want to stream to all H264 devices with just one encoding, you'll have to settle for the least common denominator in terms of quality, and you'll have to pick a resolution that works subpar for all of them.

Now, content producers hate this sort of situation. They just want to get the signal out there, because that's what matters. The codec and the streaming is just the technological means to get it across the internet. And now the market is asking them to put a bunch of machines in their facilities, learn a lot of technologies they'd rather not worry about, consume heaps of bandwidth to send each version online, and then have to do it all over again each time something changes out there - a new codec, a new device, a new favorite resolution, ...

Our answer to this problem is simple: send us one encoding, we will do the rest. Our service will take your live stream, transcode it to as many different encodings as you want, and hand them off to a CDN. That's basically it. Want full HTML5 coverage ? We'll do it for you - H264 single and multibitrate, Theora, WebM, and a Flash fallback. Want Silverlight, Flash RTMP, Windows Media MMS ? All there.

Services like this already exist for ondemand - see zencoder and encoding.com and Panda. Live is just inherently more difficult - you don't get to work with nice single finished files, and it has to happen right now. But this is exactly the sort of thing a framework like GStreamer is good for.

In reality we aren't doing anything new here - Flumotion runs a CDN that already provides this service to customers. The difference is that this time, you will be able to set it up yourself online. A standard integration time with any CDN is around two weeks. This service will cut that time down to five minutes. We're not quite there yet, but we're close.

What's that you say ? Something about an offer ? Oh, right. It's always pained me to see that, when we wanted to stream a conference for free, it was still quite a bit of work in the setup stage for our support team, and hence we didn't stream as many conferences as I would have liked to. Similarly, it pains me to see a lot of customers not even considering free formats.

So the offer is simple. If you are running an event or a conference that flies under a Free/Open banner, and you're willing to stream only in free formats (meaning, Theora and WebM), and you're willing to ride the rough wave of innovation as we shake out our last bugs, we want to help you out. Send us the signal, we'll do the rest. Drop me a line and let's see how we can set it up. Offer limited, standard handwavy disclaimers apply, you'll have to take my word for it, etc...

If you're in the streaming industry, I will be demoing this new service next week on Wednesday around 2.00 pm local time in New York City, at Streaming Media East. And after that our Beta program starts.

Feel free to follow our twitter feed and find us on Facebook somewhere, as the kids these days say...

Happy streaming!

Well, the cat has been out of the bag for a few days and I have been too busy to blog about it. But today as I wait for my...

I have been hacking on Paisley more recently. I actually got to hack during work time on Paisley, because the guys needed a feature I had developed - change notification. But more on that later.

I eked out a four day hacking session over this easter weekend, and my primary goal is to make some good advances on my music database system using CouchDB. The code is still in prototype stage, and I wanted to start removing hacks, tightening things down, and adding tests. But suddenly I found myself having to add a bunch of unicode() calls on data coming back from paisley just because I was being stricter on the input to paisley functions.

I didn't want to have to deal with unicode, again, as it detracted me of the core of my application. But I didn't like paying the technical debt either of not understanding what was going wrong under the hood.

From my limited understanding, JSON is an object notation format used for exchange of information between processes and applications. A JSON string is in unicode, which is great. It would be pretty useless otherwise in today's world. So I should be able to send in unicode to JSON libraries and get unicode back out.

A recent change in Paisley by one of the maintainers prefers simplejson over the stdlib-json. I first thought this change was to blame for my problems.

And yes, when decoding a JSON object, text was returned as str instead of unicode objects. Now, this is only when the text is in fact ASCII and hence works fine both as str and as unicode. And I'm sure opinions will differ here - but I think that a JSON library should *always* deserialize text to the same type of object by default - ie, unicode.

Clearly, simplejson disagrees with me. But I didn't have this problem a few weeks ago, so something changed! What gives? And changing back to json over simplejson didn't fix it either!

After some googling, I stumbled upon this bug report. Apparently, in 2.7, the C-based implementation deserializes ASCII text as str instead of unicode. The Python-based one always returns unicode for text. And in previous Pythons, both always returned unicode for text.

In essence, my problem boiled down to this:


[thomas@ana ~]$ ipython
Python 2.7 (r27:82500, Sep 16 2010, 18:02:00)
Type "copyright", "credits" or "license" for more information.
IPython 0.10.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]: from json import decoder
In [2]: decoder.py_scanstring('"str"', 1)
Out[2]: (u'str', 5)
In [3]: decoder.c_scanstring('"str"', 1)
Out[3]: ('str', 5)


versus


[py-2.6] [thomas@ana ~]$ python
Python 2.6.2 (r262:71600, Sep 29 2009, 21:49:07)
[GCC 4.4.1 20090725 (Red Hat 4.4.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from json import decoder
>>> decoder.py_scanstring('"str"', 1)
(u'str', 5)
>>> decoder.c_scanstring('"str"', 1)
(u'str', 5)


Note how the last command returned a normal str object.

And yes, in the past few weeks since I last tested this, I did indeed upgrade my machine to Fedora 14, pulling in Python 2.7

simplejson seems to always deserialize to str when it can. I would consider that a bug - ie, 'be strict in what you produce'.

As for Paisley, I made a feature-unicode branch on github, and this commit introduces a compatibility pjson module. By default it is STRICT, ie it wants unicode back always, and tests for the buggy behaviour, and does an alternative loads implementation that falls back to the python one. I'm sure some Paisley devs will prefer simplejson still, so you can change STRICTness and prefer simplejson.

Now, back to the hack.

I have been hacking on Paisley more recently. I actually got to hack during work time on Paisley, because the guys needed a feature I had developed - change notification....