Present Perfect

This week at work we ran into a problem where one of our Python processes was consuming close to 3 GB of memory because it's not properly cleaning up a list. Because of other bugs this process could not be easily restarted without triggering other problems, so our core team asked for some suggestions and I told them "Why don't you try cleaning up the Python list using GDB and the Python C API ?" I had a vague recollection of someone on our team doing something like this a few years ago.

I also asked them to blog about it, because there aren't that many resources readily findable on the subject.

So here is Andoni's take on the problem.

If any Pythonista can suggest how he could have avoided the segfault during garbage collection, please let us know!

This week at work we ran into a problem where one of our Python processes was consuming close to 3 GB of memory because it's not properly cleaning up a...

I'm doing a webinar in a couple of minutes on Video In The Cloud: Live and On-demand Encoding and Delivery. A bit late to announce that if you want to catch it (I've been busy, sorry), but it will probably get recorded and be made available later on.

I'm there with people from Amazon, Sorenson and Zencoder, so that should give a good Q&A session.

It's a challenge doing that from Europe at the moment - I left the office past 23, and Neil even later - but it's a good opportunity to talk about the things we're working on.

I'm doing a webinar in a couple of minutes on Video In The Cloud: Live and On-demand Encoding and Delivery. A bit late to announce that if you want to...

Well, the cat has been out of the bag for a few days and I have been too busy to blog about it.

But today as I wait for my team to do a final deploy fixing a bug with too-long URL names for Flash Media Encoder, I have some spare time to mention what's going on and make some people an offer they cannot refuse.

So, for the past half year of so we've been hacking away at a new service to solve a very specific problem in streaming. From 2005-2010 the streaming world mostly settled on Flash as a common platform, which was an unstable equilibrium for everyone involved, but it seemed to work. However, with the amount of codecs, devices and platforms there are today, this equilibrium has been falling. The introduction of iPhone, Microsoft's heavy pushing of Silverlight (paying companies to stream in it - and funnily enough those companies usually stop using Silverlight when the money faucet closes), GoogleTV, the introduction of WebM, the arrival of HTML5 (ironically pushed by Apple - yay - even though their HTML5 sites usually only work in Safari - boo)... all these movements served to upset the status quo once again.

To the eye of the casual observer, it would seem that all streaming has standardized on H264, and so transmuxing technologies are popping up - taking the same video encoding and just remux it for different technologies. However, in practice, H264 is a collection of many techniques and profiles, different levels of complexity, and not all devices support the same profiles and techniques. If you want to stream to all H264 devices with just one encoding, you'll have to settle for the least common denominator in terms of quality, and you'll have to pick a resolution that works subpar for all of them.

Now, content producers hate this sort of situation. They just want to get the signal out there, because that's what matters. The codec and the streaming is just the technological means to get it across the internet. And now the market is asking them to put a bunch of machines in their facilities, learn a lot of technologies they'd rather not worry about, consume heaps of bandwidth to send each version online, and then have to do it all over again each time something changes out there - a new codec, a new device, a new favorite resolution, ...

Our answer to this problem is simple: send us one encoding, we will do the rest. Our service will take your live stream, transcode it to as many different encodings as you want, and hand them off to a CDN. That's basically it. Want full HTML5 coverage ? We'll do it for you - H264 single and multibitrate, Theora, WebM, and a Flash fallback. Want Silverlight, Flash RTMP, Windows Media MMS ? All there.

Services like this already exist for ondemand - see zencoder and encoding.com and Panda. Live is just inherently more difficult - you don't get to work with nice single finished files, and it has to happen right now. But this is exactly the sort of thing a framework like GStreamer is good for.

In reality we aren't doing anything new here - Flumotion runs a CDN that already provides this service to customers. The difference is that this time, you will be able to set it up yourself online. A standard integration time with any CDN is around two weeks. This service will cut that time down to five minutes. We're not quite there yet, but we're close.

What's that you say ? Something about an offer ? Oh, right. It's always pained me to see that, when we wanted to stream a conference for free, it was still quite a bit of work in the setup stage for our support team, and hence we didn't stream as many conferences as I would have liked to. Similarly, it pains me to see a lot of customers not even considering free formats.

So the offer is simple. If you are running an event or a conference that flies under a Free/Open banner, and you're willing to stream only in free formats (meaning, Theora and WebM), and you're willing to ride the rough wave of innovation as we shake out our last bugs, we want to help you out. Send us the signal, we'll do the rest. Drop me a line and let's see how we can set it up. Offer limited, standard handwavy disclaimers apply, you'll have to take my word for it, etc...

If you're in the streaming industry, I will be demoing this new service next week on Wednesday around 2.00 pm local time in New York City, at Streaming Media East. And after that our Beta program starts.

Feel free to follow our twitter feed and find us on Facebook somewhere, as the kids these days say...

Happy streaming!

Well, the cat has been out of the bag for a few days and I have been too busy to blog about it. But today as I wait for my...

Our team is working on this great new system. It's really cool even though I can't say much about it. They've been surprising me with some inventive stuff. I saw a cool testing tool last month that allowed simulating the whole system on one machine, and it worked, and it detected bugs in the code that they were then able to fix and test again.

But we can't find a good QA engineer to come help us make this thing really rock. Most QA curriculums I get are written in .doc or .docx, show the person having only very vague Unix knowledge, and think Python is a snake best stayed away from.

We contracted a QA consulting company, and after a four day audit they concluded that we were not doing a simple web framework and they did not have the skills to test our system.

I'm sure there's a great QA engineer out there eager to test our multi-datacenter multi-server cluster system doing cool stuff, with some QA experience, some SCRUM experience maybe, some Linux experience please, and some Python experience if possible. But most of all, with a desire to learn, and some capability to start out on his/her own in one project and slowly grow a QA team from the inside out.

Come on, I know you're out there. Send us a mail!

If you're interested, here's the job description.

Our team is working on this great new system. It's really cool even though I can't say much about it. They've been surprising me with some inventive stuff. I saw...

Had a good time at the OpenVideo Conference in New York City, as well as today at (cut short for me) FOMS.

I took the Amtrak train from NYC to Boston and was looking forward to doing four hours of hacking, knocking up a quick prototype of doing chunked streaming of Ogg and WebM.

I didn't actually get to that point however because my test stream triggered the streamer going lost, meaning that its main loop is stuck. So attaching gdb I was reminded how the 'pystack' macro in gdb (which was incredibly useful - it prints a python stack) hasn't worked for me for the last couple of years.

And this time I set out to properly fix it. If only the docs on gdb were easy to understand. My knowledge of gdb doesn't go much beyond the standard 'thread apply all bt' stuff and doing simple inspection. I ended up understanding why the macro hangs (it compares the value of $pc to known function names, and goes up the stack until it finds PyMain. In most of my cases there is no PyMain however, and when you go 'up-silent' in gdb at the top of the stack, it just silently fails leaving you at the same frame forever).

I don't claim to understand exactly how gdb works; for example, there are cases where $pc stays the same going up a frame, seemingly because a function is wrapped into a macro which seems to get its own frame. So I ended up writing a loop that goes up until the $pc changes, and then go back down, leaving me at the frame that calls a python method with a code object.

So, now I have my pystack working again in the cases I tested, so that's good.

Incidentally, if anyone knows gdb well enough, here are some questions:

• Is there a way I can detect in a script that 'up' didn't actually go up ?
• Is there a programmatic way to know how many frames there are, and what frame number you're on ?

After fixing that, I was again bitten by a bug where - not always, but often - a flow with Vorbis and Theora has a timestamp/duraiton discontinuity of 1 nanosecond. I can't reproduce it on the command line with -launch, so I think it's related to a set_base_time call on the pipeline.

So I started writing a unit test, but for some reason make check in gst-plugins-base doesn't work for me because it can't find capsfilter and other core elements. So I get to dive into the test setup code again to figure out why a registry is getting generated that blacklists coreelements...

The train pulled into Boston as I was looking into how the new registry code works and why it blacklists coreelements, so I'll have to save this test for a later day...

Incidentally, I have a day off in Boston this week. I really want to finally go to MIT and follow a class there, in computer science or mathematics. Is that sort of thing open in the US ? Can you just sit in on a class ? Any tips ?

Had a good time at the OpenVideo Conference in New York City, as well as today at (cut short for me) FOMS. I took the Amtrak train from NYC to...