Present Perfect

Algernon recently posted a reply to Christian's claim that flumotion is the best choice for streaming. (This is of course an example of Christian's youthful enthusiasm, crystallized in his "Promote early, promote often" motto.) He did make some good points, so I wanted to address them. I just wanted to take the time to give the anwer the time it deserves, so I'm a bit late.

Let me start by saying something that needs to be said: Icecast is *GREAT*. It's been around for quite some time, and back in the day when I was a student (my personal version of "When I was in the army...."), we put our university radio station on the Net with it as soon as it worked well enough. We were definitely the first in Belgium to stream live radio on the net, and probably the first in Europe, though it's hard to verify such a claim because there is very little info on this. But this was around 1995, I think (geez, where has the time gone :)) Also, one of the most active current hackers, Karl Heyes, is a very nice guy, always willing to help, and has given me some useful insight. Icecast has a good community.

I'll say some more about icecast later, but let me first address algernon's points and offer suggestions on how they can be done.

• Flumotion does not currently support more than one stream on the same port.

  This is correct, but a pure implementation choice because we were lazy and knew this is easy to fix. It's just that there are so many easy ways to choose from to fix this :) The main limitation here is that only one process can listen on a port. As soon as I get round to this, I'll do the most straightforward fix: have the streamer component take feeds from multiple other components, have the main thread listen to the port, and have Twisted serve one Resource per feed, on the same port, but different URL's. It's not a hard thing to do, it just needs doing. Estimated developer time: 4 hours.

• This also means - as far as I understand it - that if I want multiple streams, I need multiple flumotion instances.

  Not really - you start just one flumotion-worker binary on that server. This worker binary can fork multiple components all by itself. So you just put some more streamer components in the flow, as many as you like. Only one binary to manage. Unless by "instance" you meant to include these forks as well, but this is standard practice - look at Apache for example.

• There's no easy way to send a stream to flumotion, from a random GStreamer application. (...) For this, I would need to learn python, which I don't intend to anytime soon.
  Actually, there is. Here's what you can do to achieve this:
  1. start up Flumotion and the GUI
  2. set up a stream with only a video test producer
  3. verify that runs correctly and export the XML config
  4. edit the XML config, replace the videotest producer with a base producer type that takes pipelines
  5. use tcpclientsink for the pipeline with a given port and the GDP protocol
  6. restart flumotion with this new config
  7. producer component being hungry - nothing is feeding it data yet
  8. start up the following GStreamer pipeline: gst-launch videotestsrc ! video/x-raw-yuv,format=(fourcc)I420, framerate=5.0,width=320,height=240 ! tcpclientsink port=3333 protocol=gdp

  And there you go ! Total implementation time: 13 minutes, including feeding the cat (and at 9 am in the morning). Number of lines of Python code hacked: 0.

  I was surprised myself actually, I would have expected to have to fix at least one bug in the code. Thanks for prodding me to actually try this out. Now obviously this is not the "best" solution. If you stop the producing client, you need to restart the producer component as well, because the simple pipeline component type has no way to recover from the broken connection. And that's where you move from "quickly test something out with what's available" to "turn it into a 15-line component". Exercise for the reader.

• Flumotion does not have fallback mounts yet..

  IIRC, the fallback mounts can only be whatever the same icecast server is serving. So with the solution to the "multiple URL" problem, this can be done immediately as well. Twisted doesn't care *what* thing is providing a Resource tied to a path. So we'll probably get to this soon.

• With icecast, I can easily make relays.

  IIRC, Icecast has two types of relay. One is where you import a stream from another server that can be any HTTP streaming server: Icecast, Shoutcast, ... This one is easy to implement: you use a pipeline producer that uses gnomevfssrc to get the stream. There's only one catch at the moment - there needs to be a GStreamer element to parse the Ogg stream properly so that it can figure out what the streamheaders are. oggdemux ! oggmux should do this, but it has a bug, which I noticed while trying this just now. So, that's the fix needed. (Estimated developer time: 6 hours)

  The other mode is icecast-specific: icecast pulls in an .xml file from the other one that describes all streams, and then relays them all automatically. This is also doable, but would mean including icecast-specific support. This is something we'd really like to do - make Flumotion work really well with icecast. I see icecast more as an ally than as competion. The competition we see is Windows Media Server, RealServer, Quicktime Server (and HelixServer and Darwin Streaming Server). I'd have to look at exactly how this relay info is pulled though to say how long it would take to do this.

  (Zaheer reminds me that in Flumotion itself, it's of course even easier to do relays. You can create as many streamer components in your flow as you want, connecting to any other producing component, and the streamer can be on any machine in your network. And all of that is managed from the GUI. So in Flumotion, it's easy to do relays. We still need to make something nice though to go through firewalls transparantly - an advantage Icecast currently has because the clients are push-based).

• Icecast has this nice stuff with me being able to apply random XSLs to its XML output.

  If you tell me what this means, I'll cook this up for you. Estimated developer time: 1.5 hours

• Icecast has a usable C library that can be used to feed it

  As shown above, GStreamer is that library.

• I couldn't, for the life of me, get flumotion into a usable state under FreeBSD. It's not flumotions fault, per se, but of its dependencies.

  Ok, this is a point where I might get defensive and ranty, because it's not a fair point IMO. We all have limited resources and time. Most focus is spent on Linux, and this is the same with many projects. BSD users know this and are typically more used to - and apt to - fix problems themselves and get stuff installed. We know GStreamer and gst-python runs on FreeBSD. So does Twisted. We haven't tested it ourselves, we don't run FreeBSD. But it can't be too hard. You can't get a lot more portable than Python.

  Of course, if you don't tell us exactly where the problem is, there's nothing we can do to help. And we'd love to help it get set up. But it's not fair to blame a vague "I can't install this on FreeBSD" problem on Flumotion :) Talk to us and we'll help you get it fixed.

• While I think flumotion has a damn good design, I think icecast follows my ideals better. As the UNIX paradigm goes: it does one thing, and does that well. It doesn't try to be a complete streaming solution. It is simply a streaming server. And that I love. I know that flumotion is usable this way too (or will be, as soon as there is a flumotionsend element), but it does stuff I do not need at all. (...)That makes it harder to use for what I need, and makes the developers focus on things I will never ever use, so it lacks key features I do need.

  There are pros and cons to going either way. We think in the end that a product that can carry the task through from beginning to end is better. It's better integrated, it has a unified feel, and it allows you to do more powerful stuff. If one of my icecast streams is down, I need to leave the Realm of Icecast and go see what happened to my application producing the data. In Flumotion you just see in your GUI (which soon will have a notification area applet) what's up. However, you still can use it in "be only a server" mode, and yes, we want to make that work correctly. Saying that we focus on things you will never use, does not automatically mean we should lack key features you do need. You don't seem to be streaming AAC, yet icecast has spent time on getting that working.

• Oh, and you might probably ask why I didn't report these to the flumotion developers. That's a good question, and I admit I should have done that. However, I had deadlines, and that didn't allow me to properly report issues.

  This is indeed a crucial point. We're all strapped for time. You had a deadline so you didn't inform us of these problems you had. We have deadlines too. Yet here I am making the time to address your problems, even if it's after the fact. We care about our project. We care about people using the project. If they have a problem, we want to help them. It doesn't take much to get us to go out of our way to do so. Here I am replying to your points. Someone recently wanted to try it on SuSE and he was missing deps and built stuff by hand and something below the stack segfaulted. So I started looking for someone to make SuSE packages and now we have them (I still need to test them though, but VMWare seems to lock up when trying to install SuSE).

  You're in our IRC channel, you see us discuss. Feel free to bring up these points when you run into them, even if it's just cursory or you don't have time to go in details.

So here's the simple point: these issues you have you can let us know on IRC, on the mailing list, in our issue tracker, and we'll comment on them. When faced with our own deadlines, we don't spend the time to psionically probe all people for problems they might have with Flumotion. But we still spend time addressing valid points about it when we know what they are.

But don't take my word for it ! Try it out. Give us some stuff you want to see happen. And if you prefer to wait until we're done on our own, that's fine too. Let's meet back in a year and see where Flumotion is, feature-wise.
Flumotion is almost a year old. In software land, that's young. For a normal use case (which you indeed aren't, since you prefer to hack yourself and code up your own streaming source), I do think it's a better match than Icecast.

You don't really need relaying, you can have multiple streamers streaming the same stream, for the same effect, except that you have one less failure point. You have a nice GUI and wizard that makes it easy to set up. And the design underneath is flexible enough to handle more complex tasks, and the code is simple enough to get started with it quickly.

Now, back to icecast. I haven't used it much lately because I do not work for radio stations anymore since I started for Fluendo. The friend I originally did the streams with now works for an ISP, where they have 5 round-robin icecast servers with about 1200 streams each. He says the machines are easily pulling 60% CPU and more, which is why he needs to round-robin in the first place. One of our customers is using Flumotion to stream to 4000+ clients, pushing 80 Mbit. The CPU usage is below 5%. I wish I could test icecast in these conditions myself to see where the difference lies. Maybe they use an inefficient polling method ? Maybe something else is completely different ? But when he asked about it, they told him that they don't really have many users that serve that many streams, and they don't focus on those conditions.

One other thing that annoys me is that, while Icecast is closely affiliated with Xiph.org, it has always supported mp3 streaming, and now has added AAC streaming. You can sort of make a point for mp3 - it was around before Vorbis, it was the first thing icecast supported, ... But adding AAC does not gel well with Xiph.org's mission. Flumotion from the start has made it clear that we do not only focus on free formats. We can't do that, business-wise. But we actively support Xiph.org and the codecs they make, and we make sure they are first-class citizens in our world. And that this support is available for free. If you want non-free codecs support, you'll get non-free code. I think that's fair, and promotes free codecs better than Icecast does. We make people aware that there is a cost involved, both in money and in freedom, when using non-free codecs. And we go out of our way to make it easy to avoid that cost by using free codecs.

Flumotion is an active and young project, and I hope this shows that it's a very flexible project, easy to extend, and that as developers we are very open to other people trying out ideas and suggesting stuff we haven't thought about.

Algernon recently posted a reply to Christian's claim that flumotion is the best choice for streaming. (This is of course an example of Christian's youthful enthusiasm, crystallized in his "Promote...