Present Perfect


Picture Gallery
Present Perfect

Rebuilding older rpms

Filed under: Fedora — Thomas @ 1:56 pm

1:56 pm

Today I needed an xml diff tool. There seem to be an xmldiff and a diffxml, neither of them packaged by Fedora at the moment. I found an old src.rpm for Fedora 6 for xmldiff 0.6.8

The src.rpm doesn’t rebuild as is for various reasons: its %check stage imports unittest2, so I disabled check. Then it didn’t find debuglist.files, so I disabled building a debug package. And then it found an installed egg file which it didn’t like, so I disabled checking for installed files.

Since I’m going to forget how I did this in the future when I will need this again for some obscure reason, and because if you ever build rpms you may find this useful, here is the command that did it for me for Fedora 15:

rpmbuild --rebuild -D '%check exit 0' -D 'debug_package %{nil}' -D '_unpackaged_files_terminate_build exit 0' xmldiff-0.6.8-1.fc6.src.rpm

Now, back to comparing xml files.

GStreamer 0.11 Application Porting Hackfest

Filed under: Conference,Flumotion,GStreamer,Hacking,Open Source — Thomas @ 11:16 am

11:16 am

I’m in the quiet town of Malaga these three days to attend the GStreamer hackfest. The goal is to port applications over to the 0.11 API which will eventually be 1.0 There’s about 18 people here, which is a good number for a hackfest.

The goal for me is to figure out everything that needs to be done to have Flumotion working with GStreamer 0.11. It looks like there is more work than expected, since some of the things we rely on haven’t been ported successfully.

Luckily back in the day we spent quite a bit of time to layer parts as best as possible so they don’t depend too much on each other. Essentially, Flumotion adds a layer on top of GStreamer where GStreamer pipelines can be run in different processes and on different machines, and be connected to each other over the network. To that end, the essential communication between elements is abstracted and wrapped inside a data protocol, so that raw bytes can be transferred from one process to another, and the other end ends up receiving those same GStreamer buffers and events.

First up, there is the GStreamer Data protocol. Its job is to serialize buffers and events into a byte stream.

Second, there is the concept of streamheaders (which is related to the DELTA_UNIT flag in GStreamer). These are buffers that always need to be send at the beginning of a new stream to be able to interpret the buffers coming after it. In 0.10, that meant that at least a GDP version of the caps needed to be in the streamheader (because the other side cannot interpret a running stream without its caps), and in more recent versions a new-segment event. These streamheaders are analogous to the new sticky event concept in 0.11 – some events, like CAPS and TAG and SEGMENT are now sticky to the pad, which means that a new element connected to that pad will always see those events to make sense of the new data it’s getting.

Third, the actual network communication is done using the multifdsink element (and an fdsrc element on the other side). This element just receives incoming buffers, keeps them on a global buffer list, and sends all of them to the various clients added to it by file descriptor. It understands about streamheaders, and makes sure clients get the right ones for wherever they end up in the buffer list. It manages the buffers, the speed of clients, the bursting behaviour, … It doesn’t require GDP at all to work – Flumotion uses this element to stream Ogg, mp3, asf, flv, webm, … to the outside world. But to send GStreamer buffers, it’s as simple as adding a gdppay before multifdsink, and a gdpdepay after fdsrc. Also, at the same level, there are tcpserversink/tcpclientsrc and tcpclientsink/tcpserversrc elements that do the same thing over a simple TCP connection.

Fourth, there is an interface between multifdsink/fdsrc and Python. We let Twisted set up the connections, and then steal the file descriptor and hand those off to multifdsink and fdsrc. This makes it very easy to set up all sorts of connections (like, say, in SSL, or just pipes) and do things to them before streaming (like, for example, authentication). But by passing the actual file descriptor, we don’t lose any performance – the low-level streaming is still done completely in C. This is a general design principle of Flumotion: use Python and Twisted for setup, teardown, and changes to the system, and where we need a lot of functionality and can sacrifice performance; but use C and GStreamer for the lower-level processor-intensive stuff, the things that happen in steady state, processing the signal.

So, there is work to do in GStreamer 0.11:

  • The GStreamer data protocol has not really been ported. gdppay/depay are still there, but don’t entirely work.
  • streamheaders in those elements will need adapting to handle sticky events.
  • multifdsink was moved to -bad and left with broken unit tests. There is now multisocketsink. But sadly it looks like GSocket isn’t meant to handle pure file descriptors (which we use in our component that records streams to disk for example)
  • 0.11 doesn’t have the traditional Python bindings. It uses gobject-introspection instead. That will need a lot of work on the Flumotion side, and ideally we would want to keep the codebase working against both 0.10 and 0.11 as we did for the 0.8->0.10 move. Apparently these days you cannot mix gi-style binding with old-style binding anymore, because they create separate class trees. I assume this also means we need to port the glib2/gtk2 reactors in Twisted to using gobject-introspection.

So, there is a lot of work to be done it looks like. Luckily Andoni arrived today too, so we can share some work.

After discussing with Wim, Tim, and Sebastien, my plan is:

  1. create a common base class for multihandlesink, and refactor multisocketsink and multifdsink as subclasses of it
  2. create g_value_transform functions to bytestreams for basic objects like Buffers and Events
  3. use these transform functions as the basis for a new version of GDP, which we’ll make typefindable this time around
  4. support sticky events
  5. ignore metadata for now, as it is not mandatory; although in the future we could let gdppay decide which metadata it wants to serialize, so the application can request to do so
  6. try multisocketsink as a transport for inside Flumotion and/or for the streaming components.
  7. In the latter case, do some stress testing – on our platform, we have pipelines with multifdsink running for months on end without crashing or leaking, sometimes going up to 10000 connections open.
  8. Make twisted reactors
  9. prototype flumotion-launch with 0.11 code by using gir

That’s probably not going to be finished over this week, but it’s a good start. Last night I started by fixing the unit tests for multifdsink, and now I started refactoring multisocketsink and multifdsink with that. I’ll first try and make unit tests for multisocketsink though, to verify that I’m refactoring properly.

using xargs on a list of paths with spaces in a file

Filed under: Hacking,sysadmin — Thomas @ 7:18 pm

7:18 pm

Every few weeks I have to spend an hour figuring out exactly the same non-googleable thing I’ve already needed to figure out. So this time it’s going on my blog.

The problem is simple: given an input file listing paths, one per line, which probably contain spaces – how do I run a shell command that converts each line to a single shell argument ?

Today, my particular case was a file /tmp/dirs on my NAS which lists all directories in one of my dirvish vaults that contains files bigger than a GB. For some reason not everything is properly hardlinked, but running hardlink on the vault blows up because there are so many files in there.

Let’s see if wordpress manages to not mangle the following shell line.

perl -p -e 's@\n@\000@g' /tmp/dirs | xargs -0 /root/hardlink.py -f -p -t -c --dry-run

N900 life support

Filed under: Hacking — Thomas @ 10:35 am

10:35 am

I don’t want to simply be complaining about Nokia’s sad Linux story ending.

It’s obvious that things aren’t going to get better though for Maemo device owners.

Here are two things that caused me trouble over the last few months, and the fix that did it for me, in case you were suffering from the same problems.

      The Facebook photo sharing functionality just stopped working for me. It would upload the file, then give me an error without specifying any reason. I had simply assumed the Facebook API had changed, and since afaik this plugin is sadly closed-source (what on earth possessed Nokia to make a social media sharing component closed to begin with is beyond me – what kind of IP secrets could you possibly have in there?) I thought I would have to do without for now. And it really is a hassle to manually copy photos off then share them from the desktop.

      Turns out that I simply had to re-authenticate the Sharing Account in Settings. No idea why – maybe Facebook changed some authentication system in the last few months ? Now it works again.

      (As a side note, this plugin’s behaviour is really annoying when it comes to uploading photos. It will always try to upload as soon as you connect to a network, although usually all you get is access to some web page on which you have to authenticate, usually by paying, to get on to the net. The sharing plugin already blasts photos at facebook, then fails, gives you a non-useful error message, and then sits there forever without any option to retry. All you can do is cancel the transfer, in which case you will have to re-upload the photos from your library. After some time I figured out that a reboot caused it to retry all pending uploads on the next network connect, and then after that I figured out a kill of a sharing manager process did the same thing. But really, Nokia engineers – a simple ‘retry’ button was too hard ?)

      For the last two weeks my GPS stopped getting a lock completely. This wreaked havoc on my barriosquare/foursquare checkins as well, which simply don’t register without a GPS lock (yes I still have a half-done port of bsq to the new foursquare API, but last time I tried I was still stuck on the simply terrible browser coming with the device that seems to be unable to properly complete SSL requests in emulator mode).

      I stumbled across this page and simply changed the AGPS server to google’s. Worked like a charm on the next connect. Nokia, I don’t know what you did to your AGPS server – surely other phones you have are using it too, not just the Maemo ones ?

      I swore I wasn’t going to buy an N9 because there’s no point in buying an EOL’d phone if I plan to develop for it. The reviews when it actually came out almost persuaded me to get it, and the five minutes I got to play with Luis de Bethencourt’s phone got me really close. I’ve even seen plans in Belgium offering this phone! But really, there isn’t much point if Nokia isn’t going to support this phone any more and services are just going to get worse, and important parts of the stack remain closed and thus unfixable down the road.

      This month’s Android course at work at least got me familiar developing for the phone and I was actually impressed by Eclipse this time around, and while Java still seems like a bitch to program in, the whole emulator setup is easy to use… Who knows, my next phone may in fact be an Android.

      In the meantime, it’s nice to see that some of my fixes go noticed. That motivates me to possibly fix that other annoyance in erminig-ng – all-day events triggering an alarm at midnight and waking me up :)

ssh friction

Filed under: Hacking,sysadmin — Thomas @ 12:57 pm

12:57 pm

I haven’t been too good this year at removing friction from my workflow. Today I wanted to change that. And the random friction thrown my way today has to do with ssh.

You see, somewhere along the line I read that it is a good idea to create separate keys for separate identities. So I have an identity for all work-related stuff (which I consider ‘ring 1’: it’s unlikely to change but everyone can get fired or change jobs), one for personal stuff on machines I actually control (‘ring 0’: they’d have to pry it out of my dead hands), another for my ‘public online default’ identity (‘ring 2’: I can always pull a whytheluckystiff and pull myself of the net and reinvent myself), and then per-project identities (‘ring 3’: I may lose interest in being a fedora or gstreamer contributor without massive changes in my personality).

I started splitting ring 3 per project when it made sense – for example, Fedora recently enforced a key change even if your account wasn’t compromised and even if you already have a strong passphrase on your key (like I had), and of course a massive flamefest ensued. I shrugged and decided to split off a new key and set that on all my machines.

But the problem is, this whole tower of ssh doesn’t really work well in practice. I chose a long passphrase for the new fedora keys, but obviously I do not want to type that every time I clone a package or commit changes. So I use ssh-agent. In theory, ssh-agent adds your keys and asks you for the passphrase once, and is then able to offer those identities to the other side.

The problem is a lot of ssh servers out there only give you a few tries. So your ssh agent will offer identity by identity until it gets refused. If my fedora identity was added as the fourth identity I lose – I can’t clone a package.

Specifying IdentityFile in the ssh config is useless. It is poorly documented, but IdentityFile files actually come after your ssh-agent identities. So your agent blasts all the wrong keys at the host first, and you get denied.

So you can specify IdentityOnly to make sure that only the identity file you want is being used. Sadly in that case it will not use the ssh-agent at all, so it will ask you for the password to your key file – the whole reason you want agents to be used in the first place.

Now obviously ssh has all the pieces it needs to Do The Right Thing. If my config says to use this identity and this identity only, ssh should be able to request ssh-agent to present that identity, and that identity only, and make the login happen without any password.

Surely I must be missing something obvious. Surely one of you uberhackers out there has set up the same thing as me. Why don’t you comment about it here and help the rest of us?

« Previous PageNext Page »