Present Perfect

WARNING: This is a half year old entry that I've only now come round to finishing. Wherever it says "this weekend", mentally replace with "when it was still cold and rainy".

This weekend I had one of my better yak shaves. I've been having trouble with my home network mail server setup ever since Telefonica upgraded part of their network, because my postfix would tell me that my username did not exist. (Every time I had called them to ask I had gotten a different story on the problem, until a woman caved in and explained that they had done an Exchange upgrade and they lost a chunk of the mailboxes, as well as a lot of the authentication data).

After reinstating my account, I tried to get my Postfix SASL authentication working again for the past few weeks. No matter what I tried, it will still deny me to send any mail from an address that was not @telefonica.net, while before this has always worked fine. For some of the authentication mechanisms (CRAM-MD5) I think, SASL was throwing errors, so I even went as far as downloading and rebuilding SASL - but that was another dead end.

So this weekend I decided I wanted to start playing and experimenting with the postfix configuration, possibly upgrading, and keeping track of what I do (because I always manage to back myself into a corner by tweaking too much). So obviously, I felt I should use a version control system for this.

I've been wanting something that does this for a long time - tracking all my custom changes to config stuff and other files, and being able to easily go back in history and see what I had done before. Given that I mostly use subversion these days and like the nice integration with Trac, I wanted something based on subversion.

So, again like before, I look for projects that use svn and make it easy to store /etc stuff in it.

I found something that didn't really suit my needs, and searched other sites (like freshmeat) for other similar projects, but I didn't find anything useful. So I set out to make my own.

In one day, I had a simple prototype knocked together that would mostly use the svn binary for its functionality. I had made it simple on myself by just layering the svn checkout on top of the live system; it felt a little unclean to have a .svn directory in / but I decided I could live with it for the time being. I used svn properties for tracking permissions, owner and group information (by name, not by id), because for configuration data this is important to have. I proceeded to put parts of my system config for my home server in svn, hooked up a trac install so I could browse it easily, and played with postfix, commiting step by step.

Two weeks after that the hard drive in my server crashed, which was bad, but one gets used to it. It was a lot better than all the other times, though, since this time I had more than half of my custom config in an svn repository, and after reinstalling the machine with FC5 (it was FC3 before), I did a checkout and merged in my old config.

Fast-forward to now. (We're in the present again, no matter how brief the present really is, despite what the top of this post says).
Over the last half year, I had started using the tool for administering our work servers as well. Then, when I started thinking about how I was going to administer a 7-machine development platform and a 15-machine-and-rising online platform, I looked at my tool again and I discussed it with my collagues.

A general disapproval about having .svn directories everywhere was expected and expressed. Andy however found a more serious flaw with this approach. As most people know, svn keeps a complete copy of the repository version of each file in the .svn directory. So, while some file in /etc (say, /etc/shadow) may be 400, readable only by root, the copy in /etc/.svn would still be world-readable. Not a good thing

After some thinking, I decided I wanted to redo this tool from scratch, and also consider some other features that would help me use this tool for our platform deployment. The most important one I wanted was to have a concept of an overlay managing a part of the file system, using multiple layers. The layers would then combine as if they were stacked, which would be similar to how I understand things like UnionFS to work (I haven't tried UnionFS myself, only read about it). The layering would allow me to have parts of the config checkout be related to the machine's role, part to the specific host, and part to all hosts together.

Over a weekend, I reworked the tool in a branch, adding a decent testsuite as I went, and added the overlays and layers support. It's now close to ready for a first release, and I have used it to roll out our platforms since then, and besides a few quirks here and there, it is working out very well. The testsuite achieves 85% coverage, which is a lot considering I didn't put in a lot of effort to add tests after the usual add test/fail/implement feature/succeed cycle.

So, yay for the beauty of yak shaving. For those interested, the tool is called "savon", and you can browse trunk, or do a checkout and start playing with it. I will put up more info later when I make some more time for it, but feel free to ask me questions in the meantime if you're interested.

WARNING: This is a half year old entry that I've only now come round to finishing. Wherever it says "this weekend", mentally replace with "when it was still cold and...