Present Perfect

moap is a swiss army knife for maintainers and developers.

This is MOAP 0.2.7, "MMM...".

Coverage in 0.2.7: 1424 / 1899 (74 %), 109 python tests, 2 bash tests

Features added since 0.2.6:
- Added moap vcs backup, a command to backup a checkout to a tarball that
can be used later to reconstruct the checkout. Implemented for svn.
- Fixes for git-svn, git, svn and darcs.
- Fixes for Python 2.3 and Python 2.6

I've been fixing things left and right for python 2.6, and in the process I noticed that moap hasn't had a release for over a year. This release contains mostly bug fixes collected over the year, and a new feature that isn't implemented yet for all VCS's. Basically it's an automatic replacement for something I was doing manually every time I removed an old GNOME cvs/svn/git checkout: figure out what's in that tree that's not in the repository (diffs, unversioned files, ...), so I can delete everything else and free some disk space.

The only problem with this release is that, after doing the release, I noticed that Freshmeat removed their XML-RPC interface. Apparently they have some new kind of interface they want people to use. Sigh. But that means 0.2.8 is right around the corner!

moap is a swiss army knife for maintainers and developers. This is MOAP 0.2.7, "MMM...". Coverage in 0.2.7: 1424 / 1899 (74 %), 109 python tests, 2 bash tests Features...

Last week, after upgrading my home desktop to F11, I had palimpsest tell me one of my disks was broken on the desktop machine. The desktop is running on two 250 GB drives in software raid. It was time to get new drives.

After a weekend of fiddling with new 1 TB disks for my home desktop, trying failure scenarios, making sure the system can boot from each of the two drives, and waiting for the 4 hour resync of the software RAID in between each step, I finally closed up the desktop machine and cleaned up under my desk again, thinking I was done with my halfyearly messing about with broken disks.

I guess I was tempting faith anyway. Doing a routine operation on my home server after all the configuration stuff I'd done to set up asterisk last week, suddenly an rsync aborted, a journal errored out, a partition changed to being mounted read-only, and the log was full of scary drive errors. Ouch.

Well, that's why I keep around a big box of old drives - for when some drive fails and I want to tempt fate even more by reusing an old drive that's probably going to fail real soon too. And anyway, I had just spent my hard drive piggybank on the new desktop drives.

Luckily, I seemed to have a 400 GB SATA drive lying around that used to belong to my media center. I don't remember why I swapped it out, given that the media center has a 160GB drive for the OS (and two 1.5 TB raid drives for the data, of course), but this was a lucky break. I booted with a rescue cd, and tried copying the root filesystem of my CentOS 5.2 home server partition to this new drive. Which worked fine, except that /var was where I triggered an Input/Output error and some more drive errors in the kernel log.

So, powered off, took out the broken drive, and put it in a USB chassis. The advantage of a USB chassis is that you can easily just replug the drive to try again, instead of locking up your system terribly and having to reboot. Sadly, /var was broken beyond repair. I ran an e2fsck hoping to recover the contents, and that partly worked, but some of the important stuff is missing even from lost+found (apart from the annoying situation where you have to reconstruct file names, which I usually end up not bothering with).

But really, how important can /var be ? Turns out, rather important. As in, you need it to boot in the first place. And also, it holds your rpm database. Crap.

Some Googling gave me some posts on how to reconstruct your rpm database from log files (using --justdb --noscripts --notriggers). But to use those, you actually need those log files. Where are those ? On /var as well. Crap. And they're not in lost+found either.

Ok, so time to get creative. Here's what I ended up doing:

• create /var/lib/rpm, and run rpm --rebuilddb to end up with an empty rpm database
• Based on the contents of /usr/bin, figure out what packages ought to be installed:
  
rpm -qf /etc/* | grep 'not owned' | cut -f2 -d' ' > /tmp/unowned
yum --enablerepo=c5-media --disablerepo=base --disablerepo=updates --disablerepo=addons --disablerepo=extras whatprovides `cat /tmp/unowned` | cut -f1 -d' ' | sort | uniq > /tmp/missing
yum --enablerepo=c5-media --disablerepo=base --disablerepo=updates --disablerepo=addons --disablerepo=extras install `cat /tmp/missing`


  This works by first listing all files that are not owned by rpm (on the first run, that's all of them), figure out what packages can provide these files, then installing those packages.

• Repeat the process for other important directories, like /bin, /sbin, /usr/sbin, /usr/lib, /usr/include, ...
• Clean up .rpmnew files that don't actually contain differences:
  
find / -name *.rpmnew | sed s/.rpmnew//g > /tmp/rpmnew
for c in `cat /tmp/rpmnew`; do echo $c; diff $c $c.rpmnew && mv -f $c.rpmnew $c; done


• Same for *.rpmorig:
  
find / -name *.rpmorig | sed s/.rpmorig//g > /tmp/rpmorig
for c in `cat /tmp/rpmorig`; do echo $c; diff $c $c.rpmorig && mv -f $c.rpmorig $c; done

• Inspect the remaining ones, and merge changes.

While it's not an experience I hope to repeat any time soon, it worked out surprisingly well!

Last week, after upgrading my home desktop to F11, I had palimpsest tell me one of my disks was broken on the desktop machine. The desktop is running on two...

I managed to completely skip updating to F10. All my machines (work desktop, home desktop, laptop, media center) where running F9 without any real problems I worried about.

But of course I was curious. And, especially with the move to python 2.6, things I care about where bound to break.

So, last weekend I took the plunge, and after little over a week here are my first impressions:

• Overall F11 looks slick. Nice work on the artwork! I particularly liked the GDM background, looking like an ancient brushed metal object, reminding me of how I used to love playing Gods by the Bitmap Brothers.
• Apparently anaconda now has bugzilla integration, allowing you to file a bug directly from inside anaconda. Luckily for Jeremy (who I assume still maintains it) it has some code in there to look for existing bug entries with the same backtrace. Very nice!
• Of course, I wouldn't have found out if I hadn't run into exceptions in anaconda. I ran into while setting up two completely new hard disks with 2 software raid partition and LVM on the second one.
• I first installed my work desktop, as usual putting the new installation on a separate partition, keeping my old one around in case the install goes wrong or F11 just isn't stable enough for me. For me, that involves having a /mnt/alpha and /mt/omega partition between which I alternate. At some point I should figure out if other people do this too and if it makes sense for anaconda to support something like this and at least allow me to keep my GRUB configuration for the older installation. For now I do this manually, using a hugeupgrade text file I follow each time I upgrade, accumulating more and more steps each time I perform the procedure.
• On my home machine, when I booted into F11, as usual my second monitor didn't work (I have a Radeon GeCube Pro 2400). My own fault really - I should have tried to get a patch upstream into the default radeon driver the same way I sent a patch for the radeonhd driver that I still use. A rebuild later, I at least had the old radeonhd driver rebuilt to get my second screen working again.
• Having the second screen now made me change my mind completely about the GDM wallpaper. That lion on the right hand side that I didn't see before completely ruins the style for me. Sorry!
• Upon logging in to the work desktop, I had no network. Completely puzzled as to why, until I figured out that I had to actually right-click on NetworkManager's tray icon, choose to configure, and activate eth0 by default. After some browsing it seems that this was a deliberate choice to increase security. While I can possibly sympathize with the motivation for doing so, it really is terrible to change this by default and not provide *any* indication during or after installation. At the very least, the following things could have been done:
  1. provide a clear notice during installation, and allow a user to choose to enable it anyway, assuming the security risk
  2. the same, but during firstboot
  3. after logging in, having the network manager tooltip say 'the network is disabled by default in this new release, here's how you enable it

  I am not entirely sure what the security problems are with enabling the network after installation. The default firewall is pretty locked down, SELinux is enabled by default, and there's no way I can install updates without the network anyway. But I'm sure that I could find huge bikeshedding threads on fedora-devel about this if I really cared why this was decided.

• Upon logging in to the home desktop, I was greeted with a tooltip saying that one of my drives was going bad. That was a nice touch! Really good idea to have something like that be monitored by default. This prompted me to ponder to finally replace my desktop's 250 GB PATA drives with real SATA drives - a story for another post.
• Various deprecation warnings pop up running various Python programs, including my own. Flumotion needed a patch for running against 2.6 (I rebuilt and pushed to F11). So I have some cleanup ahead, and I should revisit pychecker soon.
• The first piece of functionality I checked was Evolution's Google Calendar integration. It still seems a bit shaky, given that I had to restart Evolution a few times as it froze doing stuff with the net, but it does seem to work. That means I will finally be able to accept work invitations done through Outlook and get them on my Google Calendar! Awesome. Now if only I didn't have to manually configure each of the ten calendars I'm interested in...
• At work, when I played a video using XVideo, my machine instantly froze. Seems to be a known bug. The intel drivers are being rewritten. I've never quite understood why rewriting is an excuse for breaking stuff that worked (I should check if Firewire video finally works reliably now when I have the chance, for example), but all in the name of progress I guess.
• I don't know why it's happening, but once in a while my screens blank. Even in the middle of doing stuff. If I were a gamer I'd be hugely annoyed as my character would be shot through the head in that split instant. The closest bug I can find is this one, where I commented. Hugely annoying bug because I don event know how to begin debugging a bug like this that I can't catch in the act.
• PulseAudio integration in GDM seems a bit fragile. I have my pulseaudio configured to send audio to my media center pulseaudio server. Sometimes, after choosing a username in GDM, it doesn't manage to play the audio sample related to that action, and then GDM is stuck there not showing me the password entry dialog. Pretty sure it's due to blocking on pulseaudio, because when I kill it the password dialog appears. Pretty painful bug for new users though.

All in all, not a bad first week experience, and seems like a solid release. Now, off to rebuild bits and pieces, and clean up Python 2.6 deprecation warnings...

I managed to completely skip updating to F10. All my machines (work desktop, home desktop, laptop, media center) where running F9 without any real problems I worried about. But of...

After some situations at work this week where I lost time where I really shouldn't have had to, combined with the observation that I get more useful strategic work done at home in Belgium, and because practically speaking going to Barcelona next week would be silly given that I can only leave on Monday and Wednesday is a day off (which I loathe - San Joan, the most dangerous night in Barcelona), I decided to stay home next week and compensate by fixing my phone setup.

You see, the only really annoying thing is that any conference call I end up in is terrible because I have a really hard time hearing the other side through either my mobile or my fixed phone, as the audio cuts out several times a second.

So, I spent a few hours yesterday first setting up the VPN, which aside from some minor issues seems to be working fine now. This was apparently a prerequisite for setting up asterisk because asterisk needs a fixed IP address or something I've been told.

After that, I started setting up Asterisk so that I could use the same THOMSON phone we have at work from home and call people in the office over it.

All of that is not what this post is about though.

This post is about the TFTP tricks and things I always need to re-learn any time I meddle with tftp. I'm putting them here because Google usually doesn't find the problems and solutions I come up with, so maybe they're of use to you if you play with TFTP. They will definately be of use to me next time I mess with tftp.

• TFTP runs over UDP on port 69
• on Linux TFTP typically runs from xinetd. Do yourself a favour, edit /etc/xinetd.d/tftpboot and add -v -v -v to the server_args line. These lines should end up in your /var/log/messages
• For some reason xinetd is fidgety with tftp. It doesn't restart in.tftpd properly when you reload or restart xinetd, and so your verbose changes might not happen. Check with ps aux. You can kill it, but then xinetd doesn't seem to start up in.tftpd properly for a while either. Strange stuff - please tell me if you know what's going on here
• Keep a tcpdump running on your tftp server to see requests actually make it in: tcpdump | grep tftp
• Start by trying a tftp transfer on the server to localhost:tftp localhost -c get testIdeally, you should get Error code 1: File not found back immediately.
• Now try an nmap from another machine: nmap -sU -p 69 server which should come back with 69/udp open|filtered tftp. If it doesn't, you probably didn't open 69/UDP on your server's firewall. You can confirm by just turning off your firewall on the server for a quick test.
• If it shows as open|filtered, try tftp server -c get test. This should error out immediately as well. If it doesn't, it's probably because your test machine does not allow tftp in. Confirm simply by turning off your firewall. The simplest way to fix this is to load the tftp connection tracking module: modprobe nf_conntrack_tftp. This makes sure that your machine knows to accept the reply tftp request coming in on a random port. On Fedora/RedHat systems you can make this permanent by adding it int /etc/sysconfig/iptables-config to the IPTABLES_MODULES variable. This is the number one thing I keep forgetting when debugging tftp troubles.
• After that, try with actually existing files. Make sure you have the SEcontext correct; you can run restorecon -vR /tftpbooton the server for that. You can always confirm or deny whether SELinux is giving you trouble by temporarily turning it off. My auditd (the process that logs SELinux violations to /var/log/audit/audit.log) sometimes stops logging properly to the log file, and I need to restart it in that case. It's easy to spot when auditd is misbehaving because by default it even logs replies to calls like setenforce 0.
• Be careful with symlinks in /tftpboot if you use them. On your system they should actually be broken, because the tftp server will serve from /tftpboot and treat that as its root, as if it were chroot'd. So, if you have a file /tftpboot/phone/phone.inf, and you want a symlink to that file to exist and work in /tftpboot, you actually need to create a broken symlink like this: ln -sf /phone/phone.inf /tftpbootso that the symlink will work for tftpd.in This is one of those steps that I completely forget every time too.

Well, that should be it for the next time I have tftp troubles!

After some situations at work this week where I lost time where I really shouldn't have had to, combined with the observation that I get more useful strategic work done...

At home, close to the toilet because I've needed it about 20 times over two hours yesterday evening, and a bunch more this morning. Sigh.

With such a low energy level I decided to go through some of my pending tasks, picking out low-hanging fruit. That's one of the situations in which a GTD system pays off - those wasted days of low-energy where you basically want to execute simple mechanic tasks on some list without needing to do the thinking of which those tasks are in the first place.

One of those tasks is to replace my venerable booq backpack that I got right after moving to Barcelona more than 5 years ago. It has served me well, with a few minor problems along the way (the felt on the carrying handle tearing for example, now replaced with awesome-looking gaffa tape), and now recently the zipper started tearing, which could be fixable but would just take way too much time.

I'm sure there might be just as good or better bags from other brands, but in our 21st century ecosystem we vote with our cash, so I'm going to buy another booq backpack. (If you are looking for a new bag or backpack, I recommend you give them a look. I have no idea how many people ended up buying one over the years on my recommendations, I hope there are many.)

This time I went with the BOA 3M without the Vyper sleeve. The Vyper sleeve looks awesome and cool, but basically my laptop sleeve from the previous Booq backpack still works fine, and has a top pocket (in which I store 2 USB drives, a network card, some pens, earplugs, some spare batteries) and a back pocket in which I can slide documentation, so that sleeve on its own actually works really well as a light bag just for the laptop.

The sleeve might be a problem by the time I get my next laptop though, because I doubt there will be any more 14 inch 4:3 laptops around by that time. For now it will serve me just fine.

These backpacks don't come cheap (I just paid 185 euros including shipping from Germany) but given the durability of my previous backpack, which I've basically used daily for 5.5 years, I have no doubt the pack will be worth the money.

My only real gripe with my previous backpack was the fact that I would get a sweaty back at the end of my 30 minute walk/metro ride to work. The BOA should be slightly better for that, though I guess all backpacks (except the ones with metal wire frames that lift the pack off your back) make you suffer from a sweaty back.

Looking forward to walking around with my new bag soon ! One task knocked off the list.

At home, close to the toilet because I've needed it about 20 times over two hours yesterday evening, and a bunch more this morning. Sigh. With such a low energy...