Present Perfect

It's two weeks shy of a year since the last morituri release. It's been a pretty crazy year for me, getting married and moving to New York, and I haven't had much time throughout the year to do any morituri hacking at all. I miss it, and it was time to do something about it, especially since there's been quite a bit of activity on github since I migrated the repository to it.

I wanted to get this release out to combine all of the bug fixes since the last release before I tackle one of the number one asked for issues - not ripping the hidden track one audio if it's digital silence. There are patches floating around that hopefully will be good enough so I can quickly do another release with that feature, and there are a lot of minor issues that should be easy to fix still floating around.

But the best way to get back into the spirit of hacking and to remove that feeling of it's-been-so-long-since-a-release-so-now-it's-even-harder-to-do-one is to just Get It Done.

I look forward to my next hacking stretch!

Happy ripping everybody.

It's two weeks shy of a year since the last morituri release. It's been a pretty crazy year for me, getting married and moving to New York, and I haven't...

Yesterday, I was wondering if there shouldn't be a new Fedora out by now and if it would fix a bunch of my current GNOME 3 annoyances.

So I checked, and lo and behold, the final release date was yesterday! Excellent.

Let's do some completely unscientific scoring this time around. In part one, it's bound to get ugly because you always run into the negatives first when doing an upgrade.

First challenge was finding the torrent links for the full DVD. Apparently the DVD is now a well-hidden option, and the torrent even more so - I had to google for it, I couldn't find any links on the download site. -1 and -1. I appreciate that there is a small CD with a live installer and everything, but I have to upgrade 3 computers in total so I prefer to download once as much as I can - although it's likely all of them will need to upgrade gazillions of packages soon after.

The second issue: after booting, by default it gives you the second option - test media and install. I didn't realize that, and just hit Enter. Then anaconda starts counting something without telling you what it's doing, at which point I figured it would be a media check as it was really slow. But if you hit Esc to abort it, you get dropped into a rescue shell, instead of just continuing. Err, OK. I don't know if anyone out there really uses or prefers the media check option, but I never do.

Reboot, make sure to go up to select the first option (which, really, should be switched to the second if you're not going to default to it ?)

-1 for being confusing and defaulting to wasting my time.

This is the first anaconda that is actually full-screen on my laptop, nice. +1

The first impression of anaconda is that it looks good and it looks very GNOME 3-y. Not entirely sure I like the 'things popping up on my screen as anaconda presumably checks stuff' without telling me, because there's potential for getting it wrong (-1), but I'll accept it for now. Definitely liking that it figured out my network connection automatically, if NetworkManager is behind it then I'll be darned - yay for NetworkManager! (+10 alone for that one, so NetworkManager pulls out slightly from minus infinity.)

Date & Time, since you now can rely on network possibly working maybe you should look me up by IP as a sane default instead of New York ? Network Time was on by default, but no. No points for or against though.

On to partitioning. I've always used a custom layout. The new dialog scares me a little; I checked "I don't need help" but the only option forward is 'reclaim space', so I'm not sure it's not going to do anything bad to my drives. -5

On the next screen, I see a tree with New Fedora 18 Installation, Fedora Linux 15, Fedora Linux 16, and Unknown. I typically have two or three root partitions so I can test different Fedora versions and fall back to older ones when I'm upgrading. It's a little confusing to use the tree, but basically I figured out how to go through the Fedora Linux 15 config and get it to move the ones I wanted to use to the New Fedora 18 Installation. I definitely see the potential for this being easier to use than the old way but it needs a bit more documentation or tooltips or explanation to make it really feel safe to use. Maybe it would help too to have a final overview page when finishing partition so you can confirm that it looks like it's going to do the right thing. -1 for the confusion, but +5 for finding partition info from all my roots.

It's a small touch, but it's nice it's asking for your root password *while* already installing packages. Makes it a little faster to get it done. +3 guys! I'm wondering if it couldn't do more of that - your time zone config for example ?

The redesigned anaconda really looks nice, and fits in well with the GNOME 3 experience I'm now used to. Gutsy move, but this is going to pay off in the long run. +10

After waiting for the packages to install, I clicked Reboot, and it dropped me in a text mode that said [terminated]. oh well, nothing's perfect I guess. -2

On a hard reboot, I got greeted with a reasonably nice GUI that had managed to pick up my old boot options - even the Windows partition I have on this machine. +5 The text looked ugly and stretched (-1), but it could have been worse.

Firstboot starts and greets me first with a big white square around my cursor (-1), and then the nice-looking GUI. Firstboot asks me for date and time info again, not sure why. Maybe an oversight. -1

And then we're on to the login screen. And it definitely looks nice! +5

Logging in. Being told there are updates. Holy crap - 218 updates - for a release that's a day old. Does the word 'release' mean anything anymore? -5. Seriously, freeze that crap for a few days, only real security issues or facepalm bugs.

My updated failed to process - because I had installed the rpmfusion rpm and it doesn't have the GPG key. Yet again, by default updating packages fails completely when anything in the config is not working, instead of at least getting me the updates that can be installed, in the name of, you know, security. -10 because this is a persistent attitude problem for yum.

Create some missing symlinks, and the upgrade can continue. So I leave for lunch.

And when I come back, I am greeted by some kind of lock screen. It looks pretty. (+3) It's like a video game, those arrows. Yes, that's it - it reminds me of when I pretend to be Batman in Arkham City and I'm on a mission and it's telling me to glide down in the direction of the three floating arrows. Except, it's not actually acting like a lock screen - when I click it, something happens and I go to a user selection ? It looks like I got logged out behind my back ? Really ? Is it doing some kind of automatic logout after upgrading ? I hope not, that would be horrible as a default. No clue what happened. -5

I log in again, and recover my vim sessions that got so brutally killed.

I start running the install commands that are part of my general upgrade checklist. In the meantime, I check out this rumour I heard that Fedora 18 installs with kernel 3.6 by default but the one day old upgrades install 3.7, so I run rpm -qa | grep kernel.

Oh my. It's spewing db errors halfway through the query. Three times in a row. Contrary to popular belief, rpm is really robust, and you really need to do evil things to get it to corrupt, like drop your hard drive or kill -9 during package installs. But here it just fails simply querying, presumably for the first time in my experience it can't handle querying while installing ? -5

After letting it sit there and install some more, I get that lock screen again. I click it, and some arrows flash. Maybe I'm supposed to drag it up or something ? But before I can do, the screen flashes, and I'm back to the login prompt. Oh, so even worse - this new lock screen crashes my whole desktop somehow ? Ouch. -5

Evolution forgot my sort settings (per folder) and 3-pane window. -3 for making me suffer through having to sort every single folder by date, descending again (really, is unsorted a sane default to anyone ?)

The lock screen looked cool at first glance, but after what feels like lifting up the door to my garage four times today already it's getting on my nerves. -3 Same with the 'pressure-triggered' notification area, which is starting to cause pain in my hand on my laptop, and I never have that kind of trouble. I wonder if these things got designed with a console joypad as an interface, where you could accept that pressure-triggered actions make sense. -3 for sucking and another -3 for making me think originally that it looked cool until I actually had to use it.

My first login as a 'fresh' user (I don't mount my real home until I'm sure all the basics work ok) is very zippy and GNOME 3 looks tidier. +5

However, my second login, with my old user, takes a good 30 seconds before anything at all appears beside the desktop. I don't know which dead weight I'm dragging along from before, but this upgrade is not liking it one bit. No feedback whatsoever on what's going wrong though. -3

Total score so far: -13.

It didn't pull back to breakeven, but don't despair - now that the basics are done, it's bound to get better in the next part.

(editor's note - see if you can tear this whole article to pieces by pointing out a counting error in the score, cleverly invalidating my already unimportant opinion!)

Yesterday, I was wondering if there shouldn't be a new Fedora out by now and if it would fix a bunch of my current GNOME 3 annoyances. So I checked,...

Well, it's been a record 399 days, but they have come to an end. Last weekend, a drive in my home desktop started failing. I had noticed some spurious SATA errors in dmesg before, and load times were rising (although lately in the 3.4/5/6 kernels I've been running I've actually seen that happen more and more, so it wasn't a clear clue).

Then things really started slowing down, and a little later I noticed the telltale clicking sound a drive can make when it's about to give up.

Luckily life has taught me many valuable lessons when it comes to dealing with hard drives. The failing drive was a 1TB drive in a RAID-1 software raid setup, so fixing it would be simple - buy a new 1TB drive and put it in the RAID, and just wait for hours on end (or, go to sleep) as the RAID rebuilds.

A few years ago I started keeping track of my drives in a spreadsheet, labeling each drive with a simple four digit code - the first two digits the year I bought the drive in, and the second two digits just a sequence (and before you ask, the highest those two digits got so far is 07 - both in '11 and '12). The particular drive failing was 0906, so the drive was about 3 years old - reasonable when it comes to failure (given that it has been running pretty much 24/7), but possibly still under warranty, and I've never had the opportunity to try and get a disk back under warranty, although this particular one was bought in Belgium.

But I digress.

Of course, I seldom take the simple route. When buying hard drives, I basically only follow one rule - buy the biggest drive with the cheapest unit price. And at last, Barcelona stores have gotten to the 3TB drives being at that sweet spot. So, why buy a comparatively expensive 1 TB drive and not get to have any fun with complicated drive migration?

So I settled on a 3TB Seagate Red drive (this is a new range specifically for NAS systems, although I'm not convinced they're worth the 6% extra cost, but let's give it a try) so I could replace the penultimate 2TB drive in my ReadyNAS, get 1TB of extra capacity on that, and then just use the newly freed 2TB drive in my desktop computer.

Of course, that's when I ended up with two problems.

Problem 1 was the NAS. The ReadyNAS was at 10TB already, having 4 3TB drives and 2 2TB drives with dual redundancy. I took out a tray, replaced the drive, put it back in, and then waited a good 18 hours for the array to rebuild. (The ReadyNAS has something they call XRaid2 which really is just a fancy way of creating software raids and grouping them with LVM, but in practice it usually works really well - figuring out a number of raid devices it should create using the mix of physical drives).

This time, it had correctly done the raid shuffle, but then gave me an error message saying it couldn't actually grow the ext4 filesystem on it because it ran out of free inodes. Ouch. A lot of googling told me that I should try to do an offline resize, so I stopped all services using the file system, killed all apple servers that somehow don't shut down, and did the offline resize. And then I rebooted.

The ReadyNAS seemed to be happy with that at first, saying it now had more space (although depending on the tool you use, it still says 10TB, because of the 2 to the 10/10 to the 3rd number differences adding up). But soon after that it gave me ext3 errrors. Uh oh.

With sweaty palms, I stopped all services again, unmounted the file system, and fsck'd it. And almost immediately it gave me a bunch of warnings about wrong superblocks, wrong inodes, all in the first 2048 sectors. Sure I have backups, but I wasn't looking forward to figuring out how up-to-date they were and restoring up to 10 TB from them.

I gasped for air and soldiered on, answering yes to all questions, until it churned away, and I went to sleep. The next morning, a few more yeses, and the file system seemed to have been checked. Another reboot, and everything seemed to be there... Phew, bullet 1 dodged.

On to problem number 2 - the desktop. The first bit was easy enough - although I've never been able to use gdisk to copy over partition tables like I used to with fdisk - it seems to say it did it, but it never actually updates the partition table. Anyway, I created it by hand copying the exact numbers, then added the partitions to the software raid one by one, and again waited a good 6 hours.

And looking at my drive spreadsheet, I noticed I had a spare 2TB drive lying around that I was keeping in case one of the NAS drives would fail - but given that most of them are 3TB right now, that wouldn't be very useful. So, after the software raid rebuilt in my desktop, I switched out the working 1 TB drive as well, and repeated the whole dance.

So now I had 2 2TB drives, 1 TB of which was correctly used. At this point I would normally figure out how to grow the partitions, then the md device, then the LVM on it, and then finally grow my ext4 /home partition. But since it's using LVM and I never played with it much, this time I wanted to experiment.

I still had the working 1TB drive which I could use as a backup in case everything would fail, so I was safe as houses.

At first I was hoping to do this with gparted live, but it seems gparted doesn't understand either software raid or lvm natively, so it's back to the command line.
Create two linux raid partitions on the two 2TB drives, assemble a new md device, and spend a lot of time reading the LVM howto.

In the end it was pretty simple; step 1 was to use vgextend to add the new md to the volume group, and then lvextend -l 100%FREE -r to grow the logical volume and resize the file system all at once. That automatically fsck's (which you can follow progress of by sending USR1) and then resize2fs (which you can't really check progress of once it's started)

(By now, we're over a week into the whole disk dance, in case you were wondering - doing anything with TB-sized disks takes a good night for each operation).

Except that now rebooting for some reason didn't work - grub complained that it didn't know the filesystems it needed - /boot is on a software raid too, and even though I don't recall running anything grub-related in this whole process, I had swapped out a few disks and may have botched something up when transferring boot records.

At the same time, I was also experimenting with Matthias's excellent new GLIM boot usb project (where you finally just drop in .iso files if you want to have multiple bootable systems on your usb key, without too much fidgeting), so I tried doing this in system rescue cd.

Boot into that, manually mount the right partitions, chroot into that, and then grub2-install /dev/sda.

Except that grub complained saying

Path `/boot/grub2' is not readable by GRUB on boot. Installation is impossible. Aborting.


Most likely this was due to it being on a software raid. Lots of people seemed to run into that, but no clear solutions, so I went the dirty way. I stopped the raid device, mounted one half of it as a normal ext file system (tried read-only first, but grub2-install actually needs to write to it), ran grub2-install, unmounted again. Then I recreated the software raid device for /boot again by reassembling, and that somehow seemed to work.

Reboot again, and this time past GRUB, but dropped in a rescue shell. My mdadm.conf didn't list the new raid device, so the whole volume group failed to start. Use blkid to identify the UUID, add that to /etc/mdadm.conf (changing the way it's formatted, those pesky dashes and colons in different places), verify that it can start it, and reboot.

And finally, the reboot seems to work. Except, it needs to do an SELinux relabel for some reason! And in the time it took me to write this way-too-long blogpost, it only managed to get up to 54%.

And I was hoping to write some code tonight...

Oh well, it looks like I will have 1TB free again on my NAS, and 1TB of free space on my home desktop.

There is never enough space for all of the internet to go on your drives...

UPDATE: SELinux relabeling is now at 124%. I have no idea what to expect.

Well, it's been a record 399 days, but they have come to an end. Last weekend, a drive in my home desktop started failing. I had noticed some spurious SATA...

For a website for work, I wanted to make sure the web guy writes valid HTML code.

I found a validation middleware for Django, which uses tidy.

But the website is in HTML5, so the normal tidy doesn't validate it properly.

Luckily, there's tidy-html5, a fork of tidy for html5, and it's a drop-in replacement for tidy - to the point where it even works with python-tidy.

So I packaged it up for Fedora 16/17 and put it in my package repository.

The package conflicts with the tidy packages; I'm not sure if I should set it to Obsoletes: the tidy package instead, and I don't know if validates non-HTML5 the same way tidy does.

If anyone uses tidy regularly, give me some feedback. If anyone wants to take this into Fedora proper, let me know.

For a website for work, I wanted to make sure the web guy writes valid HTML code. I found a validation middleware for Django, which uses tidy. But the website...

In the middle of my
CouchDB
Security
Series
I made an unlucky fall during basketball, falling on my back, and feeling my head continue its downward trajectory until it was halted painfully by the cement floor. As I tried to get up the world turned, and as I tried to walk to the bathroom five minutes later I involuntarly kept veering off to the left.

It took me a few weeks to recover from that, and I managed to go back to playing basketball after a month. But my amount of off-work hacking was zero.

That's now been over two months, so I've finally reserved part of the weekend for some hacking again. And now I'm busy tying up loose ends, of which this is one - a new mach release for Fedora 17. Nothing very special, just warming up the muscles again.

I'm comically amused by what is still my first ever python program, but I have no desire to redo it, clean it up, or continue on the half-done mach3 version (which uses novelty programming techniques like, you know, 'more than one file').

In the middle of my CouchDB Security Series I made an unlucky fall during basketball, falling on my back, and feeling my head continue its downward trajectory until it was...