![[lang]](/images/eye.png)
Help me ext2 Software RAID experts |
2008-04-17
|
Backdrop for the story - after a weekend or two fiddling with all my power supplies to try and revive my home server, I finally got it back online. Well, sort of. Disk 1 of the software RAID for my vault drives (that have backups of all my stuff) was making strange ticking noises at bootup and the BIOS did not find it.
Well, that' s why I have RAID in the first place, right ? Drives fail, they seem to do so more often now that I have too many of them, and the solution is simple. I get a new drive, replace the broken one, and do a hot add, and there you go.
I've been so good the last year about backups. This drive has dirvish backups of my laptop and my work machine, and it was doing great! I was sure I was never going to lose data again. Now I wasn't so sure anymore.
So I turn off the server, disconnect the vault drives (to make sure that the other one has no chance to start failing), and fly back to Barcelona, where I go drive shopping during the week. No more 400 GB drives to be found, so I get a 500 GB one. (And while I'm at it, I buy the two 1 TB drives I've been trying not to buy for a long time).
Come back home to Brussels, put the drive in, boot, and try to mount the slave from the vault Software RAID. Because that' s the beauty of RAID-1 Software in Linux, isn't it ? Either drive can be mounted on its own just as if it was a normal drive.
Well, except that there was no partition table, and I couldn't mount the drive. Now I was really worried :/
So, data recovery. Let's start with a dd from the old slave to the new 500 GB master drive, so we can try funky partitioning stuff on there. Three hours in, I accidentally made a motherboard box drop from the shelf, and of course it lands right on the slave vault drive, knocking out the power. Oops.
Start over. Meanwhile, learn about gpart, a tool that scans your drive for partition information. Let that run for three hours, finds absolutely nothing. Nothing. (The smart ones among you have already figured out how stupid I am by now). I'm starting to get really worried.
With half the dd done, I run fdisk on the master (which has the copy) and partition the drive, then try to mount the partition. Still no go. It says I should try and find an alternate superblock. I try some numbers, but nothing works. I start to get really worried.
Google some more, and find a tool called scandrive. I compile it, run it, and limit to the first few sectors:
[root@onzenbak ~]# ./scandrive -v /dev/hdd -C 100
scandrive v1.00 (2002/02/01) - steve@unixwiz.net
I/O buffer: 256 sectors of 512 bytes
Device /dev/hdd is open (capacity = 100 sectors)
Loop 0: scanning sector 0 ( 0.00%)...
Found ext2 magic at sector 2 (size 97677824, #0)
Finished scanning.
What the ? There's an ext2 magic string right at the beginning.
And then it hit me. Doh, was I really that stupid ? Wait, shouldn't I have config for the software RAID in the first place ?
[root@onzenbak ~]# cat /etc/mdadm.conf
# these devices can be used as part of RAID arrays
DEVICE /dev/hdc /dev/hdd# md0 is our mirrored RAID partition made from our two drives
ARRAY /dev/md0 devices=/dev/hdc,/dev/hdd
Yeah, now I remember that time in my life where I thought partitioning was a waste of space, and I might just as well use the drive directly.
So now I know why I should not do that - I have a 400 GB drive and a 500 GB drive in my machine, and I should create a second partition on the bigger drive so I have some 100 GB of extra useful disk space.
So, here's the question to you smart people: how can I fix up my /dev/hdd (the 400 GB drive) to have its ext3 file system inside a partition ? And can I then just simply change the devices line to use hdc1 and hdd1 ? How do I get this right without losing data ?
At a guess: Create a backup of /dev/hdd. Resize it to be slightly smaller than 400Gb (resize2fs). Back that up to a second drive. Create your partition on /dev/hdd, copy the data from the second drive to /dev/hdd1. Resize2fs again because your partition will be slightly larger than the filesystem in it. Now recover your RAID.
Those two 1Tb drive would come in handy now.
Comment by Kristof Provost — 2008-04-17 @ 21:56
Can’t do it. The old 400G drive is using all of its space for the md0 array. By creating a partition table of any kind on it, you’re going to end up with usable space smaller then the original drive. So even if you add the 500G drive to the mix and disable the old 400G drive. When you create the partition on it you’ll be unable to add it properly because you won’t have enough space to mirror to it.
I don’t think there’s any way to grow / shrink raid arrays at the RAID level as far as partiton size goes (you can always add / remove drives though) So here’s what I think your best bet is… MAKE A BACKUP.
Boot via rescue disk.
Shrink the filesystem using resize2fs on your 400G drive. Just a few M would be fine.
create a /dev/hdc1 thats less then 400G on your 500G drive.
# hdd = 400G hdc = 500G hdc1 = <400G
dd if=/dev/hdd of=/dev/hdc1
This will fail but, in theory, the parts that fail you don’t need because you resized below it.
Then run a chkfs on /dev/hdc1.
Next fdisk /dev/hdd and create a new partition of the same size that you created for the hdc1 partition.
Then re-assemble the array with just the hdc1 partition (hdc == the new drive)
mount /dev/md0 and make sure everything’s there.
Then add /dev/hdd1
monitor /proc/mdstat make sure it comes back up.
alter your /etc/mdstat.conf
reboot.
Alternatively you can use the old 400G /dev/hdd and just create a 400G /dev/hdc1 and add it.
I’m not totally sure if this will actually work or not, it’d be an interesting lab though, I might try it tonight when I go home.
Comment by Mike McGrath — 2008-04-17 @ 22:28
I’d say :
Partition the 500G with two partitions : one of 400GB (a little more than the size of the full 400G disk), and one of the remaining capacity.
Now you have, /dev/hdd (the 400G) and /dev/hdc1 + /dev/hdc2 (the partitionned 500G).
Now let’s copy the data with “dd if=/dev/hdd of=/dev/hdc1”
And that’s it!
Now if you want to partition hdd into hdd1, I think that you’ll have to use resize2fs, do the above procedure, and then partition /dev/hdd, and use dd the other way (“dd if=/dev/hdc1 of=/dev/hdd1”), but I don’t really know if this is really useful :p.
Comment by tuXXX — 2008-04-17 @ 23:16
Maybe you can partition only the 500 GB disk, set up a partition exactly as large as the entire 400 GB /dev/hdd disk and then use /dev/hdc1 and /dev/hdd for your RAID?
Comment by Kevin Kofler — 2008-04-18 @ 00:16
You should be able to do this, the DOS style partition table only lives in a single sector (inside the MBR) and even if the partition table is empty, that sector probably still lives outside of the filesystem. Notice it found the ext2 magic in sector 2…what is in sector 0 and sector 1? I suggest:
dd if=/dev/hdd bs=512 count=2 | hexdump -c
Read up on the MBR (http://en.wikipedia.org/wiki/Master_boot_record) and see if what you have there is an empty one, that can be safely changed with gparted, if so, creating a primary partition across the span of the filesystem and you’re rolling.
Comment by Andrew — 2008-04-18 @ 01:05
You may also want to look at testdisk, which is a much more mature product than gpart. Although I hope you never need it :-)
You discovered exactly why I always tell companies to partition their disks, even if it is just to use the disk as a physical volume or a RAW device. At least the partition type can give information to what the disk is being used for. Without it your future colleagues are at a loss.
Comment by Dag Wieers — 2008-04-18 @ 01:08
You can’t. the Linux Software RAID subsystem doesn’t support shrinking devices, which would be required for your setup. However, the subsystem *does* support having two mirrors of a RAID-1 array being completely strange and different things. You can have a partition on one disk, and the full disk on the other. You can run NBD for one mirror, and a local disk for the other. I guess if you’re creative with losetup, you probably could use a local file as a mirror, too (would have to check that).
Given all of that, the RAID system has to support two mirrors that are not equally sized. Just create a partition on the 500G disk that is slightly larger than the 400G disk; Linux will use the space it needs, and ignore the leftover.
Comment by Wouter Verhelst — 2008-04-18 @ 03:13
Just my R0.02 worth of noise …
Not sure how much space one saves by not partitioning a drive, but I use it as a minor security feature … If somebody loans/steals a drive that is part of one of my raids, they will not have an idea that it’s part of my raid by the partition table … But that is just me … (Greetings Dag!)
You should be able mount the drive without the partition table, something like mount /dev/hdd /mnt/tmp -o ro … Or even do an e2fsck -n /dev/hdd for read-only to see if the filesystem is intact …
You could also check if the raid member is also recognized with something like mdamd -E /dev/hdd … If so,. I would then start an md with mdadm -A /dev/md1 –force and then add then new member to the array, using something like mdamd /dev/md1 -a /dev/hdc1 …
These are just guide lines and your mileage might vary …
Best of luck!
Mailed
LeeT
Comment by C.Lee Taylor — 2008-04-18 @ 08:35
Create a new Raid-1 with one disk on the new 500G disk to the amount of the old disk, partition it, mount it, copy files over, remove 400G from old Raid, add it to new Raid, let it sync and be done.
Comment by Peter — 2008-04-18 @ 13:42
Make a new 400G partition of type fd (linux raid auto) on the 500G drive, make sure it will be the same size or smaller at the largest possible partition on the 400G drive. Something like 1000 blocks shorter that the total size of the unpartitioned block device.
Then make a new raid1 mirror device using the new 400G partition on the new 500G drive.
Tell mdadm that the other half of the mirror is missing. Make a filesystem on it and copy the data from the original raid1 md device to the new one.
Once you are certain the data is copied, stop the original md device and make a new partition on the 400G disk covering the entire disk. Add a partition on the 400G drive covering the entire drive. Then just add that partition to the new md device, and let is sync.
Comment by Thomas Stewart — 2008-04-18 @ 13:51
[…] you ask ? Well, Remember my previous problem. I never followed up on that post, because it was just too embarassing. Here’s what happened […]
Pingback by thomas.apestaart.org » Nerd Night part 2 — 2008-05-23 @ 17:20