Despite certain frustrations with our Apple Xserve G5s, I’ve found that they make decent Linux servers, especially considering their age. Recently I wanted to repurpose one as a development server, and in that role I really wanted disk redundancy for added protection of the data. Unfortunately there’s no RAID controller in these servers, so software RAID is the only option. Originally RAID wasn’t crucial and I gave up on software RAID as it seemed too complicated to justify the time spent figuring out how to do it. This time, however, it was more important and I was determined to get software RAID functioning.
Never having used software RAID before, this was quite a steep learning curve for me, especially given that the Xserve G5 is a ‘New World’ class of Mac, which means that it uses the Apple Partition Map partitioning scheme, which is fairly rare in a Linux world that is largely involved with master boot records. I use Yellow Dog Linux on our Xserves because it’s based around Red Hat, and we use Red Hat-based distributions as standard. Yellow Dog Linux 6.2 actually makes it very easy to set up RAID as part of the installation process, although there are one or two aspects I prefer to do a different way as a result of experimenting with this process. Also, manipulating existing arrays and making new disks bootable took me some time to figure out.
So, let’s discuss the method which ended up working for me. The Xserve has three disks, and I wanted two of the disks to contain RAID 1 partitions with the third disk as a spare. I used RAID 1 because I wanted a RAID level which would continue to work well even if only one disk was left functioning on the server. RAID 5 requires a minimum of two disks with Linux software RAID, and it’s slower than RAID 1, so it’s generally less desirable for me. Also, you can’t use a RAID 5 array for your /boot partition, so since you’ve got to use RAID 1 for /boot then I figure you might as well use it for everything else.
Before starting the Yellow Dog Linux installation, I firstly used mac-fdisk in Finnix to partition the disks myself, because this gives more control over the process, and I found that odd and unhelpful things happened when trying to do the initial partitioning in Yellow Dog Linux. Finnix, incidentally, is a superb live sysadmin Linux distribution which still has PowerPC support in version 93.0 and is thus ideal for working on Xserves. mac-fdisk must be used because it’s the only Linux partitioning tool I’m aware of which understands the weird partition types needed for things to function on a New World Mac.
So, boot into Finnix, for which you’ll need to choose ‘finnix64’ on these G5 machines. Next we’ll do the partitioning. In mac-fdisk, for each of your disks, firstly press I to initialise the partition scheme. This will remove all existing partitions then create the partition map, which is a special partition that must always be the first partition on each disk. Then press B to create the special Apple bootstrap partition, which should ideally be the second partition on the disk (specify 64 as the first block for this partition so that it follows on from the first partition). Then press C to add each Linux partition that you will use as part of your RAID partitions. I normally make the first one of these the /boot partition with a size of 100MB, then the second one the / partition to use up most of the disk, then finally a partition to use as swap. Don’t forget to press W to write the new partition map. After that, make a note of the exact scheme and partition sizes you have used on the first disk, then do the same on the other two disks.
Next, boot off the Yellow Dog Linux install CD. When you get to the partitioning screen, choose ‘Create custom layout’ and continue. You should now see a representation of the partitioning scheme you just created within Finnix. It will probably say ‘ext3’ or ‘Foreign’ for each of the Linux partitions you created. Edit each of these on /dev/sda and /dev/sdb these so they are all formatted as ‘software RAID’. Just leave /dev/sdc alone for now – we’re not going to do anything with this third disk as it’s just there as a spare. Actually, Yellow Dog Linux does allow you to set this up as a hot spare for your RAID array partitions, but I found in practice that this seemed to cause more problems than it solved, so I avoid that now.
Then click the ‘RAID’ button and choose to ‘Create a RAID device’. Assuming you’ve followed a similar partitioning scheme to me, choose ‘/boot’ as the mount point, set the RAID level to ‘RAID1’, and tick the boxes next to ‘sda3’ and ‘sdb3’. This will create the RAID device /dev/md0 for the /boot partition. Then do the same for the other RAID partitions you need, including the swap partition (if you have one). Then proceed with the rest of the install, configuring the other options as you want them.
Once Yellow Dog Linux is installed, boot your Xserve and log in. When you do watch cat /proc/mdstat, you should see information about your RAID arrays showing a healthily active state and containing the disk partitions you specified in the installer. Some of your RAID partitions may still be resyncing; don’t worry about this – it’s just a normal part of the process. So now you can start using your Xserve, and you won’t need to worry about the RAID setup again until one of the disks fails.
In Yellow Dog Linux, when a disk fails, you will get notified via email because mdadm is automatically started up at boot time as a daemon in monitor mode. For these notifications to work you just need to make sure you have your root email from this server going somewhere useful. When you get a notification like this, you’ll see that the partitions on the failed disk will be marked as failed, e.g. if it’s the second disk which has failed then mdadm --detail /dev/md0 will show that /dev/sda3 is active but that /dev/sdb3 has failed. You should now remove all the failed partitions with mdadm /dev/md0 -r failed and equivalent for each of your RAID partitions. Then shut the machine down, take out the failed disk, and replace it with the third disk which we allocated as a spare. It’s best to put the spare disk into the slot where the failed disk was, so that we don’t encounter any confusion with drive ordering. Then start the server up again.
As we already partitioned this new disk earlier on, we should be easily able to add the partitions to our RAID arrays with mdadm /dev/md0 --add /dev/sdb3 and equivalent for each partition. We can then watch the new partitions being synchronised into the arrays with watch cat /proc/mdstat. Once that’s done, we’re back up and running with full redundancy and only a tiny amount of downtime to swap the disks around.
We still need to make the new disk bootable, however, so that we can boot the server if the original disk fails. This is a case of setting up the yaboot boot loader properly on the Apple bootstrap partition on the new disk. As you may remember, the bootstrap partition is the second partition on the disk which we created as part of the partitioning process. For the sake of these examples we’ll assume the new disk is the second disk (/dev/sdb), but if it’s the first disk then just change /dev/sdb to /dev/sda in each example.
So, firstly we need to make sure that /etc/yaboot.conf is set up correctly. The ‘boot’ parameter must be set for the correct partition on the correct drive, so in this case this line should be boot=/dev/sdb2. There shouldn’t be any need to edit anything else in this file. We then format the bootstrap partition with mkofboot -b /dev/sdb2. And then we install the yaboot bootloader with ybin -b /dev/sdb2. You should then be able to boot off the new disk, although it’s worth testing that just to be on the safe side.
We can now add a new third disk to the Xserve and partition it in the same way as we did earlier, so this disk will act as a spare next time a disk failure occurs. And there we have it – full disk redundancy on an Apple Xserve G5 using Linux software RAID.