[gentoo-amd64] Re: RAID1 boot - no bootable media found

[Duncan]

Mark Knecht posted on Sun, 28 Mar 2010 10:14:03 -0700 as excerpted:

> I brought up new hardware yesterday for my first RAID install. I
> followed this Gentoo page describing a software RAID1/LVM install:
>
> http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml
>
>   Note that I followed this page verbatim, even if it wasn't what I
> wanted, with exceptions:
>
> a) My RAID1 is 3 drives instead of 2
> b) I'm AMD64 Gentoo based.
> c) I used grub-static

Had you gotten anything off the other list, I see no other replies here.
Do you have that install or are you trying over as you mentioned you might?

That post was a bit long to quote in full, and somewhat disordered to try 
to reply per element, so I just quoted the above and will cover a few 
things as I go.

1) I'm running kernel/md RAID here, too (and was formerly running LVM2, 
which is what I expect you mean by LVM, and I'll continue simply calling 
it LVM), so I know some about it.

2) The Gentoo instructions don't say to, but just in case... you didn't 
put /boot and / on LVM, only on the RAID-1, correct?  LVM is only for non-
root non-boot.  (Actually, you can put / on LVM, if and only if you run an 
initrd/initramfs, but it significantly complicates things.  Keeping / off 
of LVM simplifies things considerably, so I'd recommend it.)  This is 
because while the kernel can auto-detect and configure RAID, or the RAID 
config can be fed to it on the command line.  The kernel cannot by itself 
figure out how to configure LVM -- only the LVM userspace knows how to 
read and configure LVM, so an LVM userspace and config must be available 
before it can be loaded.  This can be accomplished by using an initrd/
initramfs with LVM loaded on it, but things are MUCH less complex if / 
isn't LVM, so LVM can be loaded from the normal /.

3) You mention not quite understanding how /boot works on md/RAID -- how 
does grub know where to look?  Well, it only works on md/kernel RAID-1, 
and that only because RAID-1 is basically the same as a non-RAID setup, 
only instead of one disk, there's several, each a mirror duplicate of the 
others (but for a bit of RAID metadata).  Thus, grub basically treats each 
disk as if it wasn't in RAID, and it works, because it's organized almost 
the same as if it wasn't in RAID.  That's why you have to install grub 
separately to each disk, because it's treating them as separate disks, not 
RAID mirrors.  But it doesn't work with other RAID levels because they mix 
up data stripes, and grub doesn't know anything about that.

4) Due to personal experience recovering from a bad disk (pre-RAID, that's 
why I switched to RAID), I'd actually recommend putting everything portage 
touches or installs to on / as well.  That way, everything is kept in sync 
and you don't get into a situation where / including /bin /sbin and /etc 
are a snapshot from one point in time, while portage's database in /var/db 
is a different one, and stuff installed to /usr may be an entirely 
different one.  Not to mention /opt if you have anything installed 
there...  If all that's on /, then it should all remain in sync.  Plus 
then you don't have to worry about something boot-critical being installed 
to /usr, which isn't mounted until about midway thru the boot cycle.

4 cont) What then goes on other partitions is subdirs of the above, 
/usr/local, very likely, as you'll probably want to keep it if you 
reinstall, /home, for the same reason, /var/log, so a runaway log can't 
eat up all the space on /, it's limited to eating up everything on the log 
partition, likely /tmp, which I have as tmpfs here but which otherwise you 
may well want to be RAID-0 for speed, /var/tmp, which here is a symlink to 
my /tmp so it's on tmpfs too, very possibly /usr/src and the linux kernel 
tree it contains, as RAID-0 is fine for that as it can simply be 
redownloaded off the net if need be, same with your portage dir,
/usr/portage by default tho you can point that elsewhere (maybe to the 
same partition holding /usr/src, but if you use FEATURES=buildpkg, you 
probably want your packagedir on something with some redundancy, so not on 
the same RAID-0) if you want, etc...  If you have a system-wide mail setup 
with multiple users, you may want a separate mail partition as well (if 
not, part of /home is fine).  Desktop users may well find a separate, 
likely BIG, partition for their media storage is useful, etc...  FWIW, 
the / partition on my ~amd64 workstation with kde4 is 5 gigs (according to 
df).  On my slightly more space constrained 32-bit netbook, it's 4 gigs.  
Used space on both is ~2.4 gigs, with the various other partitions as 
mentioned separate, but with everything portage touches on /.  (That 
compares to what appears to be a 1-gig / md3 root in the guide, with /var 
and /usr on their own partitions/volumes, but they have an 8 gig /usr, a 4 
gig /var, and a 4 gig /opt, totaling 17 gigs, that's mostly on that 4-5 
gig /, here.)

5) The hexidecimal digits you mentioned during the BIOS post process 
indicate, as you guessed, BIOS POST and config process progress.  I wasn't 
aware that they're documented, but as your board is an Intel and the link 
you mentioned appears to be Intel documentation for them, it seems in your 
case they are, which is nice. =:^)

6) Your BIOS has slightly different SATA choices than mine.  Here, I have 
RAID or JBOD (plain SATA, "just a bunch of disks", as my two choices.  
JBOD mode would compare to your AHCI, which is what I'd recommend. (Seems 
Intel wants AHCI to be a standard, thus killing the need for individual 
SATA controller drivers like the SATA_SIL drivers I run here.  That'd be 
nice, but I don't know how well it's being accepted by others.) 
Compatibility mode will likely slow things down, and RAID mode would be 
firmware based RAID, which on Linux would be supported by the device-
mapper (as is LVM2).  JBOD/SATA/AHCI mode, with md/kernel RAID, is 
generally considered a better choice than firmware RAID with device-mapper 
support, well, unless you need MSWormOS RAID compatibility, in which case 
the firmware/device-mapper mode is probably better as it's more compatible.

6 cont) So I'd recommend AHCI.  However, the on-disk layout may be 
different between compatibility and AHCI mode, so it's possible the disk 
won't be readable after switching and you'd need to repartition and 
reinstall, which you were planning on doing anyway, so no big deal.


OK, now that those are covered... what's wrong with your boot?

Well, there's two possibilities.  Either the BIOS isn't finding grub 
stage-1, or grub stage-1 is found and loaded, but it can't find stage 1.5 
or 2, depending on what it needs for your setup.  Either way, that's a 
grub problem.  As long as you didn't make the mistake of putting /boot on 
your LVM, which grub doesn't groke, and since it can pretend md/kernel 
RAID-1 is an ordinary disk, we really don't need to worry about the md/
RAID or LVM until you can at LEAST get to the grub menu/prompt.  

So we have a grub problem.  That's what we have to solve first, before we 
deal with anything else.

Based on that, here's the relevant excerpt from your post (well, after a 
bit of a detour I forgot to include in the above, so we'll call this point 
7):

> NOTE: THIS INSTALL PUTS EVERYTHING ON RAID1. (/, /boot, everything)
> I didn't start out thinking I wanted to do that.

7) Well, not quite.  /boot and / are on RAID-1, yes.  But the guide puts 
the LVM2 physical volume on md4, which is created as RAID-0/striped.  I 
don't really agree with that as striped is fast but has no redundancy.  
Why you'd put stuff like /home, /usr (including stuff you may well want to 
keep in /usr/local), /var (including portage's package database in /var/
db), and presumably additional partitions as you may create them (media 
and mail partitions were the examples I mentioned above) on a non-
redundant RAID-0, I don't know.  That'd be what I wanted on RAID-1, here, 
to make sure I still had copies of it if any of the disks died.

7 cont) Actually, given that md/raid is now partitionable (years ago it 
wasn't, with LVM traditionally layered on top to overcome that), and after 
some experience of my own with LVM, I decided it wasn't worth the hassle 
of the extra LVM layer here, and when I redid my system last year, I 
killed the LVM and just use partitioned md/kernel RAID now.  If you want 
the flexibility of LVM, great, but here, I decided it simply wasn't worth 
the extra hassle of maintaining it.  So I'd recommend NOT using LVM and 
thus not having to worry about it.  But it's your choice.

OK, now on to the grub issue...

> So, the first problem is that on the reboot to see if the install
> worked the Intel BIOS reports 'no bootable media found'. I am very
> unclear how any system boots software RAID1 before software is loaded,
> assuming I understand the Gentoo instructions. The instructions I used
> to install grub where
>
> root (hd0,0)
> setup (hd0)
> root (hd1,0)
> setup (hd1)
> root (hd2,0)
> setup (hd2)

That /looks/ correct.  But particularly with RAID, grub's mapping between 
BIOS drives, kernel drives and grub drives, sometimes gets mixed up.  
That's one of the things I always hate touching, since I'm never quite 
sure if it's going to work or not, or that I'm actually telling it to 
setup where I think I'm telling it to setup, until I actually test it.

Do you happen to have a floppy on that machine?  If so, probably the most 
error resistant way to handle it is to install grub to a floppy disk, 
which unlike thumb drives and possibly CD/DVD drives, has no potential to 
interfere with the hard drive order as seen by BIOS.  Then boot the floppy 
disk to the grub menu, and run the setup from there.

One thing I discovered here is that I could only setup one disk at a time, 
regardless of whether I was doing it from (in Linux, from a floppy grub 
menu, or from a bootable USB stick grub boot menu).  Changing the root 
would seem to work after the first setup, but the second setup would have 
some weird error and testing a boot from that disk wouldn't work, so 
obviously it didn't take.

But doing it a disk at a time, root (hd0,0) , setup (hd0), reboot (or 
restart grub if doing it from Linux), root (hd1,0), setup (hd1), reboot... 
same thing for each additional disk (you have three, I have four). THAT 
worked.

However you do it, test them, both with all disks running, and with only 
one running (turn off or disconnect the others).  Having a RAID-1 system 
and installing grub to all the disks isn't going to do you a lot of good 
if when one dies, you find that it was the only one that had grub 
installed correctly!

There's another alternative that I'd actually recommend instead, however.  
The problem with a RAID-1 boot, is that if you somehow screw up something 
while updating /boot, since it's RAID-1, you've screwed it up for all 
mirrors on that RAID-1.  Since RAID-1 is simply mirroring the data across 
the multiple disks, it can be better to not RAID that partition at all, 
but to have each disk have its own /boot partition, un-RAIDed, which 
effectively becomes a /boot and one more (two in your case of three disks, 
three in my case of four disks, tho here I actually went with two separate 
RAID-1s instead) /boot backups.

That solves a couple problems at once.  First of all, when you first 
install, you install to just one, as an ordinary disk, test it, and when 
it's working and booting, you can copy that install to the others, and do 
the boot sector grub setup on each one separately, as its own disk, having 
tested that the first one is working.  Then you'd test each of the others 
as well.

Second, when you upgrade, especially when you upgrade grub, but also when 
you upgrade the kernel, you only upgrade the one.  If it works, great, you 
can then upgrade the others.  If it fails, no big deal, simply set your 
BIOS to boot from one of the others instead, and you're back to a known 
working config, since you had tested it after the /last/ upgrade, and you 
didn't yet do this upgrade to it since you were just testing this upgrade 
and it broke before you copied it to your backups.

So basically, the only difference here as opposed to the guide, is that 
you don't create /dev/md1, you configure and mount /dev/sda1 as /boot, and 
when you have your system up and running, /then/ you go back and setup 
/dev/sdb1 as your backup boot (say /mnt/boot/).  And when you get it setup 
and tested working, then you do the same thing for /dev/sdc1, except that 
you can use the same /mnt/boot/ backup mount-point when mounting it as 
well, since presumably you won't need both backups mounted at once.

Everything else will be the same, and as it was RAID-1/mirrored, you'll 
have about the same space in each /dev/sd[abc]1 partition as you did in 
the combined md1.

As for upgrading the three separate /boot and backups, as I mentioned, 
when you upgrade grub, DEFINITELY only upgrade one at a time, and test 
that the upgrade worked and you can boot from it before you touch the 
others.  For kernel upgrades, it doesn't matter too much if the backups 
are a bit behind, so you don't have to upgrade them for every kernel 
upgrade.  If you run kernel rcs or git-kernels, as I do, I'd suggest 
upgrading the backups once per kernel release (so from 2.6.32 to 2.6.33, 
for instance), so the test kernels are only on the working /boot, not its 
backups, but the backups contain at least one version of the last two 
release kernels.  Pretty much the same if you run upstream stable kernels 
(so 2.6.33, 2.6.33.1, 2.6.33.2...), or Gentoo -rX kernels.  Keep at least 
on of each of the last two kernels on the backups, tested to boot of 
course, and only update the working /boot for the stable or -rX bumps.

If you only upgrade kernels once a kernel release cycle or less (maybe 
you're still running 2.6.28.x or something), then you probably want to 
upgrade and test the backups as soon as you've upgraded and tested a new 
kernel on the working /boot.

Hope it helps...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman