Re: [gentoo-user] Re: How does grub assemble a RAID1 for / ??

[Mark Knecht]

On Sun, Apr 4, 2010 at 1:06 PM, Kerin Millar <kerframil@×××××.com> wrote:
> On 04/04/2010 20:52, Mark Knecht wrote:
>>
>> So this suggests that because I did the install from a running Gentoo
>> chroot that the mknod commands didn't stick? Somehow they were part
>> of, or because part of, the host Gentoo non-RAID install? Interesting.
>
> I don't know what the handbook says these days, but I understand that it's
> lately taken to recommending a bind mount of /dev. If that's the case, then
> none of the static device nodes that are contained with the /dev directory
> within the stage3 tarball will actually make it on to the root filesystem
> because, at the time of unpacking, they will be redirected to the (volatile)
> tmpfs mount at /dev with respect to the livecd environment.
>
> From the point of view of the problem you're having, it may not even matter
> but it's one thing to try anyway.
>
> It's interesting that the component disks are available. In principle,
> there's no reason why the array should not be assembled. Here's an example
> of how it looks during the boot process on one of my systems:
>
>  Command line: root=/dev/md2
>  Kernel command line: root=/dev/md2 quiet
>  ata1: SATA max UDMA/133 cmd 0xcc30 ctl 0xcc28 bmdma 0xcc40 irq 23
>  ata2: SATA max UDMA/133 cmd 0xcc38 ctl 0xcc2c bmdma 0xcc48 irq 23
<SNIP>
>
> You might consider disabling the framebuffer temporarily, as it may impact
> upon your ability to see what's going on early during the boot process.
>
> Cheers,
>
> --Kerin

Hi Kerin,
   First, thanks for sticking with me on this. I really appreciate it.
Second, I apologize for the length of the reply but it's still not
working and I wanted to try and clearly show the steps I've taken.
Maybe you or someone else will see the step I'm missing.

   OK, more and more it's looking to me like on the RAID I'm just not
getting the /devmdX special file that has to be there to
mount/bind/whatever the hardware to. What creates it, other than
mknod? Can I do it by hand some other way? Read on...

1) Booted non-RAID and look in /dev for md*. md0 is there. I mount the
RAID and look in the directory /mnt/gentoo/dev/ for anything named
md*. There's nothing.

2) Booting with the Live CD I modprobe raid1, assemble the RAID, and
then do the recommended chroot:

livecd usr # cd /
livecd / # mount -t proc proc /mnt/gentoo/proc
livecd / # mount -o bind /dev /mnt/gentoo/dev
livecd / # cp -L /etc/resolv.conf /mnt/gentoo/etc/
livecd / # chroot /mnt/gentoo /bin/bash
livecd / # env-update && source /etc/profile

3) I look in /dev and still no md*. (There shouldn't be)

4) Because I used md3 to get into the RAID from the Live CD I create
some new devices, but specifically don't do md3:

livecd / # mknod /dev/md0 b 9 0
livecd / # mknod /dev/md1 b 9 1
livecd / # mknod /dev/md2 b 9 2
livecd / # mknod /dev/md4 b 9 4
livecd / # mknod /dev/md5 b 9 5

5) ls the directory and I see the ones I created plus md3 bound from
the chroot above:

livecd / # ls /dev/md*
/dev/md0  /dev/md1  /dev/md2  /dev/md3  /dev/md4  /dev/md5

/dev/md:
3

6) I exit the chroot and look again at what's now in the Live CD /dev.
As expected I still see all 6.

livecd / # exit
exit
livecd ~ #
 ls /dev/md*
/dev/md0  /dev/md1  /dev/md2  /dev/md3  /dev/md4  /dev/md5

/dev/md:
3
livecd ~ # df
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                  3052532     29600   3022932   1% /
/dev/sr0                117392    117392         0 100% /mnt/cdrom
/dev/loop0               87424     87424         0 100% /mnt/livecd
udev                     10240       216     10024   3% /dev
tmpfs                  3052532      6152   3046380   1%
/mnt/livecd/lib64/firmware
tmpfs                  3052532         0   3052532   0% /mnt/livecd/usr/portage
/dev/md3              30969528   2254188  27142180   8% /mnt/gentoo
livecd ~ #


7) Try booting RAID a couple of times with different command line
options. It still fails the same way.

8) Boot back to non-RAID and mount the RAID, this time as md0 just to
be different. I look in /dev and see only md0, as expected:

keeper ~ # df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             30969600   7244840  22151596  25% /
udev                     10240       228     10012   3% /dev
shm                    3053512         0   3053512   0% /dev/shm
/dev/md0              30969528   2254188  27142180   8% /mnt/gentoo
keeper ~ # ls /dev/md*
/dev/md0

/dev/md:
0
keeper ~ #

9) I look in /mnt/gentoo/dev to see if ANY md special files are there.
There are none:

keeper ~ # ls -al /mnt/gentoo/dev/md*
ls: cannot access /mnt/gentoo/dev/md*: No such file or directory
keeper ~ #


All the other stuff is there, but not the special files I created
while in the chroot.

So, to me, this comes down to something like when I'm booting /dev/md0
or md3 isn't available so the kernel cannot mount the RAID anywhere.
What creates the special file, at least by hand, is mknod, and for
some reason I'm unable to make that work for me. Either there's
something weird about doing this on RAID, or there's something stupid
that I'm still doing. (I vote for the latter although I've been doing
it for days so I probably cannot even see it anymore.) I guess I would
hope that some piece of software is supposed to make these special
files on the fly when booting if they don't exist. Maybe that's the
purpose of appending md=0,/dev/sdb3,/dev/sdc3 but even that's not
working for me.

Maybe that points to something in kernel config that turns that on? I
don't know.

Note that in the Gentoo RAID install guide that I'm following the only
mknod step is pre-chroot so nothing in the final RAID design ever does
that by hand. If that guide actually results in a working system

http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml

I've tried all the boot options shown at the end.

I don't know what to try next.

Cheers,
Mark

RAID fstab:

/dev/sda1		/boot		ext2		noauto,noatime	1 2
/dev/md0		/		ext3		noatime		0 1
/dev/sda2		none		swap		sw		0 0
/dev/sdb2		none		swap		sw		0 0
/dev/sdc2		none		swap		sw		0 0
/dev/cdrom		/mnt/cdrom	auto		noauto,ro	0 0

grub.conf:

default 0
timeout 30
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

title /dev/sda1 Gentoo Linux 2.6.33-gentoo
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo root=/dev/sda3

title /dev/sda1 Gentoo Linux 2.6.33-gentoo-RAID using md0
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo-RAID root=/dev/md0

title /dev/sda1 Gentoo Linux 2.6.33-gentoo-RAID using md3
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo-RAID root=/dev/md3

title /dev/sda1 Gentoo Linux 2.6.33-gentoo-RAID explicit md0=...
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo-RAID root=/dev/md0
md=0,/dev/sdb3,/dev/sdc3

title /dev/sda1 Gentoo Linux 2.6.33-gentoo-RAID explicit md3=...
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo-RAID root=/dev/md3
md=3,/dev/sdb3,/dev/sdc3

title /dev/sda1 Gentoo Linux 2.6.33-gentoo-RAID sdb3 only
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo-RAID root=/dev/sdb3

title /dev/sda1 Gentoo Linux 2.6.33-gentoo-RAID sdc3 only
root (hd0,0)
kernel (hd0,0)/boot/bzImage-2.6.33-gentoo-RAID root=/dev/sdc3