Re: [gentoo-server] Re: linux-2.6.8-gentoo-r3 .config for RAID5, help?

[Chris Smart]

OPPS :S
accidentally sent before I'd finished.

see below :)

> Chris> Out of interest, what mountpoint is /dev/md0? /home?
>
> It is /mnt/md0.
>
> Then I do:
> ln -s /mnt/md0/usr to /usr
> ln -s /mnt/md0/var to /var
> ln -s /mnt/md0/home to /home
> ln -s /mnt/md0/tmp to /tmp
>
> This actually caused problems, because some of the links in /usr/bin
> were to ../../sbin/xyz, but since /usr/bin/../.. is actually /mnt/md0,
> these links didn't resolve right.
>
> I wish I knew a better way, but I want to have /usr, /var, and /home
> all be on the same huge partition.  I don't want to bust up my 1.5TB
> partition just for a /usr and /var, and I don't have enough space on
> my / partition for both of them.

Hmm.. ok here's the issue:

/dev/md0 is a device, it needs to go to a mountpoint.

So, /dev/md0 must be mounted somewhere, like / or /home or /usr etc..

ahh hangon ic, have you mounted /dev/md0 to /mnt/md0??

this is a crazy way of doing things.. how did you create your links?

normally for raid5 you need to create seperate raid devices for seperate
 mount points.

so if you want /usr and /home under raid for example, then you need TWO raid
devices:
/dev/md0 and /dev/md1 (or whatever you want to name them)
then you mount /dev/md0 to /usr and /dev/md1 to /home.

by default all mountpoints go under /

so if you only have one big / partitions, then /usr /home etc will all go on
that partition.

if you want to seperate them, you need to create seperate partitions, and
mount them accordingly.

what you have done sounds rather odd, but hey - it's linux it's about your
choice! ;)

you could "cp -ap /usr /mnt/md0/" and then symlink /usr to /mnt/md0/usr/, and
the same for any other partitions you want, but like I said, this is a big
dodgy :)

> Chris> What did you do to compile your kernel?
>
> make && modules_install

you need to do:

make modules modules_install

then if you like you can even do make install, which will copy bzImage
to /boot/vmlinuz

> Chris> Did you use genkernel --menuconfig all?
>
> No, I didn't realize it was necessary.  Is that how genkernel figures
> out what modules to put in the initrd?  If that's the case, I could go
> back to using modules.

you don't have to use genkernel, but yes it will create an init for you if
 you want it.

> Chris> Can you post your bootloader config?
>
> I'm appending my current .config and my /boot/grub/grub.conf
>
> Chris> Run make menuconfig.  Under Device Drivers
> Chris> --->ATA/ATAPI/MFM/RLL support ---> there is an option in the
> Chris> kernel: [*] Boot off-board chipsets first support
> Chris> make that static, and it will boot your via controller first
> Chris> (ie make it /dev/hda)
>
> I don't understand --- doesn't this FORCE my PCI IDE controller to be
> first?

this forces linux to boot your on-board ide controller first (your onboard
controller is the one on your mainboard which is the VIA controller.

The VIA controller will become /dev/hda-d
The silicon image and other ide controllers will become /dev/hde and so on.

> I have one final question:
>
> Now that I've gone out and bought two more dual-bus IDE controllers,
> how do I know which disk was in which order in the RAID5 chain so that
> I can get the chain back again after all the drive names change?

yes, this is the question!

linux will access your pci ide controllers based on IRQ.

So you now have 3 controllers? is that right?

Each controller will have 4 devices links under linux.

that means you will have:

PCI controller 1:
/dev/hde
/dev/hdf
/dev/hdg
/dev/hdh

PCI Controller 2:
/dev/hdi
/dev/hdj
/dev/hdk
/dev/hdl

PCI Controller 3:
/dev/hdm
/dev/hdn
/dev/hdo
/dev/hdp

But because you want to use only PRIMARY hard drives on one channel each, then 
you will only end up using:

PCI controller 1:
/dev/hde
/dev/hdg

PCI Controller 2:
/dev/hdi
/dev/hdk

PCI Controller 3:
/dev/hdm
/dev/hdo

I would get a pen and MARK your hard drives as they currently sit:

/dev/hda = 1
/dev/hdc = 2
/dev/defgh = 3,4,5,6,7

/dev/hda will stay on the via controller

/dev/hdc will move to channel 1 on the first ide controller
/dev/hdd will be MASTER and go to channel2 on the first ide controller
/dev/hde will move to channel 1 on the second ide controller
/dev/hdf will be MASTER and go to channel2 on second controller
/dev/hdg will move to channel 1 on the third ide controller
/dev/hdh will be MASTER and go to channel 2 on the third controller

IE: this is what it will become:
/dev/hdc =/dev/hde
/dev/hdd = /dev/hdg
/dev/hde = /dev/hdi
/dev/hdf = /dev/hdk
/dev/hdg = /dev/hdm
/dev/hdh = /dev/hdo


NB!!:
As I mentioned above, linux will order the controllers based on IRQ.

So linux might actually see the controller in slot 2 before the controller in 
slot 1.

Honestly I am not sure whether it will destroy your raid5 array if the drive 
order is changed, so you might be better off making sure the order is the 
same!

You specified the devices when you created your raidtab. But AFAIK, linux only 
reads that when it creates the md devices. When linux boots it does not read 
the raidtab, but looks as the ID on each partition, and therefore knows which 
partition is a part of which array.
What I DON'T know is whether it also numbers them, so it goes "oh yeah this 
partitions is number ONE on THAT array".

So unless someone else can verify that, then it might be better to make sure 
they are in the same order anyway.

The only way I can think to check which order linux sees them is to add one 
controller at a time, and work it out yourself.

Remember the device layout I set above?

PCI controller 1:
/dev/hde
/dev/hdf
/dev/hdg
/dev/hdh

PCI Controller 2:
/dev/hdi
/dev/hdj
/dev/hdk
/dev/hdl

PCI Controller 3:
/dev/hdm
/dev/hdn
/dev/hdo
/dev/hdp

I would have one master hard drive connected to your current controller 
(so /dev/hde), then I would put a SLAVE on channel 1 of the first new ide 
controller.
Boot to your kernel - or gentoo livecd, and if the new controller is detected 
AFTER the first, then that slave drive SHOULD be /dev//hdj.
If it is /dev/hdf, then the NEW controller was detected before the old ide 
controller (and /dev/hde would be /dev/hdi, not hde).

Make sense?

Then you need to add the third controller, and I would put TWO MASTER drives 
on each channel (just so you can tell the differences between the controllers 
based on what hard drives are connected).

Boot back to your kernel or livecd. 

If the controllers were all detected in the order you put them in, then you 
should have:
/dev/hde (the first hdd)
/dev/hdj (the slave you put in second)
/dev/hdm (one of the 2 master drives you put in third)
/dev/hdo (the other master drive)

If you get something like:
/dev/hde (the first hdd)
/dev/hdi (one of the 2 master drives you put in third)
/dev/hdk (the other master drive)
/dev/hdn (the slave you put in second)

then the second and third controllers are being detected the other way around.

Make sense?

You'll have to see what devices are which, and then I would number your ide 
controllers.

Then put the right hard drives on the right controllers.

There are other ways than the above, like specifying certain IRQs to each PCI 
slot, but it's still tricky cause some are bound to use the same IRQ etc etc.

Also maybe edit your raid tab to reflect the new devices incase linux DOES 
read that too (but I don't think so cause if /etc/ if raided it can't read 
raidtab in order to create the array! ;))

Man I hope you have some cooling in that case of yours!

Hope this helps, and doesn't destroy your RAID5 data.

Chris

> Thanks,
> Dave

-------------------------------------------------------