[gentoo-amd64] Re: hp proliant 585 cciss grub problems and solving

[Duncan]

Benny Pedersen posted on Wed, 22 Sep 2010 03:30:35 +0200 as excerpted:


> with 2.6.35 there is problem to load firmware to tg3 ethernet, it works
> in 2.6.34

As I run direct Linus-tree git (tho I'm a couple rcs behind ATM), even 
2.6.35 is somewhat back in history for me.  But IIRC, somewhere in there 
(tho I had thought it was 2.6.34??), they switched around the kernel 
firmware loading options a bit, and a make oldconfig didn't really quite 
do it in terms of getting the new settings working like the old ones did.  
So that might be you're seeing.  Get it setup correctly, however, and it 
should again be "totally copacetic, man!" =:^)

FWIW, I have tg3 here and it works fine.  Actually, at least my hardware 
works fine even without the firmware -- all the firmware does is hardware 
offloading of something or other, AFAIK, so it simply handles it in 
software without it.  But the kernel will complain about not having the 
firmware if it can't find it, so loading it shuts up that complaint and as 
mentioned, apparently does a bit more in the hardware/firmware instead of 
on the main CPU, with it.

Anyway, here's how I'm configured here.  (It's worth noting that I build-
in everything, so no modules to load or keep track of, firmware built-in 
as well.  FWIW, I'm a free software guy and won't run proprietary drivers, 
but if it's simply ROM alternative firmware, I don't worry about it -- 
that's where I personally draw my line.  So I don't worry about either the 
tg3 or the radeon firmware I run.)

In menuconfig under Device Drivers, Generic Driver Options, Include in-
kernel firmware blobs in kernel binary, CONFIG_FIRMWARE_IN_KERNEL.  (You 
can probably do without this if you load tg3 as a module.).  Also, in the 
same place, Userspace firmware loading support CONFIG_FW_LOADER, altho 
here, that's hard-configured on by some other option I have configured, 
it's not something I can disable.  I also have Firmware blobs root dir set 
to /lib/firmware, tho I think that's the default.  Finally, under normal 
conditions, Prevent firmware from being built, 
CONFIG_PREVENT_FIRMWARE_BUILD, is turned on here, but AFAIK I had to turn 
it off temporarily to build the firmware once after things moved around, 
then I could turn it on again.

The other piece of the puzzle, which didn't /used/ to apply to the tg3 
firmware and I don't /believe/ it does yet, but it may well apply at some 
point in the future, is that they're gradually moving firmware that 
/used/ to be bundled with the kernel into a separate firmware package.  In 
Gentoo, this is the sys-kernel/linux-firmware package.  If you can't find 
the firmware file it's complaining about in either the kernel tree or the 
/lib/firmware/ location, you may need to install that package.

> it leads me to the next question, is there not anyplace where a default
> kernel config exists for most big server platform, or does others just
> customize it so much there is not a common good one ?

I'm guessing most server folks, at least Gentoo server folks, customize 
their kernel config.  In particular, for publicly accessible servers, 
building all modules into the kernel itself and disabling module loading 
entirely is considered the secure way to do it, because it kills the 
chance a cracker might otherwise have of loading a custom kernel module 
with all his "toys".  Since a no-module kernel is by definition a 
customized kernel...

> any tips to share ?

I'm not working at that level and not dealing with a publicly accessible 
server (nor with virtualization, unfortunately my hardware is pre-hardware-
virtualization-support =:^( ), but my hardware is now aging server-grade 
from 2003/2004-ish (looks like yours is similar age, but you're running 8-
cores total while I'm running dual dual-cores, four-cores total, and I'm 
only running 8 gig RAM compared to your 32), and I play around with kernel/
md-RAID and the like.

Some tips in general, tho.  These aren't knew and you may well already 
know them if you've been around for awhile.

1)  Are you going to be running publicly accessible or not?  At least at 
the level the machine is public (thus at least virtualized if it's all VMs 
that are public, not "domain 0", but the whole machine if not everything 
publicly accessible is virtualized), the secure configuration is don't 
have a build environment available.  Build the system on another machine, 
and only install (from binpkg, or better yet, rsync, and don't even have 
portage on the public server) what's necessary to run the public servers 
on the publicly accessible machines.

Of course, if it's not publicly accessible and all use is at least semi-
trusted, that's much less of an issue.  If it's your workstation/play-
thing as is the case here (or if it's your dedicated build-server, 
obviously), it's probably less trouble to do all your builds on the 
machine itself, putting all that memory and CPU horsepower to work.

2) If you are using it as a build machine, and this really applies to 
anything with 4 gig or more of RAM, pointing PORTAGE_TMPDIR at a tmpfs 
mounted filesystem DRAMATICALLY speeds up your builds.  General rule of 
thumb is a minimum 4 gigs tmpfs (on 4 gig systems it can swap if 
necessary, still tends to be faster than building on conventional spindle 
backed filesystems) and at least a gig per core, so you'll want to set 
your tmpfs max size to at least 8 gigs.

From my experience, the kernel doesn't get interactive-unresponsive from 
just build-jobs alone, as long as you're not hitting the disk too hard.  
So with for instance the very highly parallelizable kernel builds that use 
rather less memory and scratch space per job than most compiles will, it's 
possible to run a multi-hundred load-average even on a dual-core, and as 
long as you're not swapping or otherwise hitting disk too hard, the system 
remains actually reasonably responsive /especially/ when you consider it's 
running a multi-hundred load average!

As a result, what I tend to do is set a reasonably high -j number (or 
simply -j, unlimited), then use the -l load average limit, not to limit 
load average itself, but as an indirect way to control the memory usage.  
On my older dual-dual-core, with 6 gigs RAM (it's normally 8 but a stick 
died and I've not replaced it) and with portage building in a 6-gig tmpfs, 
I use MAKEOPTS="-j13 -l10", and use emerge --jobs=4 --load-average=7.  
That causes make to try to trigger more jobs if it can to try to keep a 
load average of 10, but if it can't and the load-average falls below 7, 
portage will start merging another package in parallel (if the dependency 
tree will let it do so), with upto 4 packages merging at once.

IIRC with the full 8 gigs, I ran something like -j20 -l16 MAKEOPTS (so a 
load of four per core), with emerge --jobs=10 --load-average=12.  With a 
full 32-gig RAM and 8 cores, you should be able to comfortably double 
that, tho I'd play around with it a bit, as it'll depend on your hardware 
to some extent.  You /probably/ don't want to go /too/ much above that, 
not because your system shouldn't handle it (it should), but because at 
some point, the system starts spending more time switching between jobs as 
opposed to actually doing them.  Once you're running near 100% most of the 
time, additional jobs are likely to actually slow things down.  I'm not 
positive of this, but given what I understand of Intel hardware of that 
age, it actually didn't parallel as well as AMD hardware did, due to 
memory bottlenecks, etc, that AMD's on-board memory controllers avoided.  
So certainly, do some experimenting, but with 32-gig memory and 8 cores, 
something like -j40 -l32 and emerge --jobs=10 --load-average=20, should be 
a good ballpark to start with.

(Note that in my experience, portage rarely paralleled more than five 
package emerges at once in any case, and I think the most I've seen was 
seven, no matter how high the --jobs and --load-average is set.  It may 
actually have gotten a bit better at it now, as the parallel merging 
functionality was pretty new back when I was experimenting with that, but 
at least back then, it seemed to be some built-in limitation on its 
dependency resolver -- it just wouldn't start more no matter how many I 
gave it permission to parallel.  So that's why I didn't up the --jobs= 
beyond 10, it didn't seem to make an difference anyway, at least here.  
Perhaps with your more-cores hardware and a newer portage, it'll actually 
run more than 5-7 parallel merges at once, given the chance, but I was 
never able to get it to.)

Finally, you probably want PORTAGE_NICENESS=19 as well.  Not only does 
that lower the build priority to idle, so other tasks including 
interactive responsiveness takes priority, but at nice=19, the kernel 
automatically puts that task in batch-scheduling mode as well, giving it 
longer timeslices, so it can get more done in each one with less task-
switching overhead.  Thus, on an otherwise lightly loaded system, it's 
actually quite possible that a heavy CPU cycles oriented task like 
building packages will complete faster at nice=19 than it would at nice=0, 
because at nice=19, the overhead from task-switching goes down.

I'll throw in one more as well, tho this has little or nothing to do with 
servers, but applies to pretty much everyone.  One of my favorite under-
advertised portage features is FEATURES=buildpkg.  To enable it you'll 
want to allow at least 4 gigs of package space for it -- I have my PKGDIR 
set to its own dedicated 4-gig partition and I can let it accumulate about 
two complete package sets, more of many packages, before I start running 
low on room, but if you've hundreds of gigs of disk space without anything 
to use it for, allowing 6 or 8 gigs would allow you to forget about it for 
quite some time (8 gigs will likely be over a year's worth of packages) 
before worrying about cleaning it out.  The binpkgs this produces have 
come in handy many times, here.  Among other things, they can be useful if 
your gcc ever breaks, because you can still install an older version from 
binpkg.  And because the binpkg format is simply a bzip2ed tarball with 
some extra metadata tacked on the end, you can browse them with your 
favorite archiver as well.  This allows you to check what files the 
package contained two versions ago, for instance, or compare the default 
config files between versions or against what's installed on your live 
system.  As a last resort, it's also possible to repair broken portage or 
python installations by simply untarring the package directly over the 
live filesystem (best to backup config files first, as that'll bypass 
portages config-protect system, etc), bypassing the broken portage 
entirely in ordered to get a working system once again.  (If you do this, 
once you do get a working portage and have replaced the default configs 
with the versions you backed up once again, the first thing to do is 
remerge the working binpkg back over itself, since you bypassed portage 
with the untar, so it knows nothing about it and still thinks you have the 
broken version installed.  This will sync portage's installed package 
database with what's on the disk once again.  As mentioned, this is a last 
resort /because/ of the config file and out-of-sync portage database 
issues, but if you have a broken portage and are otherwise stuck, it can 
save your behind, as it has mine, a couple times.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman