Re: [gentoo-hardened] kernel upgrade problems: bad page state

[Brian Kroth]

Remy Bosch wrote:
> Brian Kroth wrote:
>> I started getting these messages after an upgrade from
>> hardened-sources-2.6.20-r10 to 2.6.22.r8.  Nothing changed from the
>> old config.  I just did "make oldconfig" and answered N to everything
>> since they weren't relevant.  In particular, nothing in PAX or GrSec
>> menus were changed, though whenever I get big nasty messages like this
>> I suspect one of them.  Can anyone give me some direction as to how to
>> fix this or what might be causing it?  I can post the full configs if
>> need be.
>>
>> Thanks,
>> Brian
>
> I had bad-page errors as well. What I did was wait to see if the adres
> stayed the same. In my case it was, and I can only conclude it was bad
> ram. And with 2GB for only desktop useage, I didnot get the massage
> often. Besides, in my case memtest did not get the faulty page. Even
> after 8+ runs. atm I add the badram-patch to my kernel, so i can pass
> the faulty page address to the kernel, in order to have it ignored.
>
> I got this page broken:
> page:c1b1fdc0 flags:0x80000000 mapping:00000001 mapcount:0 count:0
>
>
> Goodluck!
>
> Remy
>
>

Where's this badram-patch and who supports it?  memtest also didn't 
report any errors for me.  I let it run for 9 passes and decided to try 
again.  This is a server with ECC ram, so I would think it would catch 
it, but I can't say for certain since I don't know the details well 
enough (at least at the moment).  A quick grep/sed of kern.log shows the 
output below, so I don't think it's the same page, but with the 
randomization features I'd have to think about it some more to be sure. 
  It also seems strange to me that only nagios and cactid would crap out 
on this.  If I run the cacti poller just through php I get no errors. 
The server also hosts ntp, portage over NFS, syslog-ng, mysql, dovecot, 
sec, postfix, and a bunch of management scripts for other servers.  It's 
a busy thing, so if bad ram were the case, I'd expect to see them in 
other places.  Possibly just bad code on cactid/nagios' part?  Should 
PAX complain then?

For now, I'm going to roll back to my 2.6.20-r10 config to see if I can 
reproduce the errors there.  I'm also adding a rule to sec to warn me 
about this and attempt to stop the service the next time it happens. 
The first time it happened cactid drove the load up so high on the 
machine that I had no other recourse than to power cycle it via IPMI.

Also including emerge --info for anyone who's interested.  Should I be 
opening a bug for this?

# pcregrep " page:" /var/log/kern.log | sed 's/.*\(page:\S\+\).*/\1/' | 
perl -e 'while (<>) { $data{$_}++ }; for (keys %data) { print $data{$_} 
. " " . $_ ; };'

1 page:c14320e0
1 page:c12afe80
1 page:c12cb860
1 page:c159a480
1 page:c1392da0
1 page:c1203f60
1 page:c112b560
1 page:c124b620
1 page:c154bcc0
1 page:c1012fe0
1 page:c1416200
1 page:c11c53c0
1 page:c1358a00
1 page:c16baec0
1 page:c10a20c0
1 page:c1278ac0
1 page:c14bf880
1 page:c105ef00
1 page:c1211b20
1 page:c10ab540
1 page:c11fd320
1 page:c10cb440
1 page:c1434c20
1 page:c1412fc0
1 page:c1060e80
1 page:c1278ae0
1 page:c1101ae0
1 page:c13071e0
1 page:c11fd400
1 page:c1467da0
1 page:c16beb20
1 page:c152d5e0
1 page:c11bc800
1 page:c138c720
1 page:c1132400
1 page:c17fdd40
1 page:c11c54a0
1 page:c11d0f20
1 page:c105cae0
1 page:c16ce580
1 page:c170fac0

emerge --info
Portage 2.1.3.16 (hardened/x86/2.6, gcc-3.4.6, glibc-2.6.1-r0, 
2.6.22-hardened-r8 i686)
=================================================================
System uname: 2.6.22-hardened-r8 i686 Intel(R) Xeon(TM) CPU 2.80GHz
Timestamp of tree: Tue, 30 Oct 2007 06:00:01 +0000
distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) 
[disabled]
ccache version 2.4 [enabled]
app-shells/bash:     3.2_p17
dev-lang/python:     2.4.4-r6
dev-python/pycrypto: 2.0.1-r6
dev-util/ccache:     2.4-r7
sys-apps/baselayout: 1.12.9-r2
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.13, 2.61-r1
sys-devel/automake:  1.7.9-r1, 1.10
sys-devel/binutils:  2.18-r1
sys-devel/gcc-config: 1.3.16
sys-devel/libtool:   1.5.24
virtual/os-headers:  2.6.22-r2
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=pentium4 -O2 -pipe -fforce-addr"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/hotplug /etc/hotplug.d 
/etc/init.d /etc/php/apache2-php5/ext-active/ 
/etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ 
/etc/revdep-rebuild /etc/terminfo /etc/udev /etc/udev/rules.d"
CXXFLAGS="-march=pentium4 -O2 -pipe -fforce-addr"
DISTDIR="/mnt/build/distfiles"
EMERGE_DEFAULT_OPTS="--nospinner"
FEATURES="buildpkg ccache collision-protect metadata-transfer sandbox 
sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://gentoo.chem.wisc.edu/gentoo/ 
ftp://gentoo.chem.wisc.edu/gentoo/ http://gentoo.mirrors.tds.net/gentoo 
ftp://gentoo.mirrors.tds.net/gentoo http://gentoo.osuosl.org/ 
ftp://distro.ibiblio.org/pub/linux/distributions/gentoo/ 
http://distro.ibiblio.org/pub/linux/distributions/gentoo/ 
http://distfiles.gentoo.org"
MAKEOPTS="-j5"
PKGDIR="/mnt/build/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times 
--compress --force --whole-file --delete --delete-after --stats 
--timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages 
--filter=H_**/files/digest-*"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/mnt/build/portage"
PORTDIR_OVERLAY="/mnt/build/portage-local"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="acl acpi apache2 bash-completion berkdb bzip2 caps chroot cracklib 
crypt erandom fam gmp gpm hardened jpeg lm_sensors logrotate maildir mmx 
ncurses nls nptl pam pcre perl pic png python readline smp snmp sse sse2 
ssl syslog tcpd threads vhosts x86 xattr xml xpm" 
ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug 
file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null 
plug rate route share shm softvol" ELIBC="glibc" INPUT_DEVICES="mouse 
keyboard" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk 
hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, 
PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS