Re: [gentoo-hardened] kernel upgrade problems: bad page state

[Brian Kroth]

pageexec@××××××××.hu wrote:
> On 31 Oct 2007 at 22:46, Brian Kroth wrote:
>
>> but not ever (yet) with this
>>
>> cactid --verbosity=5 -f 1 -l 10
>
> what does the -l switch do?

-f , -l allow you to limit the range of hostids to scan.  In the 
examples I gave the first scanned hostids 1-100, the example above scans 
hostids 1-10.  I was doing this originally to see if I could pinpoint 
one particular host check that was causing it, but it seems to have more 
to do with large hosts scans.  I think that might be because of the 
number of forks and allocations.

>
>> The version of cactid in portage is slightly old.  After updating from
>> 0.8.6i-r1 to 0.8.6j the problem seems to happen less frequently, but
>> still happens.  With that in mind might this actually be a software
>> problem and not a kernel problem?  Shouldn't PAX be preventing userland
>> software from screwing up the page table?
>
> i'm almost sure it's a bug somewhere in vma mirroring as that's the
> only thing i changed in .22 and on and it does play with page locking
> (the bad page state is triggered because a to-be-freed page is still
> locked, that's means there's a missing unlock somewhere in the code,
> but i couldn't figure it out from the code yet).

Where's the code for this?  I'm no kernel guru by any means, but I'd 
still be interested to look at it and learn.

>
>> I can send more kernel output if anyone's interested.  Any thoughts on
>> what else I should be doing to test this?
>
> i'll need your mm/memory.o from the failing kernel and if it occured on
> multiple machines or kernels, indicate which of your report corresponds
> to which .o (well, i can find it out from the disasm eventually, but it
> helps me if i don't have to ;-). then can you send me a /proc/pid/maps file
> from cactid and nagios (if you use grsec make sure that addresses are not
> hidden and preferably not randomized either)?
>

So far this is only on that single machine, and only for nagios and 
cacti.  I rebuilt the kernel with the config that's attached.  I've 
basically turned on a few more debug settings in the kernel and turned 
off the randomization features of pax (CONFIG_PAX_ASLR) and the "remove 
addresses" feature of grsec (CONFIG_GRKERNSEC_PROC_MEMMAP) like you 
asked.  Tweaked my sec script to copy the maps files before killing the 
offending processes.  Everything should be in the tar.  Let me know if 
you need anything else.

Nov  3 18:27:38 tux-mc IPMI Watchdog: driver initialized
Nov  3 18:32:59 tux-mc Bad page state in process 'nagios'
Nov  3 18:32:59 tux-mc page:c129d620 flags:0x40000001 mapping:00000000 
mapcount:0 count:0
Nov  3 18:32:59 tux-mc Trying to fix it up, but a reboot is needed
Nov  3 18:32:59 tux-mc Backtrace:
Nov  3 18:32:59 tux-mc [<c044c150>] bad_page+0x63/0x92
Nov  3 18:32:59 tux-mc [<c044cc18>] free_hot_cold_page+0x7c/0x194
Nov  3 18:32:59 tux-mc [<c0456110>] do_wp_page+0x22e/0x426
Nov  3 18:32:59 tux-mc [<c0457463>] __handle_mm_fault+0x2ad/0x305
Nov  3 18:32:59 tux-mc [<c0414576>] do_page_fault+0x1da/0x7d5
Nov  3 18:32:59 tux-mc [<c041c269>] do_fork+0x15d/0x217
Nov  3 18:32:59 tux-mc [<c041439c>] do_page_fault+0x0/0x7d5
Nov  3 18:32:59 tux-mc [<c06e9525>] error_code+0x75/0x80
Nov  3 18:32:59 tux-mc [<c06e0000>] svc_setup_socket+0x1aa/0x223
Nov  3 18:32:59 tux-mc =======================
Nov  3 18:35:07 tux-mc Bad page state in process 'cactid'
Nov  3 18:35:07 tux-mc page:c12efdc0 flags:0x40000001 mapping:00000000 
mapcount:0 count:0
Nov  3 18:35:07 tux-mc Trying to fix it up, but a reboot is needed
Nov  3 18:35:07 tux-mc Backtrace:
Nov  3 18:35:07 tux-mc [<c044c150>] bad_page+0x63/0x92
Nov  3 18:35:07 tux-mc [<c044cc18>] free_hot_cold_page+0x7c/0x194
Nov  3 18:35:07 tux-mc [<c0456110>] do_wp_page+0x22e/0x426
Nov  3 18:35:07 tux-mc [<c0457463>] __handle_mm_fault+0x2ad/0x305
Nov  3 18:35:07 tux-mc [<c0414576>] do_page_fault+0x1da/0x7d5
Nov  3 18:35:07 tux-mc [<c04682d5>] sys_read+0x68/0x6a
Nov  3 18:35:07 tux-mc [<c041439c>] do_page_fault+0x0/0x7d5
Nov  3 18:35:07 tux-mc [<c06e9525>] error_code+0x75/0x80
Nov  3 18:35:07 tux-mc [<c06e0000>] svc_setup_socket+0x1aa/0x223
Nov  3 18:35:07 tux-mc =======================


PS - would you like me to take this off list?

Thanks again,
Brian