Re: [gentoo-server] kernel oops

[Kerin Millar]

On Thu, 2004-09-30 at 15:54 +0300, Alex Efros wrote:
> Hi!
>
>
> My server hangs every 3-14 days without storing kernel oops message in logs
> (this is dedicated server at hosting, so I've no physical access to console).
> I've set up netconsole, and catch kernel oops by network on second server
> (error message below).
>
> These hangs happens on different kernel versions (current is 2.6.8-gentoo-r3).
> "SpiderAuto" process is my perl script which running using usual user account
> and 24x7 downloading websites (there number (3-7) of such scripts running
> doing parallel download of different websites).
>
> I suppose this is sort of "race condition" error related to huge number of
> simultaneous download requests...
>
> Any ideas how to fix/workaround this error? Maybe try another kernel source
> (I'm usually using gentoo-dev-sources)?

[snip]

Ouch, that's a nasty one. I suspect dabbling in sources variations will
not help a great deal because the gentoo-dev-sources are so lightly
patched in the first place. If anything, try 2.6.9-rc3. I performed a
cusory glance over the ChangeLog for "[NETFILTER]" and, while a few
patches have been applied, there was nothing that immediately suggested
that it would alleviate your problem.

Have you compiled in the ipchains/ipfwadm
(CONFIG_IP_NF_COMPAT_IPCHAINS / CONFIG_IP_NF_COMPAT_IPFWADM) support by
any chance? Apparently, it's rather buggy. For instance, this post
mentions a bug in find_appropriate_src() which only occurs when the
backward compatibility options are available:
http://www.gelato.unsw.edu.au/linux-ia64/0310/7353.html. See this also:
http://lists.netfilter.org/pipermail/netfilter-devel/2003-
October/012872.html.

Here's a description of the purpose of find_appropriate_src():
http://lists.netfilter.org/pipermail/netfilter-devel/2004-
March/014418.html.

If things persist, try stripping down you kernel to a bare-bones
configuration. Enable CONFIG_DEBUG_KERNEL, CONFIG_FRAME_POINTER and
CONFIG_MAGIC_SYSRQ. Avoid estoric options such as CONFIG_4KSTACKS,
CONFIG_REGPARM etc if possible. A futher comparative could be to also
avoid using modules where you know you need something (such as
iptable_nat and e1000).

You can emerge and use ksymoops to decode an oops message (such as the
one you provided in your post). I believe it works best if the system is
still operable after the oops, otherwise you can reboot and then decode
the message. This is the sort of thing that any hardcore kernel hacker
would need to see!

If nothing seems to resolve the problem then it might be best if you
prepare a slightly more detailed post for the netfilter mailing list.
You might also want to review the lists for related posts:
http://news.gmane.org/search.php?match=netfilter. You can use a
newsreader on news.gmane.org also I believe.

Good luck,

--Kerin Francis Millar