Re: [gentoo-amd64] Capturing hard hang info?

[Mark Knecht]

Hi Rich,
   Some progress, but questions/comments also

On Sat, Oct 19, 2013 at 6:18 PM, Rich Freeman <rich0@g.o> wrote:
> On Sat, Oct 19, 2013 at 8:25 PM, Mark Knecht <markknecht@×××××.com> wrote:
>> OK, it's a good idea just to have a Konsole terminal open. That might
>> catch something.
>
> I'm not sure if panics show up in konsole.  With a virtual console the
> kernel actually outputs the message.  Konsole under X11 is entirely
> user-mode and I'm not sure that ANY user-mode code can ever run after
> a panic.
>
> I think a virtual console is a better bet.
>

I suspect you're right about Konsole sitting on the KDE desktop. I
only meant that sometimes that catches a few messages and was hopeful
it might do that here but it's certainly not a real solution.

That said I'm not clear about the virtual console point. I thought the
virtual consoles were Alt-Ctl-F[1,2,3,..] When this even occurs my
keyboard isn't working so I don't know how to get there. You must mean
something else?


>> OK, so I remember years ago debugging something for Ingo Molnar using
>> the serial console, but in those days it was a real serial console on
>> a real serial port. None of my machine have those ports anymore. There
>> must be a more modern version of doing that. I'll go look for info.
>> Ethernet? USB? We've recently moved and the only other machine I've
>> got here at the apartment is a Gentoo laptop.
>
> That you'd have to look into.  I'm not sure if the kernel can handle a
> serial console on a PL2302/etc.  It might - it is all kernel-mode I
> think.  You'd have to attach it to another device running a terminal
> emulator, assuming you don't have a vt100/etc lying around.
>
>> There's a gentoo.wiki.org page here:
>>
>> http://wiki.gentoo.org/wiki/Kernel_Crash_Dumps
>>
>> The setup looks reasonably straight forward so I've reconfigured
>> 3.10.17 following those instructions.
>
> Yeah, I forgot - that was actually started based on my blog entry,
> actually.  It may very well have been improved on since.
>

To make progress with /etc/local.d/kdump.start it turns out I also
needed to enable

File Systems -> Pseudo File Systems -> /proc/kcore

The Gentoo wiki only talked about vmcore.

<SNIP>
>> When turned on it has options for Panic (Reboot) for both types. Seems
>> like I probably want that all turned on?
>
> You could try setting it to no and see if you actually can capture any
> meaningful logs that way - there is a chance you could recover your
> system without rebooting.  However, a panic would be the only real
> sure way to ensure a dump.
>
> Oh, and don't forget that there is a magic sysrq that triggers a
> panic.  Only issue with that is that you'll have to hunt around for
> whatever caused the actual hangup because it won't be in the panic
> backtrace (that will just lead you to the sysrq code).
>

At this point I'm a bit beyond my depth. If the hang created by
Virtualbox isn't a panic, but my keyboard is completely locked up,
then I don't know how I'm going to issue the magic sysrq to get the
dump process going.

As a test however, with all of this stuff set up, I logged out of KDE,
switched to a console, disabled X and tried

echo c>/proc/sysrq-trigger

I get a error screen and the system reboots. The first time I did it I
had a bunch of disk activity - presumably stuff being copied to either
kcore or vmcore - and then much later X & KDE came up running a single
processor. This seemed like a positive result.

I then thought maybe I shouldn't start xdm in my init scripts so I
disabled it, rebooted the box, logged in a root and tried again. This
time after the error screen and apparent reboot I gave the machine 45
minutes but never got back to a login screen.

QUESTION 1: This machine has 24GB DRAM. I've set crashkernel=256M and
hoped for the best but don't know if that's a good setting.

QUESTION 2: Am I correct that the captured dump output is going to be
a file that's roughly 24GB? Maybe this takes hours or something being
that it's that big and I'm presumably saving it to a RAID6 which is
doing a lot more parity calcs all in single processor mode. Is there a
way to estimate how long I'd have to wait to even get to a login
prompt?

So far I've been unable to save anything from /proc/vmcore.

>> As I expected about the logs. If the machine's dead then I don't want
>> stuff getting written to disk anyway. kdump sounds like the best
>> solution going right now. I'll try and see if I can get it working.
>
> Yeah - one of these days I'll see if I can get kdump working again.
> What it really needs is an initramfs that will automatically capture
> the dump and reboot.  That's how other distros handle it.  The dumps
> are pretty big though - the size of your RAM.

That would be helpful certainly.

Thanks for all the info and support!

Cheers,
Mark