Re: [gentoo-amd64] Capturing hard hang info?

[Rich Freeman]

On Sun, Oct 20, 2013 at 1:29 PM, Mark Knecht <markknecht@×××××.com> wrote:
> That said I'm not clear about the virtual console point. I thought the
> virtual consoles were Alt-Ctl-F[1,2,3,..] When this even occurs my
> keyboard isn't working so I don't know how to get there. You must mean
> something else?

It will only be helpful if the console is displayed when the panic
occurs.  This is helpful when the panics tend to happen when you're
away from the system.

> To make progress with /etc/local.d/kdump.start it turns out I also
> needed to enable
>
> File Systems -> Pseudo File Systems -> /proc/kcore
>
> The Gentoo wiki only talked about vmcore.

Feel free to update it, after you're sure you have everything figured out.

> At this point I'm a bit beyond my depth. If the hang created by
> Virtualbox isn't a panic, but my keyboard is completely locked up,
> then I don't know how I'm going to issue the magic sysrq to get the
> dump process going.

Are you SURE that it is COMPLETELY locked up?  As in alt-sysrq-b
doesn't reboot?  I've found that this almost always works, even if
sysrq otherwise appears to not work (most of the other options won't
appear do anything in a panic with the display not on a virtual
console while in a graphics mode).

> I get a error screen and the system reboots. The first time I did it I
> had a bunch of disk activity - presumably stuff being copied to either
> kcore or vmcore - and then much later X & KDE came up running a single
> processor. This seemed like a positive result.

Nothing gets "copied" to kcore/vmcore.  The state of the previous
system is already in RAM, or at least it was until KDE launched and
overwrote everything.  You need to have it boot to a crash kernel and
rescue shell for it to be of any use, unless you just want it to
auto-reboot to minimize downtime.

> QUESTION 1: This machine has 24GB DRAM. I've set crashkernel=256M and
> hoped for the best but don't know if that's a good setting.

It is probably fine - it just needs enough space to hold the kernel
and initramfs.

>
> QUESTION 2: Am I correct that the captured dump output is going to be
> a file that's roughly 24GB? Maybe this takes hours or something being
> that it's that big and I'm presumably saving it to a RAID6 which is
> doing a lot more parity calcs all in single processor mode. Is there a
> way to estimate how long I'd have to wait to even get to a login
> prompt?

You won't get a login prompt unless you have some script set to
auto-save the dump file.  Typically you'll set things up to just
launch a root shell.  Saving a 24GB file shouldn't take more than a
few minutes with nothing else touching the hard drive.

Per the wiki you should be running:
kexec -p /[path-to-kernel] --append="root=[root-device] single irqpoll
maxcpus=1 reset_devices"

Note the single option in there - you're not going to get a login
prompt.  It will just dump you at a shell.  If you've booted into that
kernel then /proc/vmcore will contain a core file from the paniced
kernel.

If you just rebooted normally then I don't think /proc/vmcore will
even exist.  That was the problem I was having a while back - kexec
wasn't actually having any effect.  I have no idea if whatever broke
it has been fixed.

Rich