[gentoo-amd64] Re: mce exception -- howto determine real problem?

[Duncan]

Alexander Puchmayr <alexander.puchmayr@×××××××.at> posted
200804271523.25677.alexander.puchmayr@×××××××.at, excerpted below, on 
Sun, 27 Apr 2008 15:23:25 +0200:

> After being abroad for a month I tried to boot my amd64 maschine and it
> crashes all the time, doing spontaneous reboots, hangs in the bios, and
> occasionally it tells me something about mce exception when loading
> hal-daemon, and immediately after that kernel panic, nothing works.
>
> The MCE-exception also says i should run mcelog --ascii, but how should
> I do that if the maschine is virtually dead in the kernel panic state?

If you can, get the MCE number, write it down or whatever.  Then get the 
program parsemce (written by Dave Jones, the kernel guy).  The sources 
are here:

http://www.kernel.org/pub/linux/kernel/people/davej/tools/parsemce.c

The google hits are here:

http://www.google.com/linux?lr=lang_en&hl=en&q=parsemce

The personal experience:

The (generic) memory I originally had in this machine was apparently not 
quite up to its on-module and sold-as rating (PC3200).  Unfortunately, 
the original BIOS and early updates didn't have memory timing control 
either; the board simply ran what the on-module rating said it was good 
for (assuming the board and CPUs, Opterons with onboard memory 
controllers, could run it as well), and that proved somewhat optimistic.

Memtest86 checked out clean, because it tests memory retention in various 
patterns, not memory timing, which it doesn't stress.  However, I'd get 
crashes often enough I knew something was wrong, and they'd happen more 
frequently under high activity, tho it didn't seem to be directly CPU 
related.  I did notice occasional bunzip2 errors claiming a corrupt 
archive, but trying again would usually work.  That seemed the only 
common user-space symptom I had -- the only other one was the MCEs and 
often (but not always) kernel panics as a result.  (I had the option to 
panic automatically on detected error turned off, setting it to try to 
continue if it could -- the only way I was able to compile things, 
sometimes.  I also learned how to restart an in-progress merge after a 
reboot...)

I suspected it might be memory but as I said memtest came up clean, and 
it could have been the memory controller on the CPU, or the on-chip 
cache, or...

Finally I found out that MCE stood for machine-check-exception, and that 
the number it reports could be checked against a chart AMD and Intel both 
provide to see what the machine check said was going wrong.  Parsemce was 
the way to do this at the time, tho it now seems there may be other 
methods, based on your comments.

Looking it up I found it was indeed memory bus errors, but it still could 
have been the board or something, not the memory itself.  Still, knowing 
it was generic memory, I figured it was.  The problem was that the 
socket-940 Opterons (which I have) take registered memory, which is 
nowhere near the commodity that unregistered non-ecc memory is, and I 
really couldn't afford it at the time.

Well, Tyan finally came out with a BIOS update that exposed the necessary 
memory knobs for me to tweak, and I found that limiting speed just one 
notch, from 400 MHz to 383 MHz (basically PC3000 speed instead of PC3200) 
solved the problem ENTIRELY.  At that speed I could even significantly 
tighten down the individual memory timings and the system was STILL 
stable as a rock -- it was the overall clockspeed that the memory just 
couldn't quite handle, at least on my board.  I suspect the reason it was 
generic memory was that it was just at tolerance at one end, and the 
board (which had originally only been rated to PC2700 speeds, so I was 
surprised it actually clocked the memory at PC3200 in the first place -- 
I figure generic memory, but PC3200, should be plenty good to run at 
PC2700 and it would have been, only by then the board BIOS had been 
updated to take up to PC3200 memory even tho it hadn't be updated to 
clock limit it if necessary) is probably near tolerance at the other end, 
so the memory simply wasn't stable at those speeds in this board, but 
would have worked just fine at lower speeds (as it did for me when I 
could finally do it with a BIOS update) or at rated speeds on many other 
boards.

So I ran the memory at PC3000 speeds but with many of the individual 
timings tightened for a year or so... at which point I could finally 
afford to upgrade memory to what I /really/ wanted all along -- the 8 
gigs memory I'm now running, tho it cost me ~US$1100 to do it.  It's 
Super Talent brand, perhaps not top of the line, but at least it's stable 
at rated speeds.

Anyway, first moral of the story is that the MCE reports were accurate, 
altho they didn't pin it down all the way, they certainly pointed the 
area, confirming my earlier suspicions, and that parsemce can help with 
the number to English description mapping.  Second is that just because 
memtest86 says it's fine doesn't necessarily mean the /timings/ are fine, 
as it tests memory cell reliability and doesn't stress timings.

Other morals would be to avoid expensive registered memory requirements 
if possible, avoid generic memory, avoid making assumptions about what a 
board will try to run the memory at despite what it's rated to run it at, 
and when shopping, confirm if possible that the BIOS contains memory 
speed tweaking knobs.  

... The reason I got Opterons in the first place was because I wanted a 
dual CPU system, and I wanted AMD64, and that's the way you got it at the 
time.  I've been very happy with the system in general, Tyan is quite 
good with Linux support on many of their boards including this one -- it 
was certified with several Linux distributions and they even had a 
preconfigured lm_sensors.conf available for download! =8^)  Additionally, 
when the dual-cores came out, all I had to do was upgrade the BIOS and I 
was ready to upgrade the CPUs (which I've now done, to dual dual-core 
Opteron 290s, at 2.8 GHz, top of the socket 940 line).  It sure would 
have been nice to have the memory tweaking stuff in the BIOS rather 
earlier, however.

Meanwhile, I plan on keeping this one for awhile (thus the money sunk in 
upgrades), but when I do upgrade again, I'd like to make it a Tyan once 
again, but 4 cores is really plenty and likely will still be plenty at 
that time, and they have that available in single socket desktops now so 
I won't need to spend the extra $$ on server class gear.

As for your problems, keep in mind that a going bad or overloaded 
powersupply can produce similar symptoms as well, or it can be unstable 
wall power.  Or overclocking altho it doesn't sound like you're into that.
But since you are getting MCEs, I definitely check what they say is wrong 
before anything else.  The machine is providing the information, why not 
use it?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@l.g.o mailing list