Re: [gentoo-dev] Major MCE problem with SMP on Gentoo kernels

[Kevin]

Thanks for your reply, Josh.

On Tuesday 11 May 2004 22:42, Josh Glover wrote:
> Quoth Kevin (Tue 2004-05-11 02:07:58PM -0400):
> > The machine is a Dell PowerEdge1600SC with a PERC-3/SC SCSI RAID
> > controller (using AMI megaraid2 driver) and a LSI Logic Corp
> > controller (using Fusion MPT base driver) for the SCSI DAT and with
> > dual 2.4GHz Xeon processors, each having a 512KB L2 Cache.
>
> Running Gentoo with a 2.6.5 SMP kernel on a Dell PowerEdge 400SC:
> : jmglov@jglover; uname -a
>
> Linux jglover 2.6.5-gentoo-r1 #1 SMP Fri Apr 30 17:37:18 EDT 2004 i686
> Intel(R) Pentium(R) 4 CPU 2.40GHz GenuineIntel GNU/Linux
>
> : jmglov@jglover; cat /proc/cpuinfo
>
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 15
> model           : 2
> model name      : Intel(R) Pentium(R) 4 CPU 2.40GHz
> stepping        : 9
> cpu MHz         : 2395.027
> cache size      : 512 KB
> physical id     : 0
> siblings        : 2
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 2
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
> bogomips        : 4718.59
>
> processor       : 1
> vendor_id       : GenuineIntel
> cpu family      : 15
> model           : 2
> model name      : Intel(R) Pentium(R) 4 CPU 2.40GHz
> stepping        : 9
> cpu MHz         : 2395.027
> cache size      : 512 KB
> physical id     : 0
> siblings        : 2
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 2
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
> bogomips        : 4767.74

Have you turned off hyperthreading?  Why is it that only two CPUs show up?  
It looks like (flags include ht) the CPUs support hyperthreading... or am 
I way off base in drawing that conclusion here?

>
> > I'm a recent Gentoo convert.  I think it's an excellent improvement
> > on the traditional Linux distros, and I'd really like to use it on my
> > server, but as long as this problem with SMP is present, I just
> > can't.
>
> I really do not think it is a Gentoo issue. I have run Gentoo on quite
> a few SMP boxen over the past several years, and never had problems
> like you describe. Sounds like a hardware issue to me, unless you are
> using (or were using) some really bogus CFLAGS.

Well, last night I did the most exhaustive set of tests available on the 
Dell Utility Partition and found zero errors.  I did 16 loops of the MATS 
test and 15 of TestY, TestA, and ECC coupling.  The memory tests ran for 
about 8 hours.  Again, zero errors.

> Quoth Kevin (Tue, 11 May 2004 17:31:32 -0400):
> > Honestly, I'm thinking that I may have somehow built some software
> > (during the stage 1 installation process) that is causing these
> > problems, but I followed the Gentoo Handbook for doing a stage 1
> > installation pretty rigidly, so I'm not sure what I might have done
> > to cause that.
>
> Why did you do a Stage 1, just out of curiousity.

Some silly sense of pride or accomplishment that every bit of code running 
on the box was actually compiled on it.  I really like that Gentoo makes 
that option a practical one.  Silly, I know...

> I recommend doing at
> least one Stage 1 install for newcomers to Gentoo, just for educational
> purposes, but after that, go Stage 3 and use as many binary packages as
> you can! There exists a Stage 3 tarball for your architecture--the
> Pentium4 one, so why not use that, just to make sure your base system
> is solid?

That probably is a good thing for me to try next, and I really would like 
to be able to use all the advantages of Gentoo on this server.  I'm fed 
up with rpm hell and the traditional distros drawbacks.

>
> > When I did the bootstrap.sh and emerge system, I was
> > running the kernel that I booted from the boot CD (2004.0 I think,
> > and probably even the smp kernel that was on that CD---IIRC, the
> > 2004.1 boot CD has some problems that prevent the use of the smp
> > kernel on that CD).
>
> I don't remember that, but I cannot say for certain that I have tried
> the 2004.1 universal x86 CD with the SMP kernel.

It may have changed (I did this just a few days after the 2004.1 CD came 
out, and I think it was the minimal one).

>
> > Are there some compiler flags or other configurable settings that, if
> > set to certain values during the bootstrap.sh or emerge system steps,
> > could end up generating software (perhaps when I built my own gcc?)
> > that would cause these MCEs to be thrown?
>
> I dunno, why don't you post your CFLAGS and MAKEOPTS from your
> make.conf here?

Did already :)

Here they are again:
On Tuesday 11 May 2004 14:07, Kevin wrote:
> Gentoo emerge info output:
> =======
> System uname: 2.4.25-gentoo-r2 i686 Intel(R) Xeon(TM) CPU 2.40GHz
> Gentoo Base System version 1.4.9
> distcc 2.13 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632)
> [enabled]
> ccache version 2.3 [enabled]
> Autoconf: sys-devel/autoconf-2.58-r1
> Automake: sys-devel/automake-1.8.3
> ACCEPT_KEYWORDS="x86"
> AUTOCLEAN="yes"
> CFLAGS="-O3 -march=pentium4 -pipe -fomit-frame-pointer"
> CHOST="i686-pc-linux-gnu"
> COMPILER="gcc3"
> CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
> /usr/kde/3.2/share/config /usr/kde/3/share/config
> /usr/lib/mozilla/defaults/pref /usr/share/config
> /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/
> /usr/share/texmf/tex/generic/config/
> /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/
> /var/qmail/control" CONFIG_PROTECT_MASK="/etc/afs/C /etc/afs/afsws
> /etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O3 -march=pentium4
> -pipe -fomit-frame-pointer"
> DISTDIR="/usr/portage/distfiles"
> FEATURES="autoaddcvs ccache distcc sandbox"
> GENTOO_MIRRORS="http://128.213.5.34/gentoo/
> http://mirror.datapipe.net/gentoo
> ftp://mirrors.sec.informatik.tu-darmstadt.de/gentoo/
> http://gentoo.eliteitminds.com"
> MAKEOPTS="-j3"
> PKGDIR="/usr/portage/packages"
> PORTAGE_TMPDIR="/var/tmp"
> PORTDIR="/usr/portage"
> PORTDIR_OVERLAY=""
> SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
> USE="X Xaw3d acl acpi afs alsa apache2 apm arts avi berkdb bonobo caps
> crypt
> cups doc emacs emacs-w3 encode esd ethereal evo firebird flac
> foomaticdb gdbm
> gif gnome gpm gstreamer gtk gtk2 gtkhtml guile hardened icq imagemagick
> imap
> imlib innodb ipv6 jabber jack java jikes jpeg kde kerberos krb4 ldap
> libg++
> libwww mad mcal mikmod motif mozilla mpeg mysql ncurses nls odbc
> oggvorbis opengl oss pam pda pdflib perl plotutils png ppds prelude
> python qt quicktime
> readline ruby samba sasl sdl slang slp spell sse ssl svga tcltk tcpd
> tetex tiff
> truetype unicode usb vhosts x86 xinerama xml2 xmms xv zeo zlib"
> =======
>

Thanks again for the thoughtful reply, Josh.

-- 
-Kevin

--
gentoo-dev@g.o mailing list