Re: [gentoo-dev] "Distro Day (Measuring the benefits of the Gentoo approach)"

[Chris Gianelloni]

On Wed, 2003-08-13 at 18:49, William Kenworthy wrote:
> I'll stick my hand up and say I was the person who installed gentoo for
> this test.  For those who made the previous posts (mostly crap, and who
> dont seem to have read the article very well - though it could have been
> more informative), perhaps a few facts may help:
>
> 1. was fully bootstrapped and compiled as stage 1/2/3 on the machine -
> not a binary install

Great.  I read the article and found no mention of the USE flags
employed.  I think you should have honestly posted any information on
things you changed.

> 2. gentoo-sources 2.4.20 was used - Mandrake came with a newer kernel
> than gentoo's reccomended one (still does), debian was a dogs breakfast
> because stable is so old.  We actually tried to put the gentoo kernel on
> mandrake/debian when tracking down the ide cable prob, but got too hard
> - not the way some posts tried to imply)

Were preemption and low latency turned on?  Was the kernel compiled with
the >gcc31 selection for the CPU?  Better yet, why not post the .config
from the 3 kernels?
 
> 3. optimisations were EXACTLY as recommended by both the make.conf
> entries, which were supported by the cflags from the forum for this cpu:
> a 2G celery (P4 based core)  I am not sure now, but I believe I ran
> prelink as well (to match mandrake) - need to find and check the notes.
> 4. Gnumerics problems have been identified and come down to the
> particular version - is fixed in the upcoming stable release even before
> this was found, but the project was unaware that what they believed was
> a slightly slower mod in this version, could be so bad on particular
> data sets - i.e., 30 odd mins in 1.0.13, but is less that 30s on 1.0.19
> on my laptop

I hope you only used optimizations listed in the forums for the actual
version of GCC you're running.  From the sounds of it, you did not since
you used pentium3 and the pentium4 problems were fixed in the most
recent stable GCC.  You also should have definitely used a "default"
Gentoo install with no changes made.  The default profile setup would
have been used instead.  Your optimizations could have been researched
from GCC rather than taking the word of a bunch of "armchair compiler
experts" on the forums.  No offense meant to anyone, but you mention
below that you do much scientific work, yet followed a very poor
scientific model and research documentation for this article, which is
why it has been torn apart so adamantly.  Had you given out all of the
information, even if it were simply links to the files from within the
article, it would have given your article much more credibility.

> There seems to be quite a few myths about this test and people upset
> that months were not spent tuning gentoo and every effort made to
> cripple the competition! (one person even suggested the faulty ide cable
> should have been left in the debian box, as that was the way it was
> delivered!)  Read the article, and if you need extra information to
> reproduce it, email me or or the author (Indy).  It is reproducable - if
> you can obtain the same hardware - I would be very interested if someone
> has this and the time to really go into the why these results occurred
> in more detail than I had the chance to.

The same machine should have been used for the testing, rather than
three machines.  This alone is reason enough to discount your data. 
Three different machines WILL have three different levels of
performance.

> and why was this the result?  Daniel Robbins suggested on this list that
> gentoo-sources may be the problem, but tests on another machine (we had
> the trial machines for only a couple of days, all of which time was used
> to build gentoo right up until I ctrl-c'd the OO build so we could do
> the tests before handing the hardware back) showed that turning off
> pre-empt and low-latency had zero effect, but changing to an open-mosix
> kernel 2.4.20  was ~10% slower (no thread export).  It seemed to come

I agree with Daniel on some of this.  The default Gentoo kernel is not
the fastest out there, it is the most feature rich to meet the various
needs of our user base.  I do agree that this kernel should have been
used rather than any other.  Also, preempt and the low-latency are
interactivity increases, not raw performance increases.  Their
modifications are not easily quantifiable.  If you want to test them, I
suggest you look into ConTest
(http://members.optusnet.com.au/ckolivas/kernel/) which was designed for
testing this sort of thing.

> down to the fact we used -O3 instead of -O2 (think spider might have
> suggested this ?)- in effect over-optimised, and we didnt have a chance
> to correct. From my perspective, most of the "he should have used ...

No, you definitely "should have used" -O2 rather than -O3.  Also,
-fomit-frame-pointer and -mfpmath=sse would have given dramitic
improvements.  I'm not going to go into any other optimizations because
the rest are essentially very specific to the hardware/software being
used.  I think these are the only "sensible" extra defaults that can be
used on a machine with SSE.

> may actually have made performance even worse! And besides the time
> issue, these were supposedly the safe, reccomended flags so we went with
> them.  Please note that even Mandrake made only a slight gain on debian,
> so 386.586/686 does not make a lot of difference in real world tasks
> (the original aim of the tests) - the tests did tasks that particular

386, 586, 686 make little difference compared to 386, 586, pentium4,
which is how it should have been.

> people used linux for in their day-to-day work - no special tests, so no
> special bias.  Yes, I could choose tests that make gentoo shine, or
> debian, or windowsXP.  But I dont do those tests every day, whilst that
> spreadsheet was/is used as part of my normal work.  And its the same
> with the other tests.

I actually agreed with most of your tests.  You had a hard time being
very time constrained.  Honestly, were I in your position, I would not
have made this report at all unless I had a MUCH longer time to test
things.  You should look into the kinds of testing that many of the
hardware sites out there use.  They tend to take WEEKS on a single
article.  It doesn't take their full attention that entire time.  After
all, there's only so much interaction you need to do when running a
script which performs hundreds of actions and logs results to a file.

> So how many gentoo systems out there have every possible optimisation in
> the book, and are actually running slower than ideal?  This is a real

I use quite a few optimizations, which I benchmarked on my machine with
my application/data set and it is the fastest I was able to come up
with.  I have actually turned OFF quite a few of the optimizations
recommended by many of the "airmchair compiler experts" out there
because they either provided little to no improvement or actually
decreased performance.  I really don't care if something is 0.001%
faster if it takes 400% as long to compile.  Especially being a
developer and compiling quite a bit of stuff several times over.

> problem, and I will be interested in how the cflags projects around
> handle this, as most seem to aim at setting the maximum possible flags:
> not actually tune the system for the ones that work best/most stably.  A
> live benchmark test might be more appropriate.

I agree 100% here.

> Most posts on irc and lists have settled down to "he doesnt know what
> he's doing" (I do), or the tests were unfair to gentoo (they werent, but
> then the same criteria were met by all 3 systems, but with some question
> marks over debian because of its mix - some packages had to be compiled
> locally, not binary) - but the thrust of the article was not that gentoo
> was a dud, but that this was the result within the criteria and time we
> were given, not what we expected, so we need to find out why.  Also note
> that this was not intentionally a debian/mandrake/gentto distro test.

Not being able to tune Gentoo essentially means you did not participate
in the "Gentoo Approach" but rather kludged it together fairly untuned
and pitted against a tuned binary installation and debian.

> We may be getting a P4 hyperthreaded system to play with, but under
> different rules, where I get to do a bit of tuning first.  I have
> already built the core system on another machine using gcc-3.2.3,
> "-march=pentium4 -O3 -pipe -fomit-frame-pointer"  I note that the
> pentium4 warning still appears in make.conf, though I believe it no
> longer applies to this gcc.

It does not apply to the newest stable GCC, so you are correct.

> A while ago I emailed this list and asked for information on tests and
> settings for HT P4's, without a reply.  So again, has anyone done any
> tests on a HT P4 and is willing to support the flags they chose as being
> "the best"?  In particular, does -ffast-math give a measurable gain?

There is not much in the way of HT as it is looked at as a SMP machine
under Linux.  All you really do is enable SMP and make sure you use ACPI
in the kernel.  The default Gentoo kernel does not have many of the HT
scheduling changes which have gone into the making of the 2.6_test
kernels.  There are backports for these, but I would consider that going
a bit overboard, as hand-patching your kernel sources would yield better
results on all three systems and should be left alone.  After all,
you're wanting to test the results of the three systems, not of your
hand-made kernel.  If you were to decide to use another kernel, I would
say to use the latest vanilla kernel and possibly the latest 2.6_test
kernel on each distribution using the exact same .config to see how much
the kernel makes a difference in performance.  You should not use
-ffast-math in anything as a default, as it causes math errors which
should not be introduced into a stable system.

> Most of my machines have been built as scientific stations, so accuracy
> is more important than ultimate speed, so this is one I have never
> tested.  I am not interested in the -O9 -max-everything kiddies who have
> been so vocal, but reasoned choices.

The -O9 kiddies are the "armchair compiler experts" I spoke of earlier. 
They have zero real knowledge of compilers and optimizations at all, but
have "heard from a friend" or "read on a forum" about it so they think
they know it all.  I will gladly admit that I know little about
compilers, but I have taken the time to do actual benchmarks on my
system to test my various theories and have chosen what I feel to be the
best combinations for my own needs.

-- 
Chris Gianelloni
Developer, Gentoo Linux