Re: [gentoo-dev] "Distro Day (Measuring the benefits of the Gentoo approach)"

[William Kenworthy]

A good reply - answer point by point

On Thu, 2003-08-14 at 20:30, Chris Gianelloni wrote:
> On Wed, 2003-08-13 at 18:49, William Kenworthy wrote:

> Great.  I read the article and found no mention of the USE flags
> employed.  I think you should have honestly posted any information on
> things you changed.
>
No longer available, but I dont think its relevant to the tests we did.
Interesting for some maybe, but not a performance issue.
This was originally set up as a simple test of 3 relatively new to the
distro users installing for the first time.  i.e., if I moved from RH to
gentoo, where would I start: read through make.conf which is where the
docs say to go and set the system up that way and so on.  Again, both
time and this approach means extensive tuning, and changes that are not
part of the initial install were not done.  USE flags are to turn
options in packages off and on, not set up performance (well at least
directly), so I just used the minimum to get the install working for the
tests.  Superficially, this should be faster as it wouldn't install too
much extra

> > 2. gentoo-sources 2.4.20 was used - Mandrake came with a newer kernel
> > than gentoo's reccomended one (still does), debian was a dogs breakfast
> > because stable is so old.  We actually tried to put the gentoo kernel on
> > mandrake/debian when tracking down the ide cable prob, but got too hard
> > - not the way some posts tried to imply)
>
> Were preemption and low latency turned on?  Was the kernel compiled with
> the >gcc31 selection for the CPU?  Better yet, why not post the .config
> from the 3 kernels?
They were on, configs no longer available other than some notes I took
at the time.  Again, not really relevant in the original context of the
test - next time as it seems people are interested.
>
> > 3. optimisations were EXACTLY as recommended by both the make.conf
> > entries, which were supported by the cflags from the forum for this cpu:
> > a 2G celery (P4 based core)  I am not sure now, but I believe I ran
> > prelink as well (to match mandrake) - need to find and check the notes.
> > 4. Gnumerics problems have been identified and come down to the
> > particular version - is fixed in the upcoming stable release even before
> > this was found, but the project was unaware that what they believed was
> > a slightly slower mod in this version, could be so bad on particular
> > data sets - i.e., 30 odd mins in 1.0.13, but is less that 30s on 1.0.19
> > on my laptop
>
> I hope you only used optimizations listed in the forums for the actual
> version of GCC you're running.  From the sounds of it, you did not since
> you used pentium3 and the pentium4 problems were fixed in the most
> recent stable GCC.
Not fixed in the version at the time of the tests.  Also, in my current
make.conf, there is still that huge all capital warning saying dont use
pentium4 - nothing about any safe gcc version for the P4.

> You also should have definitely used a "default"
> Gentoo install with no changes made.  The default profile setup would
> have been used instead.  Your optimizations could have been researched
> from GCC rather than taking the word of a bunch of "armchair compiler
> experts" on the forums.  No offense meant to anyone, but you mention
> below that you do much scientific work, yet followed a very poor
> scientific model and research documentation for this article, which is
> why it has been torn apart so adamantly.  Had you given out all of the
> information, even if it were simply links to the files from within the
> article, it would have given your article much more credibility.
>
I actually did quite a lot of looking at this.  The flags used are a bit
different to what I use on my own  pentiums and athlons (I use
-fomit-frame-pointer for instance) - we limited ourselves to what the
user would see reading make.conf as suggested in the documents.
I would make the point that this is not an exhaustive no holds barred
competition -  Criteria could be described as "This was originally set
up as a simple test of 3 relatively new to the distro users installing
for the first time.  i.e., if I moved from RH to gentoo, where would I
start: read through make.conf which is where the docs say to go and set
the system up that way and so on." 

I agree it is not a "scientific test", - it was not meant to be, but a
simple this one looks faster than that one when I do the same work I do
every day - not a special performance suite.  We are not trying to be a
microsoft and come up with an unreal figure to bolster our sales.

> > There seems to be quite a few myths about this test and people upset
> > that months were not spent tuning gentoo and every effort made to
> > cripple the competition! (one person even suggested the faulty ide cable
> > should have been left in the debian box, as that was the way it was
> > delivered!)  Read the article, and if you need extra information to
> > reproduce it, email me or or the author (Indy).  It is reproducable - if
> > you can obtain the same hardware - I would be very interested if someone
> > has this and the time to really go into the why these results occurred
> > in more detail than I had the chance to.
>
> The same machine should have been used for the testing, rather than
> three machines.  This alone is reason enough to discount your data.
> Three different machines WILL have three different levels of
> performance.
>
A couple have mentioned this, but from personnel experiance, I can say
yes, you may see some small variations (other than actually faulty
hardware), but they should be fairly close if using the same software. 
We were not using the same software, so I would expect that to submerge
any variations in hardware

> > and why was this the result?  Daniel Robbins suggested on this list that
> > gentoo-sources may be the problem, but tests on another machine (we had
> > the trial machines for only a couple of days, all of which time was used
> > to build gentoo right up until I ctrl-c'd the OO build so we could do
> > the tests before handing the hardware back) showed that turning off
> > pre-empt and low-latency had zero effect, but changing to an open-mosix
> > kernel 2.4.20  was ~10% slower (no thread export).  It seemed to come
>
> I agree with Daniel on some of this.  The default Gentoo kernel is not
> the fastest out there, it is the most feature rich to meet the various
> needs of our user base.  I do agree that this kernel should have been
> used rather than any other.  Also, preempt and the low-latency are
> interactivity increases, not raw performance increases.  Their
> modifications are not easily quantifiable.  If you want to test them, I
> suggest you look into ConTest
> (http://members.optusnet.com.au/ckolivas/kernel/) which was designed for
> testing this sort of thing.
>
I dont use conftest in my day to day work, but I often use gnumeric,
gimp and OO - the intention was not to test for pure numbers but for me
at least, If I wait for 30 mins to load my spreadsheet under Gentoo, how
much longer will poor debianites have to wait ... and the answer was not
as long! - so the focus shifted to whats gone wrong with gentoo.

connftest will be handy when tuning for the next one (depending on
time), but I doubt will be used for actual benchmarks as it is
irrelevant to everyday use.
> > down to the fact we used -O3 instead of -O2 (think spider might have
> > suggested this ?)- in effect over-optimised, and we didnt have a chance
> > to correct. From my perspective, most of the "he should have used ...
>
> No, you definitely "should have used" -O2 rather than -O3.  Also,
> -fomit-frame-pointer and -mfpmath=sse would have given dramitic
> improvements.  I'm not going to go into any other optimizations because
> the rest are essentially very specific to the hardware/software being
> used.  I think these are the only "sensible" extra defaults that can be
> used on a machine with SSE.
Couldn't use them as they were not listed as reccomended in make.conf.

Keep in mind that pentium3 implies extra flags - the following is from
an email on the gcc list:
-march=pentium3     -mcpu=i686 -msse
-march=pentium4     -mcpu=pentium4 -msse2

so sse is implied by pentium3, and sse2 is where the invalid code 
was being generated, hence the warning to stay away from pentium4!

>
> > may actually have made performance even worse! And besides the time
> > issue, these were supposedly the safe, reccomended flags so we went with
> > them.  Please note that even Mandrake made only a slight gain on debian,
> > so 386.586/686 does not make a lot of difference in real world tasks
> > (the original aim of the tests) - the tests did tasks that particular
>
> 386, 586, 686 make little difference compared to 386, 586, pentium4,
> which is how it should have been.
>
> > people used linux for in their day-to-day work - no special tests, so no
> > special bias.  Yes, I could choose tests that make gentoo shine, or
> > debian, or windowsXP.  But I dont do those tests every day, whilst that
> > spreadsheet was/is used as part of my normal work.  And its the same
> > with the other tests.
>
> I actually agreed with most of your tests.  You had a hard time being
> very time constrained.  Honestly, were I in your position, I would not
> have made this report at all unless I had a MUCH longer time to test
> things.  You should look into the kinds of testing that many of the
> hardware sites out there use.  They tend to take WEEKS on a single
> article.  It doesn't take their full attention that entire time.  After
> all, there's only so much interaction you need to do when running a
> script which performs hundreds of actions and logs results to a file.
>
A bit late, as you can only find out this after you start the test - not
cricket to say whoops, I'm not winning, go ahead without me! : so gentoo
didnt perform up to my hopes.  At least we now have a discussion as to
why, how to improve it, and perhaps reel in some of the hype.

> > So how many gentoo systems out there have every possible optimisation in
> > the book, and are actually running slower than ideal?  This is a real
>
> I use quite a few optimizations, which I benchmarked on my machine with
> my application/data set and it is the fastest I was able to come up
> with.  I have actually turned OFF quite a few of the optimizations
> recommended by many of the "airmchair compiler experts" out there
> because they either provided little to no improvement or actually
> decreased performance.  I really don't care if something is 0.001%
> faster if it takes 400% as long to compile.  Especially being a
> developer and compiling quite a bit of stuff several times over.
>
> > problem, and I will be interested in how the cflags projects around
> > handle this, as most seem to aim at setting the maximum possible flags:
> > not actually tune the system for the ones that work best/most stably.  A
> > live benchmark test might be more appropriate.
>
> I agree 100% here.
Thanks, hopefully this article has kicked this idea along a bit.
>
> > Most posts on irc and lists have settled down to "he doesnt know what
> > he's doing" (I do), or the tests were unfair to gentoo (they werent, but
> > then the same criteria were met by all 3 systems, but with some question
> > marks over debian because of its mix - some packages had to be compiled
> > locally, not binary) - but the thrust of the article was not that gentoo
> > was a dud, but that this was the result within the criteria and time we
> > were given, not what we expected, so we need to find out why.  Also note
> > that this was not intentionally a debian/mandrake/gentto distro test.
>
> Not being able to tune Gentoo essentially means you did not participate
> in the "Gentoo Approach" but rather kludged it together fairly untuned
> and pitted against a tuned binary installation and debian.
>
> > We may be getting a P4 hyperthreaded system to play with, but under
> > different rules, where I get to do a bit of tuning first.  I have
> > already built the core system on another machine using gcc-3.2.3,
> > "-march=pentium4 -O3 -pipe -fomit-frame-pointer"  I note that the
> > pentium4 warning still appears in make.conf, though I believe it no
> > longer applies to this gcc.
>
> It does not apply to the newest stable GCC, so you are correct.
>
> > A while ago I emailed this list and asked for information on tests and
> > settings for HT P4's, without a reply.  So again, has anyone done any
> > tests on a HT P4 and is willing to support the flags they chose as being
> > "the best"?  In particular, does -ffast-math give a measurable gain?
>
> There is not much in the way of HT as it is looked at as a SMP machine
> under Linux.  All you really do is enable SMP and make sure you use ACPI
> in the kernel.  The default Gentoo kernel does not have many of the HT
> scheduling changes which have gone into the making of the 2.6_test
> kernels.  There are backports for these, but I would consider that going
> a bit overboard, as hand-patching your kernel sources would yield better
> results on all three systems and should be left alone.  After all,
> you're wanting to test the results of the three systems, not of your
> hand-made kernel.  If you were to decide to use another kernel, I would
> say to use the latest vanilla kernel and possibly the latest 2.6_test
> kernel on each distribution using the exact same .config to see how much
> the kernel makes a difference in performance.  You should not use
> -ffast-math in anything as a default, as it causes math errors which
> should not be introduced into a stable system.
>
> > Most of my machines have been built as scientific stations, so accuracy
> > is more important than ultimate speed, so this is one I have never
> > tested.  I am not interested in the -O9 -max-everything kiddies who have
> > been so vocal, but reasoned choices.
>
> The -O9 kiddies are the "armchair compiler experts" I spoke of earlier.
> They have zero real knowledge of compilers and optimizations at all, but
> have "heard from a friend" or "read on a forum" about it so they think
> they know it all.  I will gladly admit that I know little about
> compilers, but I have taken the time to do actual benchmarks on my
> system to test my various theories and have chosen what I feel to be the
> best combinations for my own needs.
Wish more would do as you have done - I dont think too many follow this
approach.  Most seem to just do the reccomended, or go for the max.  I
did a few key apps, made my choices and have stayed with them.  If a
really useful cflags project gets up, it would be nice to run it
regularly and perhaps find that the new gcc just emerged gives a
measured 5% speed up if you use the new -supercharger flag!

I accept there can be a large number of criticisms made of the tests,
but most can be countered because of the criteria we set for ourselves.

Thanks for the time you spent on this

-- 
William Kenworthy <billk@×××××××××.au>


--
gentoo-dev@g.o mailing list