Re: [gentoo-user] is multi-core really worth it?

[David Haller]

Hello,

On Wed, 22 Nov 2017, Raffaele Belardi wrote:
>rebuilding system and world with gcc-7.2.0 on a 6-core AMD CPU I have
>the impression that most of the ebuilds limit parallel builds to 1, 2
>or 3 threads.

Most should build with "many" cores, but some might be limited, but ...

>I'm aware it is only an impression, I did not spend the
>night monitoring the process, but nevertheless every time I checked
>the load was very low.

I guess you're mostly observing stalls because the (single-thread)
overhead of emerge, configure (etc.), which can also be IO-bound... 
The actual builds probably goes so fast on your box, you hardly
notice unless they're rather big(gish).

Some numbers for comparison, and I generally build with just one
thread in the background on my 2 core "old" Athlon II X2 250:
dev-libs/boost took a good 45mins to build with one thread. Divide by
6 for your cores, divide again by about 2-3 for your newer cores and
you're in the 3:45min range for such a "biggie" as boost. Most other
stuff won't even peg your meter.

So, your emerge is mostly IO and "emerge"-threads bound. Solution:
adjust your build-threads[1], and then adjust your emerge jobs! See
'--jobs' in 'man emerge'. Can't find where to set a default in a
config-file for that ATM, but you could always alias/function/script
emerge to something else, e.g. put this in your ~/.bashrc:

====
eworld() {
    emerge --jobs 7 [..whatever_you_use_for_world..] @world
}
====

or use a "proper" script in ~/bin/ for that purpose.

You might want to "tune down" your "make" threads in make.conf though,
as your'll be building 7 packages in parallell ... Well, best try it
out, and have a guess at a sweet spot between emerge-jobs and
make-jobs. I'd hazard a guess of [2-4] emerge- and [3-5] make-jobs for
your six core should be nice.

Just experiment with 'emerge -j [1..7] ... @world'...

HTH,
-dnh

PS:[lines rebroken]

====
# type eworld 
eworld is a function
eworld () 
{ 
    emerge --verbose --pretend --update --changed-use \
        --newuse --deep --with-bdeps y "$@" @world;
    beep -r 1 -l 50
}
====

I usually call it as: eworld -t [--verbose-conflicts].

I then "pick and build" from the result (from the bottom up, due to
"--tree"). Even with ccache, it'd be rather stupid starting a
libreoffice, firefox or icedtea build that'll take hours when I have
to leave in 10min and won't keep the box running...

[1] I think that's good practice when doing multithreaded build on
    multicore: add (at least) one for IO. THREADS="$(( $N_CORES+1 ))"
    You can add that in your make.conf and feed $THREADS to
    MAKEOPTS="-j ${THREADS}" etc.

-- 
"I have always felt that violence was the last refuge of the incompetent,
 and emtpy threats the final sanctuary of the terminally inept."
    -- the Marquis de Carabas