Re: [gentoo-user] Jobs and load-average

[Michael]

On Wednesday, 15 February 2023 11:31:49 GMT Peter Böhm wrote:
> Am Mittwoch, 15. Februar 2023, 10:56:22 CET schrieb Peter Humphrey:
> > Hello list,
> >
> > Not long ago I read that we should allow 2GB RAM for every emerge job -
> > that is, we should divide our RAM size by 2 to get the maximum number of
> > simultaneous jobs. I'm trying to get that right, but I'm not there yet.
> >
> > I have these entries in make.conf:
> > EMERGE_DEFAULT_OPTS="--jobs=16 --load-average=32 --autounmask=n --quiet-
> > unmerge-warn --ke>

The above determine how may ebuilds will be emerged in parallel.  If you are 
rebuilding your whole system with hundreds of packages stacking up to be 
emerged, then having as high as 16 packages being emerged in parallel could be 
advantageous.


> > MAKEOPTS="-j16"

This determines how many MAKE jobs will run in parallel in any one emerge.  
Large packages like chromium will benefit from maximising the number of jobs 
here, as long as you have enough RAM.

Given you have 24 threads and your RAM is 64GB, you should be able to ratchet 
this up to -j24, but not if you specify a high EMERGE_DEFAULT_OPTS at the same 
time, or if the compiler is eating up more than 2G per process.


> > Today, though, I saw load averages going up to 72. Can anyone suggest
> > better values to suit my 24 threads and 64GB RAM?

Since you have specified up to 16 parallel emerges and each one could run up 
to 16 MAKE jobs, you can understand why you would soon find loads escalating 
and your machine becoming unresponsive.

You should consider what is more important for you, emerging as many packages 
in parallel as possible, or emerging any one large package as fast as 
possible.

Two extreme examples would be setting EMERGE_DEFAULT_OPTS at "--jobs 24", with 
MAKEOPTS at "-j1", or conversely setting EMERGE_DEFAULT_OPTS at "--jobs 1", 
with MAKEOPTS at "-j24".

On my old and slow laptop with only 4 threads and 16G of RAM my priority is to 
finish large packages faster.  I leave EMERGE_DEFAULT_OPTS unset, while 
specifying MAKEOPTS="-j5 -l4.8".  This uses a job number I determined by trial 
and error, building ffmpeg repeatedly by progressively increasing the number 
for -j from 1 to 12.  The two faster times were achieved with -j5 and -j10, 
which aligns with the old myth of using CPU+1.

Regarding RAM being used being ~2G per MAKE job, this fluctuates with 
successive compiler versions.  I have seen up to 3.4G of RAM per process, 
while emerging chromium.  For such huge packages which cause excessive 
swapping, unresponsiveness and thrashing of disk, I limit the MAKEOPTS to 3 in 
package.env.


> Maybe you are interested in this wiki article:
>
> https://wiki.gentoo.org/wiki/User:Pietinger/Tutorials/Optimize_compile_times
>
> Regards,
> Peter

I'd start by reading the suggestions in this article first, which is a good 
introduction to the concepts involved:

https://wiki.gentoo.org/wiki/MAKEOPTS