Re: [gentoo-user] per package parallel build

[Mick]

On Friday, 25 October 2019 06:31:03 BST Raffaele BELARDI wrote:
> 8-core CPU:
>
> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
>
> MAKEOPTS="-j9 -l9"
>
> Works fine except when both Firefox and Thunderbird need update, in that
> case emerge typically tries to build them in parallel and one gets OOM
> killed due to insufficient swap space (1G swap, 16G RAM). I will increase
> the swap but I'd like to know: Is there a way to tell emerge to normally
> run 9 parallel jobs but limit to 1 when it is building one of the two
> monsters?
>
> Thanks,
>
> raffaele

Good advice has already been given, but here are some supporting thoughts 
based on my experience of trying to emerge mammoth builds on old systems with 
low RAM.

I can't recall how much memory each thread of a FF or TB compile chews up, but 
if you allow for up to 2G per thread, with -j9 your 16G RAM *will* be 
exhausted at some point.  Swapping will commence then in earnest and a I/O 
offloading of RAM into swap and back again as threads gasp for adequate memory 
will be sustained, while demand for more RAM than available is present. This 
creates a bottleneck which will prolong the emerge duration.  With enough swap 
you won't suffer any more OOMs, but you could find emerge times increase 
exponentially.  Therefore reducing the total number of jobs for these two 
packages will help in both cases.

Setting up an env directive as already advised in previous responses to 
restrict both packages to fewer jobs will work, but in some cases it can be 
suboptimal.  When only one package needs updating, why should it be emerged at 
a lesser speed than your system can support?  Even when both packages are 
emerged at the same time, the memory crunch may only take place for a limited 
period, at a time when both packages will be using maximum memory per thread.  
To make matters worse ebuilds and source code changes, compilers change and 
trying to optimise your emerge soon becomes a moving beast, so some 
experimentation will be necessary.

I think reducing your job/load count to 1 would be excessive.  Reducing it 
down to 5 ought to be adequate, but I suggest to increase your swap.  You 
could just use a swap file for this purpose.

PS.  In an ideal AI world, portage would know how much memory is necessary for 
a given package and would auto-adjust the number of jobs to minimise swapping 
given any amount of RAM.  In an even more ideal world, it would be able to do 
this in real time.  :-)

-- 
Regards,

Mick