Re: [gentoo-user] per package parallel build

[Dale]

Mick wrote:
> On Friday, 25 October 2019 06:31:03 BST Raffaele BELARDI wrote:
>> 8-core CPU:
>>
>> EMERGE_DEFAULT_OPTS="--quiet-build --keep-going --jobs 9 --load-average 9"
>>
>> MAKEOPTS="-j9 -l9"
>>
>> Works fine except when both Firefox and Thunderbird need update, in that
>> case emerge typically tries to build them in parallel and one gets OOM
>> killed due to insufficient swap space (1G swap, 16G RAM). I will increase
>> the swap but I'd like to know: Is there a way to tell emerge to normally
>> run 9 parallel jobs but limit to 1 when it is building one of the two
>> monsters?
>>
>> Thanks,
>>
>> raffaele
> Good advice has already been given, but here are some supporting thoughts
> based on my experience of trying to emerge mammoth builds on old systems with
> low RAM.
>
> I can't recall how much memory each thread of a FF or TB compile chews up, but
> if you allow for up to 2G per thread, with -j9 your 16G RAM *will* be
> exhausted at some point.  Swapping will commence then in earnest and a I/O
> offloading of RAM into swap and back again as threads gasp for adequate memory
> will be sustained, while demand for more RAM than available is present. This
> creates a bottleneck which will prolong the emerge duration.  With enough swap
> you won't suffer any more OOMs, but you could find emerge times increase
> exponentially.  Therefore reducing the total number of jobs for these two
> packages will help in both cases.
>
> Setting up an env directive as already advised in previous responses to
> restrict both packages to fewer jobs will work, but in some cases it can be
> suboptimal.  When only one package needs updating, why should it be emerged at
> a lesser speed than your system can support?  Even when both packages are
> emerged at the same time, the memory crunch may only take place for a limited
> period, at a time when both packages will be using maximum memory per thread.
> To make matters worse ebuilds and source code changes, compilers change and
> trying to optimise your emerge soon becomes a moving beast, so some
> experimentation will be necessary.
>
> I think reducing your job/load count to 1 would be excessive.  Reducing it
> down to 5 ought to be adequate, but I suggest to increase your swap.  You
> could just use a swap file for this purpose.
>
> PS.  In an ideal AI world, portage would know how much memory is necessary for
> a given package and would auto-adjust the number of jobs to minimise swapping
> given any amount of RAM.  In an even more ideal world, it would be able to do
> this in real time.  :-)
>


I agree that it would be nice if emerge could do that automatically,
although I have no clue how to do that or even if it can be done at
all.  Back when I had less memory, I could let FF, LOo or another
package run at full speed but only if it was only one of those packages
at a time.  Thing is, on occasion two or more of those updates would hit
and due to the long compile times, end up compiling at the same time. 
Do you think there is a way for the devs to set up a method to tell
emerge not to emerge certain packages at the same time?  In other words,
if Firefox is emerging, LOo is held until it is done or vice versa. 
Maybe even have it so others can be listed.  The list of large packages
are likely small but they can have a huge impact on systems with less
memory. 

You think that a feature worth asking the devs about?  Maybe they can
figure out a way to implement that??

Dale

:-)  :-)