Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Kai Krakow]

Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@g.o>:
>
> >>>>> On Wed, 05 Jan 2022, Florian Schmaus wrote:
>
> > On 05/01/2022 09.28, Ulrich Mueller wrote:
> >>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
> >>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> >>> amount of RAM available (uses amount declared as needed
> >>> in the ebuild). Typically should be ~2GB per job.
> >> Where does this number 2 GB come from? The amount of RAM strongly
> >> depends on the programming language and other factors, so I don't
> >> believe that there's one number that can be used for everything.
>
> > Surely not, but 2 GiB seems like a good start. I guess it could become
> > an eclass variable that ebuilds could modify later (if the need
> > emerges).
>
> We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
> to introduce another one that is counting memory per job? I doubt that
> would be possible, because the amount of memory greatly varies within a
> single build. For example, it is different for compiler vs linker vs
> building the documentation.
>
> > It appears to me that the motivation for this change is to prevent
> > users from running into OOM situations when emerging a package. And I
> > believe that many users, especially novices, are not directly able to
> > determine that portage failed due to OOM, simply because there is no
> > direct hint in the build log. Hence opt-out appears to be sensible to
> > me here.
>
> That applies to all parallel builds though, not only to ebuilds
> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
> telling the user that the --jobs setting in their make.conf is wrong,
> in the first place.

Well, I'm using a safe combination of jobs and load-average, maybe the
documentation should be tweaked instead.

I'm using

MAkEOPTS="--jobs=12 --load-average=8"

on a 8x2+4 core CPU. I choose less then the available 20 cores because
I'm also running multiple ebuilds in parallel to make better use of
resources while most packages run their configure phase (which for
small packages always takes longer than the build itself):

EMERGE_DEFAULT_OPTS="--jobs=12 --load-average 10"

The "--jobs" parameter is mostly a safe-guard against "make" or
"emerge" overshooting the system resources which would happen if
running unconstrained without "--load-average". The latter parameter
OTOH tunes the parallel building processes automatically to the
available resources. If the system starves of memory, thus starts to
swap, load will increase, and make will reduce the jobs. It works
pretty well.

I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
won't starve emerge from running configure phases of parallel ebuilds.

Regards,
Kai