Gentoo Archives: gentoo-dev

From: David Seifert <soap@g.o>
To: Ulrich Mueller <ulm@g.o>, Sam James <sam@g.o>
Cc: gentoo-dev@l.g.o, qa@g.o
Subject: Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage
Date: Wed, 05 Jan 2022 20:34:36
Message-Id: 67c9c040ac2e12c53832c7df8b559d4da2645a1c.camel@gentoo.org
In Reply to: Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage by Ulrich Mueller
1 On Wed, 2022-01-05 at 09:28 +0100, Ulrich Mueller wrote:
2 > > > > > > On Tue, 04 Jan 2022, Sam James wrote:
3 >
4 > > Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
5 > > amount of RAM available (uses amount declared as needed
6 > > in the ebuild). Typically should be ~2GB per job.
7 >
8 > Where does this number 2 GB come from? The amount of RAM strongly
9 > depends on the programming language and other factors, so I don't
10 > believe that there's one number that can be used for everything.
11 > (If only considering C and C++ 2 GB seems to be excessive, i.e. it
12 > will limit parallel build more than necessary. When linking, the
13 > number
14 > may be _much_ larger.)
15
16 https://wiki.gentoo.org/wiki/MAKEOPTS
17 The rule of thumb is MAKEOPTS jobs should be the smaller number of: the
18 size of RAM divided by 2GB or the number of threads the CPU has.
19
20 You're free to propose a better scheme, but try compiling some template-
21 heavy C++ code with 2GB/thread and -O3 and then WONTFIXing all the bug
22 reports. OTOH, if you emake -j1, people complain about stuff being
23 slower than glacial (https://bugs.gentoo.org/744088). Unless I see a
24 better solution, sam's solution is the best way to avoid these annoying,
25 rare, edge-case pitfalls.
26
27 > Also not sure if I understand the arithmetic. Shouldn't it use
28 > CHECKREQS_MEMORY as the basis of the calculation?
29 >
30 > > +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
31 > > +# @USER_VARIABLE
32 > > +# @DESCRIPTION:
33 > > +# Allow packages to reduce the number of multiprocessing (e.g.
34 > > make, ninja) jobs
35 > > +# to lower memory usage.
36 > > +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
37 >
38 > If anything, the feature should be opt-in rather than opt-out.
39 >
40 > Ulrich