Gentoo Archives: gentoo-portage-dev

From: Brian Harring <ferringb@×××××.com>
To: gentoo-portage-dev@l.g.o
Subject: [gentoo-portage-dev] Max parallelization setting
Date: Tue, 10 Oct 2006 07:06:04
Message-Id: 20061010070457.GB6371@seldon
1 Resending to portage dev ml to solicit comments...
2
3 Thoughts welcome, since it's an issue that must be dealt with to
4 sanely do efficient parallelization (monkeying with make.conf and
5 hardcoding vals isn't much of a solution).
6
7 Zac- comments? You seem to have totally missed the original email ;)
8
9 -------------------------------------
10 Date: Sun, 1 Oct 2006 12:27:13 -0700
11 To: gentoo-dev@l.g.o
12 From: Brian Harring <ferringb@×××××.com>
13 Subject: Re: [gentoo-dev] Setting number of parallel builds for other build-systems than 'make'
14
15 On Sun, Oct 01, 2006 at 09:52:14AM -0700, Donnie Berkholz wrote:
16 > Tiziano Müller wrote:
17 > >-----BEGIN PGP SIGNED MESSAGE-----
18 > >Hash: SHA1
19 > >
20 > >Hi everyone
21 > >
22 > >It seems that setting the number of parallel builds using '-jN' does not
23 > >only work for make, but also for scons and bjam (and maybe others as well).
24 > >Since it isn't save to assume that '-jN' is the only option in MAKEOPTS,
25 > >some filtering is needed.
26 > >Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf and used
27 > >whenever one of those build-systems is being used. But we would probably
28 > >have to add a ...OPTS for every build-system.
29 > >
30 > >What do you think? Better ideas?
31 >
32 > I think adding it for each build system is probably a good idea,
33 > nobody's guaranteeing option-level compatibility with make. Optionally
34 > falling back to using the few valid MAKEOPTS might be a nice addition.
35
36 I might be daft (likely), but why not just introduce a var indicating
37 max parallelization instead? Tweak portage to push that setting into
38 MAKEOPTS="${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}".
39
40 Might sound daft, but -j is hardcoded parallelization; if you're
41 trying to run 3 build processes, the original var of -j isn't all that
42 useful, nor will the hardcoded -j# var set for 3 package build
43 processes be useful if you're doing a single build.
44
45 Depending on number of seperate package build processes possible, the
46 # of build processes allocated per build process needs to vary
47 (essentially).
48
49 Now... that's a bit ahead of what portage is doing atm, but folks
50 *will* parallelize portage building (see bug 147516 if in doubt), so
51 its not too far out.
52
53 Getting back to the actual topic, set this var, it's a raw int that a
54 scons/bjam eclass can use to easily set appropriate var with the
55 parallelization requested, thus unifying this particular knob; more
56 importantly, it gives them a way to do what they're after while
57 exposing a knob that the pkg manager can easily fiddle with for global
58 parallelization needs.
59
60 Only downside to it I see is that it requires mangling the pkg manager
61 to translate the parallelization setting into MAKEOPTS+=-j#; can't
62 really get around that however due to the fact MAKEOPTS is already
63 forced in installed configuration though.
64
65 Thoughts?
66 ~harring

Replies