Gentoo Archives: gentoo-user

From: Neil Bothwick <neil@××××.net>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] per package parallel build
Date: Sat, 26 Oct 2019 13:05:03
Message-Id: B799CDC8-316E-49DC-9249-1008C362D6B6@stfw.net
In Reply to: Re: [gentoo-user] per package parallel build by John Blinka
1 On 26 October 2019 12:16:37 BST, John Blinka <john.blinka@×××××.com> wrote:
2 >>
3 >> I agree that it would be nice if emerge could do that automatically,
4 >> although I have no clue how to do that or even if it can be done at
5 >> all. Back when I had less memory, I could let FF, LOo or another
6 >> package run at full speed but only if it was only one of those
7 >packages
8 >> at a time. Thing is, on occasion two or more of those updates would
9 >hit
10 >> and due to the long compile times, end up compiling at the same time.
11 >> Do you think there is a way for the devs to set up a method to tell
12 >> emerge not to emerge certain packages at the same time? In other
13 >words,
14 >> if Firefox is emerging, LOo is held until it is done or vice versa.
15 >> Maybe even have it so others can be listed. The list of large
16 >packages
17 >> are likely small but they can have a huge impact on systems with less
18 >> memory.
19 >>
20 >> You think that a feature worth asking the devs about? Maybe they can
21 >> figure out a way to implement that??
22 >
23 >
24 >There already is a mechanism you can use, but it’s not the automatic
25 >type
26 >that you (and, admittedly I) would like.
27 >
28 >I have 3 old 2 core machines, and I use distcc heavily to reduce emerge
29 >times. The “fastest” (not really) and best equipped has 16 gb memory.
30 >I
31 >do updates on this machine (with distcc help from the others) and
32 >distribute packages to the rest. After a lot of experimenting, I find
33 >that
34 >MAKEOPTS=“-j13 -l5” works the best on this fastest machine. That
35 >setting
36 >allows it to attempt a workload that it alone doesn’t have the
37 >resources to
38 >accomplish, but successfully distributes to the other machines. I use
39 >firefox, chromium, and libreoffice. Occasionally portage wants to
40 >upgrade
41 >more than one of these at a time, which I discover by running emerge
42 >—pretend. On those occasions, I’ve learned that I run out of
43 >resources
44 >and builds fail. So I just temporarily mask all but one of those
45 >updates,
46 >do the upgrade, unmask one of the masked updates, do another upgrade,
47 >and
48 >so on. Works well for me. No builds crash, essentially no swap gets
49 >used,
50 >and I have substantially accelerated compile and ebuild times.
51 >
52 >The tools exist to do what you want to do. If you were so inclined,
53 >you
54 >might even contemplate writing a script to automate what I just
55 >described.
56 >
57 >John Blinka
58
59 There's no need to mess around adding and removing masks, just use the - - exclude option.
60 --
61 Sent from my Android device with K-9 Mail. Please excuse my brevity.

Replies

Subject Author
Re: [gentoo-user] per package parallel build John Blinka <john.blinka@×××××.com>
[gentoo-user] Re: per package parallel build Ian Zimmerman <itz@××××××××××××.org>