Gentoo Archives: gentoo-user

From: John Blinka <john.blinka@×××××.com>
To: gentoo-user@l.g.o
Subject: Re: [gentoo-user] per package parallel build
Date: Sat, 26 Oct 2019 11:16:58
Message-Id: CAC_tCmpZo1vDHz7iLu8+Tf5Mdazoqwd=HqY4j28SH-D_7BG3UQ@mail.gmail.com
In Reply to: Re: [gentoo-user] per package parallel build by Dale
1 >
2 > I agree that it would be nice if emerge could do that automatically,
3 > although I have no clue how to do that or even if it can be done at
4 > all. Back when I had less memory, I could let FF, LOo or another
5 > package run at full speed but only if it was only one of those packages
6 > at a time. Thing is, on occasion two or more of those updates would hit
7 > and due to the long compile times, end up compiling at the same time.
8 > Do you think there is a way for the devs to set up a method to tell
9 > emerge not to emerge certain packages at the same time? In other words,
10 > if Firefox is emerging, LOo is held until it is done or vice versa.
11 > Maybe even have it so others can be listed. The list of large packages
12 > are likely small but they can have a huge impact on systems with less
13 > memory.
14 >
15 > You think that a feature worth asking the devs about? Maybe they can
16 > figure out a way to implement that??
17
18
19 There already is a mechanism you can use, but it’s not the automatic type
20 that you (and, admittedly I) would like.
21
22 I have 3 old 2 core machines, and I use distcc heavily to reduce emerge
23 times. The “fastest” (not really) and best equipped has 16 gb memory. I
24 do updates on this machine (with distcc help from the others) and
25 distribute packages to the rest. After a lot of experimenting, I find that
26 MAKEOPTS=“-j13 -l5” works the best on this fastest machine. That setting
27 allows it to attempt a workload that it alone doesn’t have the resources to
28 accomplish, but successfully distributes to the other machines. I use
29 firefox, chromium, and libreoffice. Occasionally portage wants to upgrade
30 more than one of these at a time, which I discover by running emerge
31 —pretend. On those occasions, I’ve learned that I run out of resources
32 and builds fail. So I just temporarily mask all but one of those updates,
33 do the upgrade, unmask one of the masked updates, do another upgrade, and
34 so on. Works well for me. No builds crash, essentially no swap gets used,
35 and I have substantially accelerated compile and ebuild times.
36
37 The tools exist to do what you want to do. If you were so inclined, you
38 might even contemplate writing a script to automate what I just described.
39
40 John Blinka

Replies

Subject Author
Re: [gentoo-user] per package parallel build Neil Bothwick <neil@××××.net>