[gentoo-portage-dev] Re: portage-2.2-rc3 parallel merges quit being parallel

[Duncan]

Followup to a rather old post...

Zac Medico posted on Sat, 26 Jul 2008 17:00:18 -0700 as excerpted:

> Duncan wrote:
>> For the first 100 or so packages, it worked quite well.  However, about
>> there, maybe package 120 or so, so about 20% of the way thru, it
>> reverted to doing them one-at-a-time again.

FWIW, that has long since been fixed. =:^)

>> Finally... I was rather confused the first time at just one job an
>> install took a bit, as that's apparently not counted as "running", so
>> it appeared nothing was going on for a bit.  Maybe an "installing"
>> count as well would be useful... and prevent that confusion.
>
> There used to be a "merges" count in the status display but somebody
> thought it was confusing (darkside/Jeremy Olexa) and I decided that it
> wasn't interesting enough to be worthy of it's display space so I
> removed it. I guess we can add it back if there's space and demand for
> it. Maybe it should only be shown when the job count drops to zero?

From my point of view, the current display (as of portage-2.2_rc67) is 
still lacking/confusing in this regard.  What I see happening here is that 
a number of packages will be building, then finish building one at a time, 
but the completed count doesn't go up.  They seem to sit there built but 
not installing for quite awhile, then all install at once.  But it's hard 
to tell as there's not an "installing" count.

Now part of this may be due to the way I have jobs setup.  

MAKEOPTS="-j13 -l10"

My emerge jobs, OTOH, are

"--jobs=4 --load-average=7"

(BTW, it would sure be nice if there was a make.conf variable for those, 
similar to MAKEOPTS, and then a simple command-line option to toggle it on 
and off.  I don't put them in my default options as when I'm 
troubleshooting a broken merge or am otherwise just merging a single 
package, it's nice to be able to run non-parallel jobs and thus have the 
output "live", but when I'm doing full updates, I want the parallel jobs.  
The logical way to handle this would be to set the --jobs and
--load-average parameters in a var similar to MAKEOPTS, and then use a 
command-line option to toggle parallel jobs mode on or off.)

Meanwhile...

The rational behind having portage jobs lower than makeopts is that since 
I have PORTAGE_TMPDIR pointed at a tmpfs, I want to favor currently 
running package emerges over starting new ones, because every new package 
started unpacks into that tmpfs, thereby using more memory.  Thus, if it's 
possible to run more parallel jobs on already started package merges, I 
want it to do that instead of starting more packages.  The way to do that 
is to keep the number of --jobs and --load-average lower than the 
corresponding MAKEOPTS parameters, so MAKEOPTS gets used first if possible 
and only if there's no more parallel jobs possible at that level, does the 
load average drop down far enough for another package to start merging 
beside the currently merging ones.

But for whatever reason, portage seems to sit there doing nothing with the 
already built and ready to install packages, preferring to start more 
package builds rather than finish off the installs of the ones it already 
has built.  So I can end up with 7 or 8 packages sitting there built, but 
apparently not installing, then it goes and installs them all at once!  
That's contrary to my strategy of trying to favor already started 
packages, and only starting new ones when the existing running ones can't 
keep the load average up high enough.

But as I said it's hard to track that, since portage doesn't track the 
current number of installings, only the number built.  I can however infer 
from the difference between the number of the last started job, the number 
of completed jobs, and the number of running jobs, plus the per-package 
installing notices, that a whole stack of packages are accumulating that 
are already built, but are apparently not installing yet.  Why isn't 
portage going ahead and installing them, instead of starting new package 
builds?

So the two issues (plus the one in parentheses above) I see are:

1) the parallel jobs display needs to say how many it's installing.

2) portage needs to follow thru on built packages and finish installing 
them as a higher job priority than starting new package builds.  
Particularly with --jobs=4, there's no reason portage should be starting a 
new package build, when there's a whole stack of packages (often more than 
the four the --jobs=4 implies should be the max) apparently sitting there 
built and ready to install, but not actively installing, and apparently 
not doing anything except sitting there taking up tmpfs memory!

3) Having an EJOBS or similar variable, parallel to the MAKEOPTS variable, 
and then a simple parallel-toggle emerge command-line option, would be 
quite useful. =:^)

Let me know if you'd prefer that I file bugs on these, and as always, 
thanks, Zac, for being so responsive.  You have a gift that's a real 
rarity in user/dev relations, and some of us really do appreciate it! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman