Re: [gentoo-portage-dev] portage-2.2-rc3 parallel merges quit being parallel

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Duncan wrote:
> For the first 100 or so packages, it worked quite well.  However, about
> there, maybe package 120 or so, so about 20% of the way thru, it reverted
> to doing them one-at-a-time again.  I'm now on package #279 and it's
> still doing them only one at a time, and this has included stuff like the
> xorg-docs (IIRC) package, that really shouldn't be pre-deps for stuff
> that has come since, so /something/ else should have been trying to
> merge, as long as load-average stays low, as it has much of the time (I
> have MAKEOPTS="-j -l20" so it's not going to be low all the time).
>
> Is this a known issue still being worked on, perhaps due to the limits of
> the package dependency and scheduling resolution such that it can't
> handle a full world remerge and defaults back to one-at-a-time after it
> reaches the extent of its mapping, or is this a new bug?

The current algorithm is intentionally as conservative as possible
in the sense that it will not build given package in parallel if
there are any packages in it's subgraph of deep dependencies
scheduled to be merged. We can add one or more options to control
the criteria for choosing packages. Those options will modify the
behavior of Scheduler._choose_pkg().

There are a couple of reasons for the current conservative
behavior. In many cases the conservative behavior is beneficial
(avoids breakage) in order to ensure that a package's subgraph of
deep dependencies is up to date before building the package itself.
In addition, the conservative behavior lessens the need to be
concerned about "invariance" requirements like those discussed in
bug 232990 [1].

> Also, a little monitoring utility that could be run in another terminal
> and just list and update all the currently merging packages, and any that
> had failed to merge, so I could take a look at them while the system is
> still working on the rest, would be quite useful.  I tried watch ls -d
> $PORTAGE_TMPDIR/*/* and it starts to work, but of course starts to error
> in a few seconds when the first package completes since the *s are
> resolved initially.  With a bit of work I'm sure can get something simple
> working, but it'd be nice if there were a pre-made utility to do it.
> Maybe there is and I just don't know about it yet?  =8^)

I'm not aware of any tool like that yet. Such a tool should probably
 use the hidden lock file located in the parent directory of a build
directory in order to detect an active build. In the future I plan
to have a daemon process to allow cooperation between multiple
emerge invocations, for things like bug 177311 [2]. Once that's
implemented, there will be a way to query the daemon for information
about running builds.

> Finally... I was rather confused the first time at just one job an
> install took a bit, as that's apparently not counted as "running", so it
> appeared nothing was going on for a bit.  Maybe an "installing" count as
> well would be useful... and prevent that confusion.

There used to be a "merges" count in the status display but somebody
thought it was confusing (darkside/Jeremy Olexa) and I decided that
it wasn't interesting enough to be worthy of it's display space so I
removed it. I guess we can add it back if there's space and demand
for it. Maybe it should only be shown when the job count drops to zero?

Zac

[1] http://bugs.gentoo.org/show_bug.cgi?id=232990
[2] http://bugs.gentoo.org/show_bug.cgi?id=177311
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkiLupEACgkQ/ejvha5XGaP8KQCffKnpIiplEioyP4JOlD8HGD21
Q2QAnAroP5voKc8zyXcCFArPxrYjsec3
=3epU
-----END PGP SIGNATURE-----