Re: [gentoo-dev] Multiple emerges in parallel (was: [RFC] RESTRICT=parallel for builds that can't be executed in parallel)

[Michał Górny]

On Tue, 13 Apr 2010 23:10:16 -0700
Brian Harring <ferringb@×××××.com> wrote:

> Running multiple emerges in parallel is already a bad idea.  The
> solution for that case is for the new/second emerge to feed the
> request into the original emerge (or a daemon).

Although such solution will be useful in many cases indeed, there are
still many advantages of having few separate emerge calls running
in parallel.

First of all, the 'emerge daemon' would have to be able to
transparently inject the additional packages onto the dependency tree.
And just appending missing dependencies and the real package at the end
doesn't seem sufficient.

For example, what if such operation would result in a conflict? If
that's only that simple as 'A and B can not be installed at the same
time', the second emerge call could reject appending the packages,
explaining the reason for that.

But what if the conflict could be resolved through pulling in some
other version of one of the packages? We could try to replace
the package in the dependency tree silently if it didn't yet start
being merged; but what if it did? Should we reject adding
the packages, encouraging user to 'try again later'? Or maybe abort
the running builds to merge another version of the depend? Same goes
for virtual dependencies which can be fulfilled by many different
packages.

Of course, that's just implementation-wide issue and running separate
emerge instances could result in even worse results of this problem.
But apart from that, there are side uses of multiple emerges which
would be hard to implement with your idea.

For example, I often use different PORTAGE_TMPDIR to build large
packages like xulrunner. Although it doesn't seem that hard
implementing the support for running merges in different temporary
directories at the same time, should we really waste time implementing
that? What about other possible configuration changes? What about
'USE' overrides within the environment? These ones should certainly be
respected.

So, now our daemon not only has to be able to efficiently mangle
dependency trees at runtime but also to support partial runtime
configuration changes.

The next thing is aborting merges. When running multiple emerges,
aborting one of them is as simple as pressing ^c. With daemon, we would
have to implement an ability of aborting/removing packages in runtime
-- and that would be another example of dependency tree mangling.

But in fact, implementing (at least limited) the support of such
mangling would be useful in some cases. With '--keep-going', it would
allow to drop the packages and continue merging the remaining ones
without waiting for current builds to complete.

And looking from the other side, the support for removing single
packages from the running emerge dependency tree would be useful too
with current emerge concept. Even if it just worked like
'--keep-going', it would be still better than the current workaround --
waiting for particular emerge to start and removing its ${WORKDIR} to
force failure.

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@××××××.ru>