Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Gregory M. Turner]

On 9/11/2012 9:54 AM, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 11/09/12 12:43 PM, Zac Medico wrote:
>> On 09/11/2012 09:36 AM, vivo75@×××××.com wrote:
>>> Dunno where to place this request, but if we go for something
>>> like EJOBS could we also make it phase specific? So compile,
>>> install and test could have a different number of jobs running.
>>> Possibly three different variables that override a predefined
>>> EJOBS.
>>
>> Per-phase sounds a little to fine-grained. Instead, I'd suggest to
>> add an ELOADAVG variable that's analogous to make's --load-average
>> option. That should be enough to compensate for any differences
>> between phases.
>
> I personally wonder about why this would be necessary from the
> perspective of the user; if the user's system at emerge time can
> handle X concurrent processes per emerge-job , i don't see why it
> would matter what phase these jobs would be launched from.
>
> At the ebuild level, certainly, but that's one of the reasons for
> EJOBS in the first place, so that it can be overridden consistently
> within a phase, if necessary for the ebuild (regardless of build
> system type), right?

I would hope so.  There are definitely real-world reasons to want to 
restrict jobs -- usually to just one -- in a particular phase... several 
ebuilds, notably several in @system, do this already by injecting -j1 
somewhere.

To say it's all about performance seems to be forgetting a major reason 
somebody might want to do this as a user or an ebuild author.

Hmm...

I should preface this by saying that it really pains me to say this, 
because now this just starts to seem so fucking complicated -- but now 
that I think about it, this really seems to highlight a kind of semantic 
discrepancy that this thread has mostly dusted under the rug.

There are really TWO things we seem to care about -- one is some kind of 
global build-parallelism frob-set, and the other is the ability to turn 
off all parallelism for certain parts of an ebuild and those are 
actually kind-of orthogonal.

In other words, imagine if we had some kind of "global parallellism 
arbiter" thingy that looked at EJOBS or ELOADAVG or maybe some other 
things we haven't thought of yet, and decided things like:

   o should portage start a new parallel emerge process?
   o What is the appropriate "MAKEOPTS" for the "emake"
     that foo-1.0.ebuild just issued?

or even, one could imagine, such things as:

   o should portage decide that parallelism has gotten out of control
     somehow and suspend or kill a running ebuild for subsequent
     resuming or restarting?

   o should portage re-nice some of its processes?

Note that, effectively, we have this already, and it's called "portage".
But one could certainly make a case for modularizing it better, since, 
in truth, we are talking about a very common, very abstract problem here 
which portage shares with any number of batch-build systems.

Such an engine could very well do exactly the right thing if it were 
faced with a constraint that a certain part of a certain build needed to 
proceed without parallelism due to limitations coming from the build.

Also, there are very large parts of most builds -- configure comes to 
mind -- that don't parallelize even if, perhaps, they should.  In such 
cases, a really smart global parallelism arbiter could easily respond by 
spawning more jobs from other builds.

Not sure what I'm suggesting we do about it, exactly, but just pointing 
out that maybe a completely "correct" solution requires a much more 
elaborate implementation than just a bunch of syntactic sugar around 
what we currently call MAKEOPTS.  Whether or not Gentoo wants to take 
that all on, right now, as part of the next EAPI, is certainly debatable 
-- in fact, I'm inclined to say maybe it's not the best idea.

Perhaps we should really be asking: is the status-quo really so 
problematic/inelegant that it needs fixing? -- before we decide how best 
to fix it.

-gmt