RE: [gentoo-user] Jobs and load-average

[Laurence Perkins]

> -----Original Message-----
> From: Rich Freeman <rich0@g.o>
> Sent: Thursday, February 16, 2023 6:24 AM
> To: gentoo-user@l.g.o
> Subject: Re: [gentoo-user] Jobs and load-average
>
> On Thu, Feb 16, 2023 at 8:39 AM Peter Humphrey <peter@××××××××××××.uk> wrote:
> >
> > I've just looked at 'man make', from which it's clear that -j =
> > --jobs, and that both those and --load-average are passed to
> > /usr/bin/make, presumably untouched unless portage itself has
> > identically named variables. So I wonder how feasible it might be for
> > make to incorporate its own checks to ensure that the load average is
> > not exceeded. I am not a programmer (not for at least 35 years, anyway), so I have to leave any such suggestion to the experts.
> >
>
> Well, if we just want to have a fun discussion here are my thoughts.
> However, the complexity vs usefulness outside of Gentoo is such that I don't see it happening.
>
> For the most typical use case - a developer building the same thing over and over (which isn't Gentoo), then make could cache info on resources consumed, and use that to make more educated decisions about how many tasks to launch.  That wouldn't help us at all, but it would help the typical make user.  However, the typical make user can just tune things in other ways.
>
> It isn't going to be possible for make to estimate build complexity in any practical way.  Halting problem aside maybe you could build in some smarts looking at the program being executed and its arguments, but it would be a big mess.
>
> Something make could do is tune the damping a bit.  It could gradually increase the number of jobs it runs and watch the load average, and gradually scale it up appropriately, and gradually scale down if CPU is the issue, or rapidly scale down if swap is the issue.  If swapping is detected it could even suspend most of the tasks it has spawned and then gradually continue them as other tasks finish to recover from this condition.  However, this isn't going to work as well if portage is itself spawning parallel instances of make - they'd have to talk to each other or portage would somehow need to supervise things.
>
> A way of thinking about it is that when you have portage spawning multiple instances of make, that is a bit like adding gain to the --load-average MAKEOPTS.  So each instance of make independently looks at load average and takes action.  So you have an output (compilers that create load), then you sample that load with a time-weighted average, and then you apply gain to this average, and then use that as feedback.  That's basically a recipe for out of control oscillation.
> You need to add damping and get rid of the gain.
>
> Disclaimer: I'm not an engineer and I suspect a real engineer would be able to add a bit more insight.
>
> Really though the issue is that this is the sort of thing that only impacts Gentoo and so nobody else is likely to solve this problem for us.
>
> --
> Rich


Expanding the capabilities of app-admin/cpulimit to tally up memory usage and suspend new compile jobs temporarily when it gets too high probably wouldn't be too horribly difficult...



LMP