Gentoo Archives: gentoo-dev

From: Francesco Riosa <vivo75@×××××.com>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] Six month major project on Gentoo
Date: Thu, 22 Dec 2011 11:12:10
In Reply to: Re: [gentoo-dev] Six month major project on Gentoo by Rich Freeman
2011/12/22 Rich Freeman <rich0@g.o>:
> On Wed, Dec 21, 2011 at 11:43 PM, Donnie Berkholz <dberkholz@g.o> wrote: >> I looked into this 6 or 7 years ago. It wasn't feasible unless you were >> on an extremely high-speed, low-latency network, beyond what was >> typically accessible at the time outside of universities and LANs. Could >> be worth exploring again now that 25-100 mbps connections are becoming >> more common. > > I tried messing around with this with Amazon EC2.  The problem was > that due to latency I only really saw the benefit for VERY high levels > of parallelization (think -j25+)..  However, make isn't actually > "distcc-aware" so it just runs 25 jobs of anything in parallel.  So, > anytime a makefile launched a ton of java or python jobs the host > ground to a halt as it wasn't distributed and it was way more than the > host could handle (especially java - which swapped like there was no > tomorrow). > > If somebody were to do a distcc-ng for a large cluster one of the > problems to solve would be having it not run jobs in parallel if it > couldn't actually distribute them. > > Rich >
Just wanted to point out that (if there is enough memory) recent kernels manage much better parallelism, even excess of it, once reached the maximum load augmenting threads only bring minimal loss of "real" time.


Subject Author
Re: [gentoo-dev] Six month major project on Gentoo "Michał Górny" <mgorny@g.o>