Re: [gentoo-soc] Project Proposal : GenCC for "Gentoo Community Compiling"

[Rich Freeman]

On Mon, Apr 1, 2013 at 10:43 AM, Donnie Berkholz <dberkholz@g.o> wrote:
> The problem in the end is that it wasn't really feasible on anything
> besides a 100 MBit *minimum* connection with low latency, otherwise you
> spent more time transferring files around than compiling them.
>
> Today you could imagine it being potentially interesting if you think
> about it in the context of public (or private) cloud living in shared
> datacenters.

I messed around with distcc in combination with EC2 thinking I might
make my life easier when my previous box was taking eons to compile
things like chromium.  The goal was to have my box at home just send
out work and do no little local compilation.  It was as much an
experiment as anything.

I found that the latency of the connection REALLY slowed things down
unless you had a very high level of parallelism (think 32-64 files in
transit at any time).  If I got the number of jobs high enough it
would perform better, and I'd have many more jobs than hosts because
the reality is that 75% of the jobs were just travelling over the
network and not running at any time.

The problems were:
1.  Few build systems could actually get that many jobs running in parallel.
2.  distcc only manages a queue for gcc jobs run in parallel, but make
-j64 runs everything in parallel.
3.  distcc still requires dependencies to be present on all of the
build systems.

#2 meant that when some java program built I'd end up with 64 java VMs
just killing my system.  Even shell or python scripts would just kill
the thing, and many build systems are a combination of C and
scripts/etc.

It might do marginally better if you could improve the build systems,
but for the most part it is local-only.  It might make more sense in
more dedicated roles like a tinderbox that only builds a single
package as it could be optimized to overcome those issues.

Rich