1 |
hi guys, ive been browsing the gentoo-dev archives and i dont think this |
2 |
idea has been suggested before, but apologies if i missed it. |
3 |
|
4 |
any feedback, or flames, appreciated :) |
5 |
|
6 |
basically, a modified distcc package is installed by users wishing to |
7 |
participate in an internet-wide distcc network. Users run the modified |
8 |
distccd which contacts a central server every 10 minutes while running, |
9 |
indicating its GCC version, GLIBC version and Architecture. |
10 |
|
11 |
Other information, eg geographic information based on timezone could |
12 |
be sent, so that geogrpahically closer users are favoured when selecting |
13 |
hosts. (This would improve build times instead of giving a british user |
14 |
10 hosts on a 56k modem in australia) |
15 |
|
16 |
The central server maintains a list of clients that are ready to |
17 |
accept work, if no message is received after 10 minutes it is assumed |
18 |
the client is no longer ready to accept work and is removed from the |
19 |
pool. The server software could be purpose made, or a php/perl script |
20 |
and a mysql database. |
21 |
|
22 |
When a user wishes to compile something from portage using the |
23 |
distributed network, they use their normal emerge command, prefixed |
24 |
with `dportage` eg: |
25 |
|
26 |
$ `dportage` emerge gnome |
27 |
|
28 |
the dportage command contacts the remote server, and fetches a list |
29 |
of 10 compatible machines ready to accept work, then prints a line |
30 |
containing environment variables needed to use the network, similar |
31 |
to how ssh-agent works. example output (this could be configurable): |
32 |
|
33 |
$ dportage |
34 |
DISTCC_HOSTS="123.123.123.123 123.123.123...." \ |
35 |
FEATURES="distcc" MAKEOPTS="-j12" |
36 |
|
37 |
if the user isnt listed in the pool of users ready to accept work, |
38 |
they are not permitted to use the network, and dportage prints |
39 |
nothing. This prevents leeching. |
40 |
|
41 |
The obvious issue is security, what stops a client returning work |
42 |
that is trojaned, and might allow compromising the host? |
43 |
|
44 |
The modified distcc client sends the work to two random DISTCC_HOSTS |
45 |
and the returned work must match for it to be accepted. If it doesnt |
46 |
match, the distcc client can resend the work to different hosts |
47 |
and possibly submit the non-working hosts to a blacklist on the |
48 |
central server, which will remove it from the pool of clients. |
49 |
|
50 |
That could be easy to abuse, so the client must have been reported |
51 |
to the blacklist by two different users for it to be removed. |
52 |
|
53 |
Benefits |
54 |
|
55 |
* Users who dont have a LAN get to benefit from distcc |
56 |
* Faster compiles for all users |
57 |
* Users can run the daemon nice, so as not to interfere with their work |
58 |
* Easy to prevent leeching. |
59 |
* Less complaints from users on slow machines who want binary |
60 |
packages. |
61 |
|
62 |
|
63 |
I've already made some tests on using distcc over the internet, it is |
64 |
oviously slower than running it over a nice LAN, but it is still fast, |
65 |
and im guessing that a 10 host DISTCC_HOSTS would negate this. The |
66 |
client can obvioulsy be tweaked to timeout quicker, and maybe change the |
67 |
order of DISTCC_HOSTS so as to favour the faster responding |
68 |
clients...etc. |
69 |
|
70 |
in case you arent familiar with it, http://distcc.samba.org/ |
71 |
|
72 |
-- |
73 |
------------------------------------- |
74 |
taviso@××××××××××××.org | finger me for my gpg key. |
75 |
------------------------------------------------------- |
76 |
|
77 |
-- |
78 |
gentoo-dev@g.o mailing list |