Gentoo Archives: gentoo-dev

From: taviso <taviso@××××××××××××.org>
To: gentoo-dev@g.o
Subject: [gentoo-dev] opt-in distributed portage network
Date: Mon, 14 Apr 2003 10:47:20
Message-Id: 20030414104714.GA20989@sdf.lonestar.org
1 hi guys, ive been browsing the gentoo-dev archives and i dont think this
2 idea has been suggested before, but apologies if i missed it.
3
4 any feedback, or flames, appreciated :)
5
6 basically, a modified distcc package is installed by users wishing to
7 participate in an internet-wide distcc network. Users run the modified
8 distccd which contacts a central server every 10 minutes while running,
9 indicating its GCC version, GLIBC version and Architecture.
10
11 Other information, eg geographic information based on timezone could
12 be sent, so that geogrpahically closer users are favoured when selecting
13 hosts. (This would improve build times instead of giving a british user
14 10 hosts on a 56k modem in australia)
15
16 The central server maintains a list of clients that are ready to
17 accept work, if no message is received after 10 minutes it is assumed
18 the client is no longer ready to accept work and is removed from the
19 pool. The server software could be purpose made, or a php/perl script
20 and a mysql database.
21
22 When a user wishes to compile something from portage using the
23 distributed network, they use their normal emerge command, prefixed
24 with `dportage` eg:
25
26 $ `dportage` emerge gnome
27
28 the dportage command contacts the remote server, and fetches a list
29 of 10 compatible machines ready to accept work, then prints a line
30 containing environment variables needed to use the network, similar
31 to how ssh-agent works. example output (this could be configurable):
32
33 $ dportage
34 DISTCC_HOSTS="123.123.123.123 123.123.123...." \
35 FEATURES="distcc" MAKEOPTS="-j12"
36
37 if the user isnt listed in the pool of users ready to accept work,
38 they are not permitted to use the network, and dportage prints
39 nothing. This prevents leeching.
40
41 The obvious issue is security, what stops a client returning work
42 that is trojaned, and might allow compromising the host?
43
44 The modified distcc client sends the work to two random DISTCC_HOSTS
45 and the returned work must match for it to be accepted. If it doesnt
46 match, the distcc client can resend the work to different hosts
47 and possibly submit the non-working hosts to a blacklist on the
48 central server, which will remove it from the pool of clients.
49
50 That could be easy to abuse, so the client must have been reported
51 to the blacklist by two different users for it to be removed.
52
53 Benefits
54
55 * Users who dont have a LAN get to benefit from distcc
56 * Faster compiles for all users
57 * Users can run the daemon nice, so as not to interfere with their work
58 * Easy to prevent leeching.
59 * Less complaints from users on slow machines who want binary
60 packages.
61
62
63 I've already made some tests on using distcc over the internet, it is
64 oviously slower than running it over a nice LAN, but it is still fast,
65 and im guessing that a 10 host DISTCC_HOSTS would negate this. The
66 client can obvioulsy be tweaked to timeout quicker, and maybe change the
67 order of DISTCC_HOSTS so as to favour the faster responding
68 clients...etc.
69
70 in case you arent familiar with it, http://distcc.samba.org/
71
72 --
73 -------------------------------------
74 taviso@××××××××××××.org | finger me for my gpg key.
75 -------------------------------------------------------
76
77 --
78 gentoo-dev@g.o mailing list

Replies

Subject Author
Re: [gentoo-dev] opt-in distributed portage network scrllock@××××××××××.com