Gentoo Archives: gentoo-user

From: Daniel Frey <djqfrey@×××××.com>
To: gentoo-user@l.g.o
Subject: [gentoo-user] Distfiles cache setup
Date: Fri, 18 Sep 2015 17:02:39
Message-Id: 55FC43A3.1050005@gmail.com
1 Hi all,
2
3 I have been running several Gentoo machines here at my house, and am
4 currently up to 7 (or was it 8?) installs.
5
6 I have been trying to reduce my resource consumption and set up an rsync
7 mirror long ago, so my [acting] server only syncs to the internet and
8 all other devices point to it. That part is working fine, I've already
9 moved it to the repos.conf configuration.
10
11 Whenever I search for running a local distfiles mirror (on this list and
12 on the web) it gets a bit murky.
13
14 The way I see it is this can be done a couple of ways:
15
16 1. Set up a lighttpd server to serve the distfiles directory.
17
18 This has the benefit of being able to sync machines outside my network,
19 although I don't know if I'd expose it to the internet.
20
21 The major issue I can see with this is that if the file doesn't exist,
22 portage will crap out saying it's not available. What I don't know is if
23 there's an easy way to "get around" this issue.
24
25 </1.>
26
27 2. Export the distfiles directory.
28
29 This seems to be a bit better of a solution, other than not being able
30 to use it outside the LAN. However, cleaning this directory becomes a
31 lot less trivial as tools used to clean it will assume that the current
32 machine is the only machine using it and clobber other workstation's
33 required distfiles.
34
35 I suppose the easiest way to sync is to wipe it completely out and run
36 `emerge -fe world` on all machines to rebuild it, but this would be a
37 fair bit of work as well.
38
39 </2.>
40
41 With those two options, neither being perfect - it made me wonder if
42 there's a Better Way(tm) to do this.
43
44 In the case of a shared distfiles, it would be best if something was one
45 the machine hosting the distfiles monitoring what workstation needed
46 what file and only removing a file when no workstations request it.
47 Alas, I don't think a tool such as that exists (although I didn't really
48 look that hard.)
49
50 Ideally, it would be nice to have some sort of caching proxy that could
51 fetch the file as it was needed, but in searching for this I encountered
52 so much noise in the search results I gave up for the time being.
53
54 Anyone have any suggestions?
55
56 Dan

Replies

Subject Author
Re: [gentoo-user] Distfiles cache setup Peter Humphrey <peter@××××××××××××.uk>
Re: [gentoo-user] Distfiles cache setup Neil Bothwick <neil@××××××××××.uk>
Re: [gentoo-user] Distfiles cache setup hydra <hydrapolic@×××××.com>
Re: [gentoo-user] Distfiles cache setup "J. Roeleveld" <joost@××××××××.org>