Re: [gentoo-user] Traffic volumes for distfiles mirror

[Mick]

On Monday 29 January 2007 17:45, Daniel da Veiga wrote:
> On 1/29/07, Alan McKinnon <alan@××××××××××××××××.za> wrote:
> > On Monday 29 January 2007 11:15, Neil Bothwick wrote:
> > > On Mon, 29 Jan 2007 00:47:47 -0800, kashani wrote:
> > > > I wouldn't bother with a full mirror. Set a local rsync server that
> > > > updates once a day and use http-replicator. That would be far less
> > > > bandwidth than trying to keep a local dist server current.
> > >
> > > If daytime bandwidth is a particular issue, you can set up a cron
> > > task on one of more machines (depending on the variety of packages in
> > > use) to do
> > >
> > > emerge --sync && emerge -uDNf world
> > >
> > > to prime the cache during the night. That should reduce your daytime
> > > downloads to almost zero.
> >
> > The daytime bandwidth is indeed the issue. This is South Africa, where
> > technologically everything is top-notch first-world. Except for
> > bandwidth. By local standards our pipe is quite big - a whopping 512k.
> > Shared amongst two offices and 140 users. At least I get to do whatever
> > I want with the bandwidth after hours - no real users to compete with,
> > just their torrents :-)
> >
> > I already use a fairly complicate solution with emerge -pvf and wget in
> > a cron on one of the fileservers, but it's getting cumbersome. And I'd
> > rather not maintain an entire gentoo install on a server simply to act
> > as a proxy. Would I be right in saying that I'd have to keep
> > the "proxy" machine up to date to avoid the inevitable blockers that
> > will happen in short order if I don't?
> >
> > I've been looking into kashani's suggestion of http-replicator, this
> > might be a good interim solution till I can come up with something
> > better suited to our needs.
>
> I'm using a different setup, of course its a small number of machines
> (like 5 or 6), but it works great. I use NFS to mount
> /usr/portage/distfiles on a server sharing this dir. Each time someone
> request a file, it goes directly to the shared dir, being available
> for all machines. This way, its only 1 request per new file, and only
> files that are needed for update of the particular software most
> machines have in common.

I've set up rsyncd and Boa on the server machine (laptop) which has its 
portage and distfiles updated daily at the office.  Then once a week or so I 
rsync the portage of the home machines with the laptop and they fetch any 
needed distfiles from the Boa server.  For details regarding the set up of 
Boa there was a thread a year or so ago on this list.

Of course there's the odd package that only exists on the LAN machines - they 
pull this off the Internet.  They also insist downloading afresh certain 
binaries (e.g. Opera browser) and some CVS packages.  I guess this is ebuild 
related, was thinking of looking into it with the thought of modifying it one 
day so that all available distfiles are pulled in from the Boa server.

HTH.
-- 
Regards,
Mick