Aw: Re: [gentoo-user] local mirrors (summary, leading to more questions)

Thanks to all who reponded:

  - Rich Freeman suggests:
    - having apache serve the local distfiles.
      How?  Just make them available in the apache root and give portage the URL somewhere?
    - use "infra scripts", but I think that's for running an up-to-date general-purpose mirror

    He explains later that repos.conf defines the single place where the
    repository is defined but that is distinct from where the distfiles
    come from, which is defined by make.conf:GENTOO_MIRRORS

  - Dale suggests:
    - a package, http-replicator, which might do Freeman's first
      suggestion.

  - Neil Bothwick explains  the difference between serving the "portage
    mirror, the repository of ebuild and associated files" and the
    distfiles, and suggests using NFS, presumably instead of, or in
    addition to /usr/portage/distfiles.

    Okay, I looked for the first time at the files directly under /usr/portage.
    Is it a correct assumption that all packages available for installation will be represented there?
    And that's it?  That's basically all there is to a gentoo installation?
    - /usr/portage/distfiles
    - the ebuilds directly under /usr/portage
    - /usr/portage/eclass
    - /usr/portage/metadata
    - /usr/portage/profiles
    - /etc/portage

    If so, then I'm delighted in the simplicity of it.  Like reading the one page definition of tcl.
    Unfortunately, it doesn't come through so well in the documentation,
    with things like layouts seemingly basic to an understanding.

  - Peter Humphrey suggests:
    -  http ftp proxy

        In what way is that different from rsync which I thought I'd already configured?




> Gesendet: Freitag, 26. April 2019 um 07:42 Uhr
> Von: n952162@×××.de
> An: gentoo-user@l.g.o
> Betreff: Aw: Re: [gentoo-user] local mirrors
>
> What I'm hoping for, though, is a configuration flag that causes any of:
> - inhibits a theoretical traversal of repos.conf databases (e.g. also /usr/share/porrage/...)
> - inhibits trying to upgrade beyond what's already installed in the mirror server
>
> > Gesendet: Freitag, 26. April 2019 um 07:32 Uhr
> > Von: n952162@×××.de
> > An: gentoo-user@l.g.o
> > Betreff: Aw: Re: [gentoo-user] local mirrors
> >
> > Thank you for the explanation.  I wonder what the local mirror page means when it talks about saving bandwidth.   What *does* it serve if not the distfiles?  And when /etc/portage/repos.conf points to my local server, why would portage disregard that?
> >
> > The rsync server on the mirror host points to the gentoo portage installation on that local mirror host.  How can any metadata  there know about anything that's not already resolved there?
> >
> > At the very least, I suspect that that local mirror page is wrong and rather refers to something that *could* be implemented (without extra packages being installed, just by configuration), but isn't yet.
> >
> > > Gesendet: Freitag, 26. April 2019 um 00:35 Uhr
> > > Von: "Rich Freeman" <rich0@g.o>
> > > An: gentoo-user@l.g.o
> > > Betreff: Re: [gentoo-user] local mirrors
> > >
> > > On Thu, Apr 25, 2019 at 5:44 PM <n952162@×××.de> wrote:
> > > >
> > > > So, I set up the rsync deamon on my "mirror server" host and the /etc/portage/repos.conf/gentoo.conf file on the client machine, run the recommended test and everything is just as described.  I sync my client, and am very happy.
> > > >
> > > > By coincidence, I happen to look in /var/log/emerge-fetch.log on the client and discover that everything got fetched remotely, from, like gentoo.org and rubygems.org, etc.
> > > >
> > > > What a disappointment!  Is there something I still have to do?
> > > >
> > >
> > > The repository and distfiles both need to be separately mirrored.
> > > Maintaining a distfiles mirror is a bit more complex as all your hosts
> > > probably don't have the same packages installed, and fetching
> > > distfiles for all the packages would use a ton of space (and
> > > bandwidth).  For Gentoo official mirrors this is exactly what is done,
> > > but of course they get used by many users.
> > >
> > > What I have done at times is run apache on one host and serve out its
> > > local distfiles cache, and then list this as the first mirror in the
> > > list for my other hosts.  So, they try to fetch from that host before
> > > going out to the internet.  However, that doesn't do anything to
> > > ensure that the needed files are on that host.  It just helps with
> > > @system packages and other packages the hosts have in common.  That is
> > > an approach that doesn't really cost you anything and probably
> > > provides 75% of the benefit.  It is also easy to do if you have a
> > > bunch of identical hosts and then yields 100% of the benefit (if
> > > they're truly identical I'd go a step further and set up a binpkg
> > > mirror as there is no point in building the same thing many times with
> > > the same flags).
> > >
> > > If you really want to run a full distfiles mirror I'm sure the infra
> > > scripts are floating around somewhere.  It probably just amounts to
> > > running an ebuild fetch on every ebuild in the repo.
> > >
> > > --
> > > Rich
> > >
> > >
> >
> >
>
>