Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method

[Andrew Savchenko]

On Tue, 18 Dec 2018 03:36:14 -0800 Raymond Jennings wrote:
> On Tue, Dec 18, 2018 at 1:56 AM Andrew Savchenko <bircoph@g.o> wrote:
> > On Sat, 15 Dec 2018 23:15:47 -0500 Alec Warner wrote:
> > > Hi,
> > >
> > > I am currently embarking on a plan to redo our existing rsync[0] mirror
> > > network. The current network has aged a bit. Its likely too large and is
> > > under-maintained. I think in the ideal case we would instead pivot this
> > > project to scaling out our git mirror capabilities and slowly migrate all
> > > consumers to pulling the git tree directly. To that end, I'm looking for
> > > blockers as to why various customers cannot switch to pulling the gentoo
> > > ebuild repository from git[1] instead of rsync.
> > >
> > > So for example:
> > >
> > > - bandwidth concerns (preferably with documentation / data.)
> > > - Firewall concerns
> > > - CPU concerns (e.g. rsync is great for tiny systems?)
> > > - Disk usage for git vs rsync
> > > - Other things i have not thought of.
> >
> > My main concern with git is downlink fault tolerance. If rsync
> > connection is broken, it can be easily restored without much data
> > retransmission. If git download connection is broken, it has to
> > start all over again. So there are cases where rsync will be always
> > much more preferable than git.
>
> Are you talking about in comparison to the initial clone?
> If so, would having the clone default to shallow mitigate this?
>
> For the curious, I ran a benchmark.
>
> With a completely purged /usr/portage:
>
> emerge-webrsync took 30.302s
> emerge-sync (with git clone --depth 1) took 33.902s
> emerge-sync (with regular rsync) took a whoping 1m25.863s
>
> After a fresh sync:
>
> emerge-sync (with regular rsync) took 7.564s
> emerge-sync (with git fetch --depth 1, and after priming the repo with
> a full clone) took 2.086s
>
>
>
> Up front, webrsync seems to be a small winner for initial setups, with
> git clone a close second, and regular rsync is 3 fold worse
>
> Routine syncs would seem to prefer git, especially if they are done
> with presistent regularity which IMO would amortize things.  My
> opinion is that over time git would also place less stress on the
> servers since it only has to look at the commit chain instead of
> checksumming every single file.
>
>
>
> That said, would I be correct to surmise that you're advancing a
> robustness issue and not simply a performance issue?

Yes, my interest here is in robustness, not performance. Sometimes I
have to use unreliable uplink and other users may face the same
problem.

I agree that in most cases git should be a preferred way to go, but
there are exceptions. So it would be nice to have rsync backup just
in case.

Daily or weekly portage snapshots available via rsync should be a
solution as well.

Best regards,
Andrew Savchenko