Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method

[Alec Warner]

On Tue, Dec 18, 2018 at 12:14 PM Andrew Savchenko <bircoph@g.o>
wrote:

> On Tue, 18 Dec 2018 03:36:14 -0800 Raymond Jennings wrote:
> > On Tue, Dec 18, 2018 at 1:56 AM Andrew Savchenko <bircoph@g.o>
> wrote:
> > > On Sat, 15 Dec 2018 23:15:47 -0500 Alec Warner wrote:
> > > > Hi,
> > > >
> > > > I am currently embarking on a plan to redo our existing rsync[0]
> mirror
> > > > network. The current network has aged a bit. Its likely too large
> and is
> > > > under-maintained. I think in the ideal case we would instead pivot
> this
> > > > project to scaling out our git mirror capabilities and slowly
> migrate all
> > > > consumers to pulling the git tree directly. To that end, I'm looking
> for
> > > > blockers as to why various customers cannot switch to pulling the
> gentoo
> > > > ebuild repository from git[1] instead of rsync.
> > > >
> > > > So for example:
> > > >
> > > > - bandwidth concerns (preferably with documentation / data.)
> > > > - Firewall concerns
> > > > - CPU concerns (e.g. rsync is great for tiny systems?)
> > > > - Disk usage for git vs rsync
> > > > - Other things i have not thought of.
> > >
> > > My main concern with git is downlink fault tolerance. If rsync
> > > connection is broken, it can be easily restored without much data
> > > retransmission. If git download connection is broken, it has to
> > > start all over again. So there are cases where rsync will be always
> > > much more preferable than git.
> >
> > Are you talking about in comparison to the initial clone?
> > If so, would having the clone default to shallow mitigate this?
> >
> > For the curious, I ran a benchmark.
> >
> > With a completely purged /usr/portage:
> >
> > emerge-webrsync took 30.302s
> > emerge-sync (with git clone --depth 1) took 33.902s
> > emerge-sync (with regular rsync) took a whoping 1m25.863s
> >
> > After a fresh sync:
> >
> > emerge-sync (with regular rsync) took 7.564s
> > emerge-sync (with git fetch --depth 1, and after priming the repo with
> > a full clone) took 2.086s
> >
> >
> >
> > Up front, webrsync seems to be a small winner for initial setups, with
> > git clone a close second, and regular rsync is 3 fold worse
> >
> > Routine syncs would seem to prefer git, especially if they are done
> > with presistent regularity which IMO would amortize things.  My
> > opinion is that over time git would also place less stress on the
> > servers since it only has to look at the commit chain instead of
> > checksumming every single file.
> >
> >
> >
> > That said, would I be correct to surmise that you're advancing a
> > robustness issue and not simply a performance issue?
>
> Yes, my interest here is in robustness, not performance. Sometimes I
> have to use unreliable uplink and other users may face the same
> problem.
>
> I agree that in most cases git should be a preferred way to go, but
> there are exceptions. So it would be nice to have rsync backup just
> in case.


> Daily or weekly portage snapshots available via rsync should be a
> solution as well.
>

Two things here. One is that in an ideal world we would run no rsync
service and any design should keep that outcome in mind. Operationally we
should continue to offer rsync until these types of problems are addressed
by the new system.

The second is that in this case I think the plan is to, as Robin mentioned,
offer "git bundles" that are over raw http and support resume-able
downloads. So instead of downloading an "rsync snapshot" you download a git
bundle over http. Infra would offer these git bundles in a similar way to
existing rsync snapshot offerings[0]. These bundles would be applied to a
machine local clone of a git repo. Does this conceptually address your
problem? I agree it will be difficult to know outside of actual practical
testing.

-A

[0] http://gentoo.ussg.indiana.edu/snapshots/ is one example of the current
system. Instead of tarballs of an 'rsync tree' these would be git
bundles[1] that you fetch and apply locally. We would support a worldwide
mirror network for these bundles.
[1] https://git-scm.com/docs/git-bundle


>
> Best regards,
> Andrew Savchenko
>