Re: [gentoo-mirrors] mirror fetch jobs and --checksum

[Robin H. Johnson]

On Thu, Jan 28, 2016 at 08:42:33PM -0200, Carlos Carvalho wrote:
> The issue is not calculating checksums, it's I/O. The gentoo repository is now
> 335GB. It's out of question to read it all at every update.
> And you should know it!
The primary module I care about is gentoo-portage (historically
gentoo-x86) is under 400MiB raw (~207k inodes however, so watch your
inode space waste).  If your mirror DOESN'T have that entire rsync
module sitting in cache already, you probably have fairly low traffic.
If it's already sitting in cache, there should be no IO hit.
Hashing that much _was_ a CPU hit a decade ago, and most people did NOT
have the memory to fit it all into cache either.

The other issues of mtimes of Manifests esp on ebuild removal have been
resolved by the various series of patches, but part of those were
artificially bumping the mtime by a single second in certain cases.

Those cases are STILL going to have a bumpy time in some situations
because Git's commit time resolution is only 1 second (ditto many
filesystems). Those situations either need sub-second resolution in the
entire ecosystem or checksums of some form.

We've had ~27k commits in the last 6 months, of which:
- 4811 colliding (commit timestamp only)
- 45 colliding (author timestamp only)
- 12 colliding (commit timestamp, author timestamp)

We're using author timestamps on the outgoing rsync files, and
eventually we ARE going to have a real collision that hits users.

This didn't happen CVS because even if you were really fast (read: local
to the CVS server, and did not go via SSH), you could only get it down
to about 2 seconds per commit, and never in the same package. Two
different devs could never a commit to the same package in the same
second.

> Also, we block --checksum from clients. Most big mirrors do it.
The rsync cached checksums patch needs to get popular again, because
then the mirrors won't have any huge burden at all:
- update the checksums when syncing from the parent repo
- compare against the checksums when queried by the client

> Concerning storing the checksums, you're asking mirrors to use a patched rsync
> version for gentoo? Forget it.
Actually, I want upstream rsync to accept the rsync cached checksums
patch; it was dropped a few years ago amongst other major changes and
never picked up again due to lack of maintainer.

If the original stats still hold up, the patch will actually improve
performance even without --checksum, as it can save on stat calls on the
server side (it caches a lot more than just the checksum).

> It's the master job to update the repository as you need. Asking mirrors to
> bear an enormous load because you cannot do your job is silly, to put it
> mildly. You'll be ignored by big mirrors, as you've been since your first post.
I've got at least TWO cases of distfiles, upstream from Gentoo, being
screwed up by the relevant authors.

These happen infrequently enough that we'll continue to deal with them
on an exception basis.

There are also known cases where attackers have interfered with mirrors
(upstream & distributions); and you can expect that an attacker could
REASONABLY set the mtime of a file (if the size & mtime are the same,
and you aren't using --checksum in your cronjob to fetch from upstream,
the attack is going to persist). Catching this in all of the cases is
very difficult to do cheaply in the middle tiers. Cappos [1] included
this in his thesis on Attacks on Package Managers.

> If you provide a list of checksums, like Debian does, I can use it. However I
> know of no other mirror that has such functionality.
MirrorBrain, as used by SUSE uses checksums internally (and provides
them to end users). It can verify checksums from mirrors. Downside is
that you have to run apache (at least on the master, and on some of the
mirrors for best results).

[1] https://www.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@g.o
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85