Re: [gentoo-user] Possible heads up: Digest verification failed

[Rich Freeman]

On Sat, Oct 29, 2016 at 5:42 AM, Neil Bothwick <neil@××××××××××.uk> wrote:
> On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote:
>
>> >     Anyone seeing this again? I've just sync'd to two servers in
>> > Australia, and then, for the hell of it, one in Canada and am getting
>> > it for:
>> >
>> > dev-libs/botan
>> > app-arch/tar
>> > media-video/libav
>> > app-crypt/qca
>> > net-print/cups-filters
>> >
>> > I suppose time will sort it out.....
>> >
>> >     Andrew
>> >
>> This shouldn't happen unless the distfiles aren't found, or someone (a
>> dev) didn't use repoman to commit, leaving an old Manifest around.
>
> It looks like repoman is the culprit
>
> https://bugs.gentoo.org/show_bug.cgi?id=598376
>

This is probably not the issue here, since Gentoo uses thin manifests
(there is nothing for repoman to update).  The manifests that are
causing the problem aren't created by regular Gentoo developers.
They're created by a script that runs as a part of the rsync mirror
process.

This is a fairly well-known problem that has been around for over a year.

You will only run into this problem if you use rsync to update your
repository, since the problem is created when creating the master
rsync mirror.  The original git repository doesn't contain the error,
and the git mirror on github doesn't mess with the Manifests.

The issue apparently has to do with Changelog generation.  In April
the Council gave Infra the option to stop generating Changelogs, which
would eliminate the problem.  I suspect those maintaining the scripts
prefer to keep them around, and I don't think anybody on the Council
has access to change the scripts.

I switched to git syncing eons ago, so I've never seen this bug.  I
recognize it has been a source of frustration for a lot of users, and
a bit of frustration for the Council, since there doesn't seem to be a
lot we can do to change it in practice.

zlg is of course right that these kinds of problems can also be caused
by maintainer failure to use repoman/etc or if an upstream distfile
changes.  If that is the problem then you'll see it no matter how you
sync your repo.  However, when you get a bunch of these after syncing
it is almost always a result of the mirror creation process.  I can't
remember the last time I saw a manifest error (granted, I'm also using
mgorny's stable mirror branch, which I think screens for these kinds
of errors).

While there can be some latency I do in general recommend syncing from
https://github.com/gentoo-mirror/gentoo .  This is a mirror of the
Gentoo developer git repository with two changes:
1.  Metadata is added to the mirror, which greatly speeds things up
compared to using the raw git repository (you can do this yourself, it
is one of the steps done by the rsync generation process as well, but
this one is not buggy).
2.  The default stable branch of this mirror screens for numerous
issues before accepting commits.  That means it is generally a little
behind the main branch (at this moment the main branch is 2 minutes
old, and the default stable branch is 20min old), but a lot of the
really annoying issues that are caused by devs skipping repoman won't
be seen.   Now, if a maintainer breaks a package then this mirror will
quickly get out of date until the problem is corrected, but Gentoo QA
gets warnings when this is happening and usually the maintainer is
being pestered or somebody else is fixing it.  I suspect this process
has probably reduced the error rate for everybody.  I have seen this
get a few days old though, which is something to keep in mind.

It does not contain Changelogs, though if you use it you'll have a
full history so you can just run git whatchanged <path> to get
something pretty close to a changelog.

To use it just put this in /etc/portage/repos.conf/gentoo.conf:
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = git
sync-uri = https://github.com/gentoo-mirror/gentoo.git
auto-sync = yes

If you want to git-sync from some other mirror, just change the url accordingly.

If you switch your mirror I suggest just renaming /usr/portage and
letting portage re-create it.

The other big benefit of git syncing is that if you sync every day it
is a lot faster.  If you sync less often it will become slower
compared to rsync.  git is much more efficient at finding what has
changed, but rsync is not burdened with transferring a complete
history.  If you only sync once every few months rsync will be a lot
faster.

-- 
Rich