Re: [gentoo-user] Possible heads up: Digest verification failed

[Daniel Campbell]

On 10/29/2016 03:11 AM, Rich Freeman wrote:
> On Sat, Oct 29, 2016 at 5:42 AM, Neil Bothwick <neil@××××××××××.uk> wrote:
>> On Fri, 28 Oct 2016 23:53:03 -0700, Daniel Campbell wrote:
>>
>>>>     Anyone seeing this again? I've just sync'd to two servers in
>>>> Australia, and then, for the hell of it, one in Canada and am getting
>>>> it for:
>>>>
>>>> dev-libs/botan
>>>> app-arch/tar
>>>> media-video/libav
>>>> app-crypt/qca
>>>> net-print/cups-filters
>>>>
>>>> I suppose time will sort it out.....
>>>>
>>>>     Andrew
>>>>
>>> This shouldn't happen unless the distfiles aren't found, or someone (a
>>> dev) didn't use repoman to commit, leaving an old Manifest around.
>>
>> It looks like repoman is the culprit
>>
>> https://bugs.gentoo.org/show_bug.cgi?id=598376
>>
>
> This is probably not the issue here, since Gentoo uses thin manifests
> (there is nothing for repoman to update).  The manifests that are
> causing the problem aren't created by regular Gentoo developers.
> They're created by a script that runs as a part of the rsync mirror
> process.
>
> This is a fairly well-known problem that has been around for over a year.
>
> You will only run into this problem if you use rsync to update your
> repository, since the problem is created when creating the master
> rsync mirror.  The original git repository doesn't contain the error,
> and the git mirror on github doesn't mess with the Manifests.
>
> The issue apparently has to do with Changelog generation.  In April
> the Council gave Infra the option to stop generating Changelogs, which
> would eliminate the problem.  I suspect those maintaining the scripts
> prefer to keep them around, and I don't think anybody on the Council
> has access to change the scripts.
>
> I switched to git syncing eons ago, so I've never seen this bug.  I
> recognize it has been a source of frustration for a lot of users, and
> a bit of frustration for the Council, since there doesn't seem to be a
> lot we can do to change it in practice.
>
> zlg is of course right that these kinds of problems can also be caused
> by maintainer failure to use repoman/etc or if an upstream distfile
> changes.  If that is the problem then you'll see it no matter how you
> sync your repo.  However, when you get a bunch of these after syncing
> it is almost always a result of the mirror creation process.  I can't
> remember the last time I saw a manifest error (granted, I'm also using
> mgorny's stable mirror branch, which I think screens for these kinds
> of errors).
>
> While there can be some latency I do in general recommend syncing from
> https://github.com/gentoo-mirror/gentoo .  This is a mirror of the
> Gentoo developer git repository with two changes:
> 1.  Metadata is added to the mirror, which greatly speeds things up
> compared to using the raw git repository (you can do this yourself, it
> is one of the steps done by the rsync generation process as well, but
> this one is not buggy).
> 2.  The default stable branch of this mirror screens for numerous
> issues before accepting commits.  That means it is generally a little
> behind the main branch (at this moment the main branch is 2 minutes
> old, and the default stable branch is 20min old), but a lot of the
> really annoying issues that are caused by devs skipping repoman won't
> be seen.   Now, if a maintainer breaks a package then this mirror will
> quickly get out of date until the problem is corrected, but Gentoo QA
> gets warnings when this is happening and usually the maintainer is
> being pestered or somebody else is fixing it.  I suspect this process
> has probably reduced the error rate for everybody.  I have seen this
> get a few days old though, which is something to keep in mind.
>
> It does not contain Changelogs, though if you use it you'll have a
> full history so you can just run git whatchanged <path> to get
> something pretty close to a changelog.
>
> To use it just put this in /etc/portage/repos.conf/gentoo.conf:
> [DEFAULT]
> main-repo = gentoo
>
> [gentoo]
> location = /usr/portage
> sync-type = git
> sync-uri = https://github.com/gentoo-mirror/gentoo.git
> auto-sync = yes
>
> If you want to git-sync from some other mirror, just change the url accordingly.
>
> If you switch your mirror I suggest just renaming /usr/portage and
> letting portage re-create it.
>
> The other big benefit of git syncing is that if you sync every day it
> is a lot faster.  If you sync less often it will become slower
> compared to rsync.  git is much more efficient at finding what has
> changed, but rsync is not burdened with transferring a complete
> history.  If you only sync once every few months rsync will be a lot
> faster.
>
Wow, quite the mail there! I actually wasn't aware of thin vs. thick
manifest issues (or mgorny's work on a stable mirror). Thanks for
sharing and teaching.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6