Re: [gentoo-alt] gemato in prefix

[Michael Weiser]

Hi Fabian,

On Wed, Feb 21, 2018 at 10:04:42AM +0100, Fabian Groffen wrote:

> To give you an idea; the current tree is getting its Manifests from
> hashgen.c, which you can find in scripts/rsync-generation/hashgen.c.
> The hashverify tool, which I'm currently working on, is basically an
> addition to that file (doing argv[0] detection) to perform the
> verification.  At this time of writing, I have the gpg-verification and
> single file entry verification in place.  I'm still trying to close the
> gap in checking the dirs in particular looking for files that are not
> listed in the Manifest.

I've been following or rather post-reading the discussion about GLEP-74
and it seems a lot of thought and security considerations have gone into
it. Are you following it for your implementation?

I see you're using OpenMP. That looks very neat and could certainly make
it scale better than gemato.

BTW: I remember OpenMP being plain missing from clang until recently.
Ah, https://clang-omp.github.io/ says 3.7 and onwards have native
support.

> hashgen currently runs in 30s or so on the tree to generate manifests.
> I hope it can verify in the same amount of time (we're talking about a
> Quad G5 PowerPC machine here with rusty old spinning disks), leaving it
> in a much better position to be used for Prefix, since we tend to have
> slower/older machines around.

I can only believe that number if the portage tree fits into the fs
cache (RAM). Then it would be purely CPU-bound, I guess, and C could
play out its advantage.

On a machine that has just synced the tree using rsync it would also
still be in RAM as a working set (if there was sufficient RAM) and the
same consideration would apply. On my MacBook Air I get 625012k from du
for the tree. So this should be alright for any machine with 1GB+ of RAM
that's not doing anything else at the time.

As soon as the tree doesn't completely fit into RAM, the later files of
the rsync will push the first files out of the cache. So for
verification, the reading will start from scratch, making the
verification I/O-bound and the advantage of a C implementation mostly
irrelevant, won't it?

As a quick number: On an ARM SBC of mine with a Dual-Core 1.2GHz
Cortex-A7, 1GB of RAM and fs on a microSD card that does 22MB/s
sequential read I get 400 seconds flat for the first run of gemato
verify and 386 for the second when the fs cache is primed already. I do
see a bit of I/O-boundness on the first run and on the second, gemato
fully hogs the CPU. It would be interesting to compare how much of that
is python overhead and how much the actual crypto. Unfortunately the gcc
on that board doesn't have openmp support (yet).

gemato create runs 667s on the same board and falls back to a single
thread for some reason about a minute in. So I'd certainly take a
22fold speedup if I could get it. :)

> We're not really looking forward to 15
> minutes of verification as some bugs have been reported to with gemato.

Well, with webrsync I was looking at around three minutes of download
and five minutes of unpacking and local rsync on my ARM SBCs. With
gemato it's now about 30 seconds to three minutes of rsyncing depending
on the amount of changes and six minutes of gemato checking. So
basically I've neither gained nor lost anything but feel much more
efficient by once again not downloading what hasn't changed.

It also shows that I have a high pain threshold in dealing with Gentoo
on small and old machines. Considering how long Gentoo users typically
wait for compiles to finish I wouldn't have expected a bit of a wait for
gemato or hashverify to be much of an issue.

> Portage used to do 1) checking the digests of ebuilds and 2) checking
> for missing and extra files.  I noticed that at least 1) is no longer
> present, which I find weird.  I need checking this on normal Gentoo,
> (simply edit an ebuild and try to emerge it without updating its
> digest), but I have the suspicion this got disabled because the full
> tree verification should catch this.  Needless to say, that's
> suboptimal, and not very secure IMO.

Ah, now I understand what you were getting at. I just verified: Both
checks are working on Gentoo Linux but not in Prefix Mac. It seems to
have to do with setting "thin-manifests" in layout.conf of the prefix
tree. Once I set that to false, prefix portage behaves the same as
Gentoo Linux portage.

Gentoo Linux:

[root@linux:/usr/portage/xfce-base/garcon] sha512sum garcon-0.6.1.ebuild 
dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7 garcon-0.6.1.ebuild
[root@linux:/usr/portage/xfce-base/garcon] grep EBUILD.*0.6.1.*SHA512 Manifest 
EBUILD garcon-0.6.1.ebuild 1018 SHA512 dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7
[root@linux:/usr/portage/xfce-base/garcon] sed -i -e "s,econf,fooconf," garcon-0.6.1.ebuild 
[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild compile
 * Digest verification failed:
 * /usr/portage/xfce-base/garcon/garcon-0.6.1.ebuild
 * Reason: Filesize does not match recorded size
 * Got: 1020
 * Expected: 1018

[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild digest
>>> Creating Manifest for /usr/portage/xfce-base/garcon
[root@linux:/usr/portage/xfce-base/garcon] mkdir files
[root@linux:/usr/portage/xfce-base/garcon] echo foo > files/foo
[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild prepare
 * garcon-0.6.1.tar.bz2 BLAKE2B SHA512 size ;-) ...
[ ok ]
 * checking ebuild checksums ;-) ...
[ ok ]
 * checking miscfile checksums ;-) ...
[ ok ]
!!! A file is not listed in the Manifest:
'/usr/portage/xfce-base/garcon/files/foo'
[root@linux:/usr/portage/xfce-base/garcon] touch garcon-0.7.0.ebuild
[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild prepare
 * A file is not listed in the Manifest: '/usr/portage/xfce-base/garcon/garcon-0.7.0.ebuild'
[root@linux:~] grep thin /usr/portage/metadata/layout.conf
# Use thin Manifests for Git
thin-manifests = false

Prefix Mac:

root@mac:/gentoo/usr/portage/xfce-base/garcon $ sha512sum garcon-0.6.1.ebuild 
dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7 garcon-0.6.1.ebuild
root@mac:/gentoo/usr/portage/xfce-base/garcon $ grep EBUILD.*0.6.1.*SHA512 Manifest 
EBUILD garcon-0.6.1.ebuild 1018 SHA512 dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7
root@mac:/gentoo/usr/portage/xfce-base/garcon $ sed -i -e "s,econf,fooconf," garcon-0.6.1.ebuild 
root@mac:/gentoo/usr/portage/xfce-base/garcon $ sha512sum garcon-0.6.1.ebuild 
42f04d921b82955f8ac6237b4f2ebd78fad1cb0f2ab3cbf0f11b4b055b5a20c0cba6f2a37639b759a8912ec728bbde2f491b57d7565066f3ef5f4587e8f787a5 garcon-0.6.1.ebuild
root@mac:/gentoo/usr/portage/xfce-base/garcon $ ebuild garcon-0.6.1.ebuild compile
>>> Downloading 'http://distfiles.gentoo.org/distfiles/garcon-0.6.1.tar.bz2'
[...]

root@mac:/gentoo/usr/portage/xfce-base/garcon $ grep -r thin /gentoo/usr/portage/metadata/layout.conf 
# Use thin Manifests for Git
thin-manifests = true
root@mac:/gentoo/usr/portage/xfce-base/garcon $ sed -i -e "s,thin-manifests = true,thin-manifests = false," /gentoo/usr/portage/metadata/layout.conf 

root@mac:/gentoo/usr/portage/xfce-base/garcon $ ebuild garcon-0.6.1.ebuild prepare
>>> Existing ${T}/environment for 'garcon-0.6.1' will be sourced. Run
>>> 'clean' to start with a fresh environment.
 * Missing digest for '/gentoo/usr/portage/xfce-base/garcon/garcon-0.6.1.ebuild'

root@nindamos:/usr/local/gentoo/usr/portage/xfce-base/garcon $ mkdir files
root@nindamos:/usr/local/gentoo/usr/portage/xfce-base/garcon $ touch files/foo
root@nindamos:/usr/local/gentoo/usr/portage/xfce-base/garcon $ ebuild garcon-0.6.1.ebuild prepare
 * garcon-0.6.1.tar.bz2 BLAKE2B SHA512 size ;-) ...
[ ok ]
 * checking ebuild checksums ;-) ...
[ ok ]
 * checking miscfile checksums ;-) ...
[ ok ]
!!! A file is not listed in the Manifest: '/usr/local/gentoo/usr/portage/xfce-base/garcon/files/foo'

> Now I may be all wrong with trying to implement the verification myself,
> but that's a separate topic.  gemato should work fine.

I think it's a worthy cause just for the fact that multiple
implementations of a protocol are always good to drive improvement and
show up the corner cases.

Maybe it shouldn't be hidden away in the prefix tree so others can more
easily become aware of and contribute to it.

> I've checked some of the digests you mentioned and they look ok.  So I'm
> wondering whether perhaps you got caught in the middle of a sync.  This
> used to be much less of a problem because of per-ebuild-dir integrity,
> but now the entire tree requires to be consistent.  I'll look into
> re-activating my symlink-flip, which should make the switch atomic, but
> I don't know what rsync is doing if the symlink is flipped during a
> sync.  It reduces the invalid window somewhat I guess.

I've retried a couple of times today. I deleted timestamp.chk several
times and resynced until I saw no delta any more. There actually was a
delta on the second and sometimes third run, mostly Manifest.gzs and
metadata. It still fails reliably on second-level Manifests, e.g.
dev-dotnet/Manifest.gz and sys-libs/Manifest.gz 

It looks as if regeneration of Manifests and md5-cache runs almost
continuously, not just every 26,56 of the hour as rsync's motd says. Is
that intentional or are perhaps multiple instances of the script running
amok?
-- 
Micha
Elephants don't play chess!