[gentoo-dev] Re: About EGO_SUM

[Madhu]

* "Robin H. Johnson" <robbat2-20220608T184338-394361540Z @orbis-terrarum.net> :
Wrote on Wed, 8 Jun 2022 20:42:48 +0000:
> EGO_SUM vs dependency tarballs:
> - bloats ebuilds
> - bloats Manifests
> - bloats metadata/md5-cache/ (SRC_URI etc)
> - doesn't bloat mirrors with gentoo-unique distfiles
> - EGO_SUM is verifiable/reproducible from Upstream Go systems
> - less downloads on upgrades (only changed Go deps, not entire dep tarballs)
>
> EGO_SUM data right now adds, to every user's system:
> - 2.6MB of text to ebuilds (340k after de-dupe)
> - 7MB of text to Manifests (2M after de-dupe)
> - 6.4MB+ of text to metadata/md5-cache (I don't have a easy way to
> calc deduped amount here)
> On the server side:
> - The sum total of Go distfiles mirrored on Gentoo mirrors right now
> is only 3.4GB.
> - less downloads
>
> Dependency tarballs:
> - Right now ~15GiB on each mirror, plus storage of the primary copy
>   somewhere (dev.g.o right now, but not great)
> - Conservatively if the remaining EGO_SUM packages converted to Dep
>   tarballs, it would need another 8GB each of primary location and
>   mirrors.
> - larger downloads for users who DO want to upgrade a Go package (all
>   new deps tarball even if only one or two deps changed)
> - must be preserved much longer, unless we can introduce a guaranteed
>   way to regenerate them for any prior ebuild.
>
> I was trying to introduce a third option, but I haven't had the time to
> write an entire GLEP.
>
> The TL;DR is introducing a 2nd-level Manifest+metadata file, that tries
> to move just the metadata out of the tree, in a way that can be
> regenerated (specifically, a 1:1 reproducible creation from a given go.sum).
> It DOES need to contain slightly more data than the present Manifest,
> specifically a full SRC_URI entry for each file (upstream URI plus what
> to rename it to on Gentoo side)
>
> The 2nd-level Manifest would be listed as SRC_URI, and be handled in
> src_fetch/src_unpack. Download & verify the extra distfiles, against the
> Manifest checksum data (and for Golang against go.sum checksums).
>
> The Portage mirrordist code needs the most work in this case, as it
> would need to fetch the 2nd-level Manifests so it can populate Gentoo
> mastermirror with the distfiles mirrored from upstream.
>
> The storage costs for the proposed idea:
> - same 1:1 base distfile storage as EGO_SUM (e.g. upstream distfiles are
>   mirrored 1:1 content, just different naming)
> - Probably 1 Metadata-Manifest file per ebuild $PVR (conceptually it
>   could be split more or shared between some ebuilds/packages)
> - Main tree Manifests: 1 DIST entry per Metadata-Manifest in a given package
> - Main tree ebuilds: 1 line for the Metadata-Manifest in the ebuild.
> - metadata/md5-cache: 1 src_uri line!
> - mirrors: add the Metadata-Manifest

[Without claiming to have fully understood the proposal above: around
Apr 15th 22 I tried suggesting to WilliamH on IRC that perhaps portage
should implement the dirhash approach that go has taken to solve the
problem of upstream sources when they invented go.sum.

from hash.go in sources
 go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go

 // Hash1 is "h1:" followed by the base64-encoded SHA-256 hash of a
 summary prepared as if by the Unix command:find . -type f | sort |
 sha256sum

loosely speaking the "manifest" could publish this dirhash of contents
of go-mod/cache (which would have been bundled in the -deps.tar.xz)

The immediate motivation was to avoid the network when I already had the
sources locally: instead of downloading a -deps.tar.xz I could create it
locally and dump it in distdir. portage would check the (hypothetically)
published dirhash and let it through. the local timestamps and uid in my
tarball and the upstream tarball wouldn't upset it.

One unchecked assumption is that go-mod/cache can be recreated by
unpacking sources. If so then with a notion of a "second level manifest"
(the equivalent of go.sum) the contents can be assembled without having
to store or download the actual -deps tarball.

I didn't get very far in convincing WilliamH of my need so I dropped
the idea. (I'm not sure if I'm being any clearer, if I'm missing
something, do let me know)