Re: [gentoo-dev] About EGO_SUM

[Robin H. Johnson]

On Fri, Jun 03, 2022 at 01:18:08PM +0200, Florian Schmaus wrote:
> EGO_SUM is marked as 'deprecated' in go-module.eclass [1, 2]. I
> acknowledge that there are packages where the usage of EGO_SUM is very
> problematic. However, I wonder if there are packages where using
> dependency tarballs is problematic while using EGO_SUM would be not.
... [snip all the great points]
> Even more problematic are that dependency tarballs require additional
> steps that would not be required when EGO_SUM is used. While those steps
> appear simple, behavioral theory shows that even the tiniest additional
> steps have a huge impact (e.g., online shops loose a relative large
> share of customers if for each an additional checkout step). If we force
> dependency tarballs for Go software, then packaging Go software just
> become a little bit harder.
Your above is entirely correct, and I was against the plan to introduce
dependency tarballs.

> This leads me to the question why are we actually deprecating EGO_SUM?
> It seems like a nice alternative for Go packaging that we may want to
> keep. But maybe I am missing something?
EGO_SUM vs dependency tarballs:
- bloats ebuilds
- bloats Manifests
- bloats metadata/md5-cache/ (SRC_URI etc)
- doesn't bloat mirrors with gentoo-unique distfiles
- EGO_SUM is verifiable/reproducible from Upstream Go systems
- less downloads on upgrades (only changed Go deps, not entire dep tarballs)

EGO_SUM data right now adds, to every user's system:
- 2.6MB of text to ebuilds (340k after de-dupe)
- 7MB of text to Manifests (2M after de-dupe)
- 6.4MB+ of text to metadata/md5-cache (I don't have a easy way to calc deduped amount here)
On the server side:
- The sum total of Go distfiles mirrored on Gentoo mirrors right now is only 3.4GB.
- less downloads

Dependency tarballs:
- Right now ~15GiB on each mirror, plus storage of the primary copy
  somewhere (dev.g.o right now, but not great)
- Conservatively if the remaining EGO_SUM packages converted to Dep
  tarballs, it would need another 8GB each of primary location and
  mirrors.
- larger downloads for users who DO want to upgrade a Go package (all
  new deps tarball even if only one or two deps changed)
- must be preserved much longer, unless we can introduce a guaranteed
  way to regenerate them for any prior ebuild.

I was trying to introduce a third option, but I haven't had the time to
write an entire GLEP.

The TL;DR is introducing a 2nd-level Manifest+metadata file, that tries
to move just the metadata out of the tree, in a way that can be
regenerated (specifically, a 1:1 reproducible creation from a given go.sum).
It DOES need to contain slightly more data than the present Manifest,
specifically a full SRC_URI entry for each file (upstream URI plus what
to rename it to on Gentoo side)

The 2nd-level Manifest would be listed as SRC_URI, and be handled in
src_fetch/src_unpack. Download & verify the extra distfiles, against the
Manifest checksum data (and for Golang against go.sum checksums).

The Portage mirrordist code needs the most work in this case, as it
would need to fetch the 2nd-level Manifests so it can populate Gentoo
mastermirror with the distfiles mirrored from upstream.

The storage costs for the proposed idea:
- same 1:1 base distfile storage as EGO_SUM (e.g. upstream distfiles are
  mirrored 1:1 content, just different naming)
- Probably 1 Metadata-Manifest file per ebuild $PVR (conceptually it
  could be split more or shared between some ebuilds/packages)
- Main tree Manifests: 1 DIST entry per Metadata-Manifest in a given package
- Main tree ebuilds: 1 line for the Metadata-Manifest in the ebuild.
- metadata/md5-cache: 1 src_uri line!
- mirrors: add the Metadata-Manifest

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@g.o
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136