Gentoo Archives: gentoo-dev

From: "Robin H. Johnson" <robbat2@g.o>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] About EGO_SUM
Date: Wed, 08 Jun 2022 20:42:55
In Reply to: [gentoo-dev] About EGO_SUM by Florian Schmaus
1 On Fri, Jun 03, 2022 at 01:18:08PM +0200, Florian Schmaus wrote:
2 > EGO_SUM is marked as 'deprecated' in go-module.eclass [1, 2]. I
3 > acknowledge that there are packages where the usage of EGO_SUM is very
4 > problematic. However, I wonder if there are packages where using
5 > dependency tarballs is problematic while using EGO_SUM would be not.
6 ... [snip all the great points]
7 > Even more problematic are that dependency tarballs require additional
8 > steps that would not be required when EGO_SUM is used. While those steps
9 > appear simple, behavioral theory shows that even the tiniest additional
10 > steps have a huge impact (e.g., online shops loose a relative large
11 > share of customers if for each an additional checkout step). If we force
12 > dependency tarballs for Go software, then packaging Go software just
13 > become a little bit harder.
14 Your above is entirely correct, and I was against the plan to introduce
15 dependency tarballs.
17 > This leads me to the question why are we actually deprecating EGO_SUM?
18 > It seems like a nice alternative for Go packaging that we may want to
19 > keep. But maybe I am missing something?
20 EGO_SUM vs dependency tarballs:
21 - bloats ebuilds
22 - bloats Manifests
23 - bloats metadata/md5-cache/ (SRC_URI etc)
24 - doesn't bloat mirrors with gentoo-unique distfiles
25 - EGO_SUM is verifiable/reproducible from Upstream Go systems
26 - less downloads on upgrades (only changed Go deps, not entire dep tarballs)
28 EGO_SUM data right now adds, to every user's system:
29 - 2.6MB of text to ebuilds (340k after de-dupe)
30 - 7MB of text to Manifests (2M after de-dupe)
31 - 6.4MB+ of text to metadata/md5-cache (I don't have a easy way to calc deduped amount here)
32 On the server side:
33 - The sum total of Go distfiles mirrored on Gentoo mirrors right now is only 3.4GB.
34 - less downloads
36 Dependency tarballs:
37 - Right now ~15GiB on each mirror, plus storage of the primary copy
38 somewhere (dev.g.o right now, but not great)
39 - Conservatively if the remaining EGO_SUM packages converted to Dep
40 tarballs, it would need another 8GB each of primary location and
41 mirrors.
42 - larger downloads for users who DO want to upgrade a Go package (all
43 new deps tarball even if only one or two deps changed)
44 - must be preserved much longer, unless we can introduce a guaranteed
45 way to regenerate them for any prior ebuild.
47 I was trying to introduce a third option, but I haven't had the time to
48 write an entire GLEP.
50 The TL;DR is introducing a 2nd-level Manifest+metadata file, that tries
51 to move just the metadata out of the tree, in a way that can be
52 regenerated (specifically, a 1:1 reproducible creation from a given go.sum).
53 It DOES need to contain slightly more data than the present Manifest,
54 specifically a full SRC_URI entry for each file (upstream URI plus what
55 to rename it to on Gentoo side)
57 The 2nd-level Manifest would be listed as SRC_URI, and be handled in
58 src_fetch/src_unpack. Download & verify the extra distfiles, against the
59 Manifest checksum data (and for Golang against go.sum checksums).
61 The Portage mirrordist code needs the most work in this case, as it
62 would need to fetch the 2nd-level Manifests so it can populate Gentoo
63 mastermirror with the distfiles mirrored from upstream.
65 The storage costs for the proposed idea:
66 - same 1:1 base distfile storage as EGO_SUM (e.g. upstream distfiles are
67 mirrored 1:1 content, just different naming)
68 - Probably 1 Metadata-Manifest file per ebuild $PVR (conceptually it
69 could be split more or shared between some ebuilds/packages)
70 - Main tree Manifests: 1 DIST entry per Metadata-Manifest in a given package
71 - Main tree ebuilds: 1 line for the Metadata-Manifest in the ebuild.
72 - metadata/md5-cache: 1 src_uri line!
73 - mirrors: add the Metadata-Manifest
75 --
76 Robin Hugh Johnson
77 Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
78 E-Mail : robbat2@g.o
79 GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
80 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


File name MIME type
signature.asc application/pgp-signature


Subject Author
[gentoo-dev] Re: About EGO_SUM Madhu <enometh@××××.net>
Re: [gentoo-dev] About EGO_SUM Sebastian Pipping <sping@g.o>