Gentoo Archives: gentoo-dev

From: Oskari Pirhonen <xxc3ncoredxx@×××××.com>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] Proposal to undeprecate EGO_SUM
Date: Mon, 27 Jun 2022 06:09:27
Message-Id: YrlJipi3PZsyHgXu@dj3ntoo
In Reply to: Re: [gentoo-dev] Proposal to undeprecate EGO_SUM by Zoltan Puskas
1 On Mon, Jun 27, 2022 at 01:43:19 +0200, Zoltan Puskas wrote:
2 > Hi,
3 >
4 > I've been working on adding a go based ebuild to Gentoo yesterday and I
5 > got this warning form portage saying that EGO_SUM is deprecated and
6 > should be avoided. Since I remember there was an intense discussion
7 > about this on the ML I went back and have re-read the threads before
8 > writing this piece. I'd like to provide my perspective as user, a
9 > proxied maintainer, and overlay owner. I also run a private mirror on my
10 > LAN to serve my hosts in order to reduce load on external mirrors.
11 >
12 > Before diving in I think it's worth reading mgorny's blog post "The
13 > modern packager’s security nightmare"[1] as it's relevant to the
14 > discussion, and something I deeply agree with.
15 >
16 > With all that being said, I feel that the tarball idea is a bad due to
17 > many reasons.
18 >
19 > From security point of view, I understand that we still have to trust
20 > maintainers not to do funky stuff, but I think this issue goes beyond
21 > that.
22 >
23 > First of all one of the advantages of Gentoo is that it gets it's source
24 > code from upstream (yes, I'm aware of mirrors acting as a cache layer),
25 > which means that poisoning source code needs to be done at upstream
26 > level (effectively means hacking GitHub, PyPi, or some standalone
27 > project's Gitea/cgit/gitlab/etc. instance or similar), sources which
28 > either have more scrutiny or have a limited blast radius.
29 >
30 > Additionally if an upstream dependency has a security issue it's easier
31 > to scan all EGO_SUM content and find packages that potentially depend on
32 > a broken dependency and force a re-pinning and rebuild. The tarball
33 > magic hides this completely and makes searching very expensive.
34 >
35 > In fact using these vendor tarballs is the equivalent of "static
36 > linking" in the packaging space. Why are we introducing the same issue
37 > in the repository space? This kills the reusability of already
38 > downloaded dependencies and bloats storage requirements. This is
39 > especially bad on laptops, where SSD free space might be limited, in
40 > case the user does not nuke their distfiles after each upgrade.
41 >
42 > Considering that BTRFS (and possibly other filesystems) support on the
43 > fly compression the physical cost of a few inflated ebuilds and
44 > Manifests is actually way smaller than the logical size would indicate.
45 > Compare that to the huge incompressible tarballs that now we need to
46 > store.
47 >
48 > As a proxied maintainer or overlay owner hosting these huge tarballs
49 > also becomes problem (i.e. we need some public space with potentially
50 > gigabytes of free space and enough bandwidth to push that to users).
51 > Pushing toward vendor tarballs creates an extra expense on every level
52 > (Gentoo infra, mirrors, proxy maintainers, overlay owners, users).
53 >
54 > If bloating portage is a big issue and we frown upon go stuff anyway (or
55 > only a few users need these packages), why not consider moving all go
56 > packages into an officially supported go packages only overlay? I
57 > understand that this would not solve the kernel buffer issue where we
58 > run out of environment variable space, but it would debloat the main
59 > portage tree.
60 >
62 Rephrasing this just to ensure I'm understanding it correctly: you're
63 suggesting to move _everything_ that uses Go into its own overlay. Let's
64 call it gentoo-go for the sake of the example.
66 If the above is accurate, then I hard disagree.
68 The biggest package that I have that uses Go is docker (and accompanying
69 tools). Personal distaste of docker aside, it's a very popular piece of
70 software, and I don't think it's fair to require all the people who want
71 to use it to first enable and sync gentoo-go before they can install it.
73 And what about transitive dependencies? Suppose app-misc/cool-package is
74 written in some language that isn't Go, but it has a dependency on
75 sys-apps/cool-util which has a dependency on something written in Go.
76 Should a user wanting to install cool-package have to enable the
77 gentoo-go overlay now too? Even though app-misc/cool-package would look
78 like it doesn't need the overlay unless you dig into the deps.
80 Not a dev, just a user who really likes Gentoo :)
82 - Oskari
84 > It also breaks reproducibility. With EGO_SUM I can check out an older
85 > version of portage tree (well to some extent) and rebuild packages since
86 > dependency upstream is very likely to host old versions of their source.
87 > With the tarballs this breaks since as soon as an ebuild is dropped from
88 > mainline portage the vendor tarballs follow them too. There is no way
89 > for the user to roll back a package a few weeks back (e.g. if new
90 > version has bugs), unlike with EGO_SUM.
91 >
92 > In fact I feel this goes against the spirit of portage too, since now
93 > instead of "just describing" how to obtain sources and build them, now
94 > it now depends on essentially ephemeral blobs, which happens to be
95 > externalized from the portage tree itself. I'm aware that we have
96 > ebuilds that pull in patches and other stuff from dev space already, but
97 > we shouldn't make this even worse.
98 >
99 > Finally with EGO_SUM we had a nice tool get-ego-vendor which produced
100 > the EGO_SUM for maintainers which has made maintenance easier. However I
101 > haven't found any new guidance yet on how to maintain go packages with
102 > the new tarball method (e.g. what needs to go into the vendor tarball,
103 > what changes are needed in ebuilds). Overall this complifates further
104 > ebuild development and verification of PRs.
105 >
106 > In summary, IMHO the EGO_SUM way of handling of go packages has more
107 > benefits than drawbacks compared to the vendor tarballs.
108 >
109 > Cheers,
110 > Zoltan
111 >
112 > [1]
113 >
114 >


File name MIME type
signature.asc application/pgp-signature


Subject Author
Re: [gentoo-dev] Proposal to undeprecate EGO_SUM Zoltan Puskas <zoltan@×××××××××.info>