Gentoo Archives: gentoo-project

From: "Robin H. Johnson" <robbat2@g.o>
To: "Michał Górny" <mgorny@g.o>
Cc: gentoo-project@l.g.o, robbat2@g.o, gyakovlev@g.o, sam@g.o
Subject: Re: [gentoo-project] Call for agenda items - Council meeting on 2022-02-13
Date: Wed, 09 Feb 2022 23:16:33
Message-Id: robbat2-20220209T221608-886776481Z@orbis-terrarum.net
In Reply to: Re: [gentoo-project] Call for agenda items - Council meeting on 2022-02-13 by "Michał Górny"
1 On Wed, Feb 09, 2022 at 10:51:30PM +0100, Michał Górny wrote:
2 > On Wed, 2022-02-09 at 14:32 -0600, William Hubbs wrote:
3 > > On Wed, Feb 09, 2022 at 06:09:55PM +0100, Michał Górny wrote:
4 > >
5 > > *snip*
6 > >
7 > > > This would:
8 > > >
9 > > > 1) break Gentoo installations behind restrictive firewalls,
10 > >
11 > > Maybe, maybe not. If the go module cache is populated on the firewalled
12 > > system ahead of time things would still work. could emerge -f handle
13 > > this?
14 > >
15 > > > 2) make parallel fetching much harder,
16 > >
17 > > I do not have any comments on this.
18 > >
19 > > > 3) would require custom implementations to support caching
20 > > > and mirroring,
21 > >
22 > > Go already handles both of these with the go proxy mirrors and the
23 > > GOMODCACHE environment variable.
24 > > We could, in the go-module eclass, point GOMODCACHE to for example
25 > > <distdir>/go-mod.
26 >
27 > So basically every sysadmin running Gentoo will now have to account for
28 > special case of fetching Go sources in addition to fetching generic
29 > Gentoo distfiles.
30 You're focusing a lot on Go here, whereas on IRC I was talking about the
31 general problem space.
32
33 Yes, Go is the biggest nail sticking out right now, but it's a growing
34 problem overall.
35 - Golang modules
36 - Rust crates
37 - NodeJS modules
38 - Texlive packages
39
40
41 Third party systems would be required to provide suitable security on
42 their distfiles. Go & Rust do. I think NodeJS & Tex don't, but I'm happy
43 to be proven wrong.
44
45 Right now, a sysadmin who wants a disconnected Gentoo environment needs
46 to mirror at least the distfiles in question, or at least the critical
47 files.
48
49 The *easiest* ways for them to do:
50 - emerge -f & share that distdir
51 - just mirror all of distfiles
52
53
54 If they already do "emerge -f", this would be changing to something like
55 "emerge --deep-fetch", that runs:
56 src_fetch & src_unpack & src_fetch_extra
57
58 src_fetch_extra would be responsible for: fetching, verification, and
59 putting the downloads into $DISTDIR, or subdirectories thereof
60
61
62 > > > 4) will eventually lead to ebuilds fetching and using unverified data.
63 > > Not for go at least.
64 > And how are you going to guarantee that this humongous security hole
65 > will not be used wrong for anything else?
66 Unverifiable data, or verification bugs (a long time ago I wrote about a
67 gap in the go.sum checksum mechanism that can sneak in extra files).
68
69 The distfile security at that point becomes only as strong as the code
70 in src_fetch_extra, or anything that is called.
71
72 We need to step back and look at the situation holistically.
73
74 What is the problem statement?
75 I've seen these two so far:
76 - Manifests are getting very big, with duplicated entries between
77 packages.
78 - Environment variables are getting too big and cause errors.
79
80 But I don't think those capture the entire problem, and I'd really like
81 more clarity on what the actual problem is.
82
83
84 Potential families of solutions as I proposed:
85 ----------------------------------------------
86 1) make 1:1 upstream distfile to gentoo distfile official and accepted
87 EVEN if a package has tens of thousands of distfiles.
88 1.1) Provide a way to share duplicated Manifest entries between packages
89
90 2) make distfile bundling "official", accepting large growth in mirror usage
91 2.1) This would include tooling to make it easy, repeatable, verifiable etc.
92
93 3) Additional fetch phases, that depend on inputs from src_unpack: this
94 would run things like ecosystem-specific fetch tooling.
95 That tooling would be responsible for the steps I mentioned above,
96 parallel download, verifications, writing to distdir in a way the
97 ecosystem could use
98
99
100 --
101 Robin Hugh Johnson
102 Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
103 E-Mail : robbat2@g.o
104 GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
105 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Attachments

File name MIME type
signature.asc application/pgp-signature

Replies