1 |
On Wed, Feb 09, 2022 at 10:51:30PM +0100, Michał Górny wrote: |
2 |
> On Wed, 2022-02-09 at 14:32 -0600, William Hubbs wrote: |
3 |
> > On Wed, Feb 09, 2022 at 06:09:55PM +0100, Michał Górny wrote: |
4 |
> > |
5 |
> > *snip* |
6 |
> > |
7 |
> > > This would: |
8 |
> > > |
9 |
> > > 1) break Gentoo installations behind restrictive firewalls, |
10 |
> > |
11 |
> > Maybe, maybe not. If the go module cache is populated on the firewalled |
12 |
> > system ahead of time things would still work. could emerge -f handle |
13 |
> > this? |
14 |
> > |
15 |
> > > 2) make parallel fetching much harder, |
16 |
> > |
17 |
> > I do not have any comments on this. |
18 |
> > |
19 |
> > > 3) would require custom implementations to support caching |
20 |
> > > and mirroring, |
21 |
> > |
22 |
> > Go already handles both of these with the go proxy mirrors and the |
23 |
> > GOMODCACHE environment variable. |
24 |
> > We could, in the go-module eclass, point GOMODCACHE to for example |
25 |
> > <distdir>/go-mod. |
26 |
> |
27 |
> So basically every sysadmin running Gentoo will now have to account for |
28 |
> special case of fetching Go sources in addition to fetching generic |
29 |
> Gentoo distfiles. |
30 |
You're focusing a lot on Go here, whereas on IRC I was talking about the |
31 |
general problem space. |
32 |
|
33 |
Yes, Go is the biggest nail sticking out right now, but it's a growing |
34 |
problem overall. |
35 |
- Golang modules |
36 |
- Rust crates |
37 |
- NodeJS modules |
38 |
- Texlive packages |
39 |
|
40 |
|
41 |
Third party systems would be required to provide suitable security on |
42 |
their distfiles. Go & Rust do. I think NodeJS & Tex don't, but I'm happy |
43 |
to be proven wrong. |
44 |
|
45 |
Right now, a sysadmin who wants a disconnected Gentoo environment needs |
46 |
to mirror at least the distfiles in question, or at least the critical |
47 |
files. |
48 |
|
49 |
The *easiest* ways for them to do: |
50 |
- emerge -f & share that distdir |
51 |
- just mirror all of distfiles |
52 |
|
53 |
|
54 |
If they already do "emerge -f", this would be changing to something like |
55 |
"emerge --deep-fetch", that runs: |
56 |
src_fetch & src_unpack & src_fetch_extra |
57 |
|
58 |
src_fetch_extra would be responsible for: fetching, verification, and |
59 |
putting the downloads into $DISTDIR, or subdirectories thereof |
60 |
|
61 |
|
62 |
> > > 4) will eventually lead to ebuilds fetching and using unverified data. |
63 |
> > Not for go at least. |
64 |
> And how are you going to guarantee that this humongous security hole |
65 |
> will not be used wrong for anything else? |
66 |
Unverifiable data, or verification bugs (a long time ago I wrote about a |
67 |
gap in the go.sum checksum mechanism that can sneak in extra files). |
68 |
|
69 |
The distfile security at that point becomes only as strong as the code |
70 |
in src_fetch_extra, or anything that is called. |
71 |
|
72 |
We need to step back and look at the situation holistically. |
73 |
|
74 |
What is the problem statement? |
75 |
I've seen these two so far: |
76 |
- Manifests are getting very big, with duplicated entries between |
77 |
packages. |
78 |
- Environment variables are getting too big and cause errors. |
79 |
|
80 |
But I don't think those capture the entire problem, and I'd really like |
81 |
more clarity on what the actual problem is. |
82 |
|
83 |
|
84 |
Potential families of solutions as I proposed: |
85 |
---------------------------------------------- |
86 |
1) make 1:1 upstream distfile to gentoo distfile official and accepted |
87 |
EVEN if a package has tens of thousands of distfiles. |
88 |
1.1) Provide a way to share duplicated Manifest entries between packages |
89 |
|
90 |
2) make distfile bundling "official", accepting large growth in mirror usage |
91 |
2.1) This would include tooling to make it easy, repeatable, verifiable etc. |
92 |
|
93 |
3) Additional fetch phases, that depend on inputs from src_unpack: this |
94 |
would run things like ecosystem-specific fetch tooling. |
95 |
That tooling would be responsible for the steps I mentioned above, |
96 |
parallel download, verifications, writing to distdir in a way the |
97 |
ecosystem could use |
98 |
|
99 |
|
100 |
-- |
101 |
Robin Hugh Johnson |
102 |
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer |
103 |
E-Mail : robbat2@g.o |
104 |
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 |
105 |
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 |