Re: [gentoo-project] Call for agenda items - Council meeting on 2022-02-13

[Robin H. Johnson]

On Sun, Feb 13, 2022 at 11:03:36AM +0100, Michał Górny wrote:
> > I'll try and paraphrase what I feel the problem statement you're making
> > is:
> > - TL;DR: Golang-package ebuilds account for a disproportionate amount of
> >   Manifest space in the tree.
> > - Golang-package ebuilds account for 55% of all DIST entries in the tree.
> Let's not forget that each ebuild is almost as big as the relevant
> Manifest entries...
>
> > - Even if de-duped, it still be would be 24% of all DIST entries in the
> >   tree.
> ...so even if you dedupe Manifests, ebuilds will still account for major
> space waste.
And this is why I asked that we start by forming a clear problem
statement before.


> > "just repackage it"
> > - Is that in line with the licenses of ALL of distfiles?
> I don't know how that "goproxy" thing works exactly but I suppose if
> they can provide downloads for these packages, then we should be able
> as well.
goproxy contains 99.9% of packages from what I've seen. There are tiny
number that aren't on there.

> > - Is it easy enough for any overlay author to use?
> > - How are you going to trust every overlay's repacking of the distfiles:
> >   that they didn't include malicious code in the distfile, that wouldn't
> >   be caught in a review of the ebuild?
> How are you going to trust anything coming from overlays?  It's not like
> people have to resort to super-complex methods of adding something
> malicious.
Yes, there's already many ways to get stuff in, but repackaging makes
the review much harder.

> > So save your 18MiB of disk space of Manifest lines, or 691MiB.
> Our duty is primarily towards Gentoo users, not mirror admins.  Mirror
> admins have signed up for hosting potentially large distfiles.
>
> Then, there's a major difference between temporarily spend 1G
> on a relatively small number of mirrors, and wasting 18 MiB on *all*
> user systems.  Not to mention git history that's going to be growing
> forever, permanently wasting space on old versions of Go software.
So you want any Gentoo user for these packages to have to download the
368MB each time, even on slow internet.

> > I feel the "re-packaging" is worse than the situation that they're in.
>
> No, it's not.  The situation we're in is causing *permanent* damage,
> and the longer we stall, the worse it gets.
Only permanent to those who use a single git checkout with full history.
There is no permanent impact to users who still consume the tree via
rsync, snapshots, or other point-in-time views, including shallow Git
history.

I have a separate plan to mitigate long-term damage to Git history,
because it's something that WAS discussed back when the move to Git was
started.

> > So how do we find some reasonable solution between this?
> > - Not waste DIST lines space on every copy of the repo
> > - Not cause waste space on Gentoo mirrors
> > - Not cause waste download bandwidth when very little data has changed
> - Not waste space with humongous ebuilds
> - Not consume insane number of inodes on tiny files
So let's go BACK to the proposal of multiple fetch phases.

If we have that as an OPTION, does it solve these 5 concerns, and/or can
it be made close enough to do that?

1. A small number of distfiles, as distributed by the primary upstream,
   no dependency distfiles included.
2. EGO_SUM removed from ebuilds
3. Manifest contains a small number DIST entries per PV.
4. Downloading is still optimized because only new upstream dependencies
   are fetched.
5. Mirrors still optimized because they don't inflate the number of
   files.
6. Zero or small repackaging burden on developers or overlay authors.

How can we get this AND ensure the security you're worried about
previously?


What if there was something to convert go.sum into Manifest-like files
during packaging, and then put THOSE files as DIST.

Manifest-like:
- include gosum checksum
- include BLAKE checksum
- include SRC_URI

You only have to download these Manifest-like files if you want to build the
package.

The Manifest-like files grow linearly with the number of distfiles used,
but the Gentoo tree only has to reference the Manifest.

The packager is responsible for creating the file, specifically the
BLAKE checksums, but their work can be cross-checked by anybody else,
using the gosum checksum (so it's harder for the packager to introduce
malicious behavior; not impossible, just harder, much like today).

This solution would ALSO work for Rust & Texlive, since they provide
some level of integrity & authenticity of those distfiles (as I pointed
out, the node system doesn't seem to provide authenticity).

This probably has further applications where ebuilds themselves can
start to do additional verification of distfiles, like the sec-keys/
category, which also had the potential to greatly expand the tree size.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@g.o
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136