Re: [gentoo-project] Call for agenda items - Council meeting on 2022-02-13

[Michał Górny]

On Sun, 2022-02-13 at 07:37 +0000, Robin H. Johnson wrote:
> On Sat, Feb 12, 2022 at 10:58:27PM +0100, Michał Górny wrote:
> > The "better solution" has already been provided -- repackage it.
> I had asked people to clarify the problems, and you still didn't
> actually provide a clear problem statement.
>
> I'll try and paraphrase what I feel the problem statement you're making
> is:
> - TL;DR: Golang-package ebuilds account for a disproportionate amount of
>   Manifest space in the tree.
> - Golang-package ebuilds account for 55% of all DIST entries in the tree.

Let's not forget that each ebuild is almost as big as the relevant
Manifest entries...

> - Even if de-duped, it still be would be 24% of all DIST entries in the
>   tree.

...so even if you dedupe Manifests, ebuilds will still account for major
space waste.

> - Rust is 8% baseline, 6% if de-duped.
>
> > All of your solutions solve only part of the problem, and usually
> > introduce more problems.  The current distfile fetching solution works
> > well for over 99% of the Gentoo packages today (this is not "the 99%"
> > but actual numbers, more below).  The problem is with Go, and Go needs
> > to be fixed (or worked around), and don't extend it to "theoretically
> > Rust or NodeJS" because they're nowhere close to how badly designed Go
> > ecosystem is.
>
> "just repackage it"
> - Is that in line with the licenses of ALL of distfiles?

I don't know how that "goproxy" thing works exactly but I suppose if
they can provide downloads for these packages, then we should be able
as well.

> - If the combined gentoo-specific distfile is 100MB per $PV, you're now
>   forcing 100MB of new downloads for each version.

Lesser evil.

> - Is it easy enough for any overlay author to use?
> - How are you going to trust every overlay's repacking of the distfiles:
>   that they didn't include malicious code in the distfile, that wouldn't
>   be caught in a review of the ebuild?

How are you going to trust anything coming from overlays?  It's not like
people have to resort to super-complex methods of adding something
malicious.

>
> I haven't done a full analysis of how much space it would end up take,
>
> But as a quick version, let's consider Vault 1.8.x series.
>
> vault-1.8.6 -> vault-1.8.7 didn't change distfiles.
> vault-1.8.7 -> vault-1.8.8
>
> 1.8.6 -> 368MB worth of distfiles
> of which 22-24MB are different from 1.8.7 & 1.8.8
>
> So naively, if you re-packaged those 3 versions, 368MiBx3 => 1104MiB on
> the mirrors.
>
> Or just 413MiB of unique distfiles between the versions.
>
> So save your 18MiB of disk space of Manifest lines, or 691MiB.

Our duty is primarily towards Gentoo users, not mirror admins.  Mirror
admins have signed up for hosting potentially large distfiles.

Then, there's a major difference between temporarily spend 1G
on a relatively small number of mirrors, and wasting 18 MiB on *all*
user systems.  Not to mention git history that's going to be growing
forever, permanently wasting space on old versions of Go software.

> I feel the "re-packaging" is worse than the situation that they're in.

No, it's not.  The situation we're in is causing *permanent* damage,
and the longer we stall, the worse it gets.

> So how do we find some reasonable solution between this?
> - Not waste DIST lines space on every copy of the repo
> - Not cause waste space on Gentoo mirrors
> - Not cause waste download bandwidth when very little data has changed

- Not waste space with humongous ebuilds
- Not consume insane number of inodes on tiny files

-- 
Best regards,
Michał Górny