Re: [gentoo-project] Call for agenda items - Council meeting on 2022-02-13

[Robin H. Johnson]

On Sat, Feb 12, 2022 at 10:58:27PM +0100, Michał Górny wrote:
> The "better solution" has already been provided -- repackage it.
I had asked people to clarify the problems, and you still didn't
actually provide a clear problem statement.

I'll try and paraphrase what I feel the problem statement you're making
is:
- TL;DR: Golang-package ebuilds account for a disproportionate amount of
  Manifest space in the tree.
- Golang-package ebuilds account for 55% of all DIST entries in the tree.
- Even if de-duped, it still be would be 24% of all DIST entries in the
  tree.
- Rust is 8% baseline, 6% if de-duped.

> All of your solutions solve only part of the problem, and usually
> introduce more problems.  The current distfile fetching solution works
> well for over 99% of the Gentoo packages today (this is not "the 99%"
> but actual numbers, more below).  The problem is with Go, and Go needs
> to be fixed (or worked around), and don't extend it to "theoretically
> Rust or NodeJS" because they're nowhere close to how badly designed Go
> ecosystem is.

"just repackage it"
- Is that in line with the licenses of ALL of distfiles?
- If the combined gentoo-specific distfile is 100MB per $PV, you're now
  forcing 100MB of new downloads for each version.
- Is it easy enough for any overlay author to use?
- How are you going to trust every overlay's repacking of the distfiles:
  that they didn't include malicious code in the distfile, that wouldn't
  be caught in a review of the ebuild?

I haven't done a full analysis of how much space it would end up take, 

But as a quick version, let's consider Vault 1.8.x series.

vault-1.8.6 -> vault-1.8.7 didn't change distfiles.
vault-1.8.7 -> vault-1.8.8

1.8.6 -> 368MB worth of distfiles
of which 22-24MB are different from 1.8.7 & 1.8.8

So naively, if you re-packaged those 3 versions, 368MiBx3 => 1104MiB on
the mirrors.

Or just 413MiB of unique distfiles between the versions.

So save your 18MiB of disk space of Manifest lines, or 691MiB.

> Now, for some numbers.  According to my greps (please correct me if I've
> gotten this wrong), there 121 packages using go .mod files in ::gentoo.
> This is 0.6% of all packages.  These 121 packages make up 44.7% of total
> Manifest lines, or 46.6% total Manifest bytes (probably thanks to long
> filenames).  This is 18 MiB of DIST lines that unconditionally waste
> user's disk space and cause git history to grow larger with every
> version bump.  And the worst part is, if I understand the process
> correctly, every version bump will bring more and more .mod files
> for old versions of packages that aren't even really used to build
> the package!
The takeaway here is that we need a much better way to handle DIST lines
that are duplicated between packages. If we could achieve perfect
de-duplication for DIST lines, the 66591 entries for Golang stuff,
taking 22795192 bytes, becomes 13887 entries, with only 4798540 bytes.

The .mod entries are 8402 of those, for 2894495 bytes; my original
proposal of the Golang stuff did propose a way to remove all .mod
entries: at the cost of always requiring the .zip file downloaded
instead, when the .mod file alone would suffice for some packages.

Non-golang DIST entries account for 54042 entries, 16800665 bytes; after
de-dupe that is 43792 entries 13608721.

Golang upstream packaging absolutely needs to clean up the proliferation
of go.sum bringing in every .mod file from the past, and making it
easier for every module author to keep it clean.

> Again, let me repeat.  We are talking about inventing super complex
> solutions to make 121 bad packages suddenly look good.  Why isn't
> anybody discussing fixing these packages instead?  Why did people think
> it's a good idea to commit them as-is?
I feel the "re-packaging" is worse than the situation that they're in.

So how do we find some reasonable solution between this?
- Not waste DIST lines space on every copy of the repo
- Not cause waste space on Gentoo mirrors
- Not cause waste download bandwidth when very little data has changed

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@g.o
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136