Re: [gentoo-project] Call for agenda items - Council meeting on 2022-02-13

[Michał Górny]

On Fri, 2022-02-11 at 00:53 -0600, William Hubbs wrote:
> On Thu, Feb 10, 2022 at 07:27:15PM +0100, Andreas K. Huettel wrote:
> > > I just realized a down side to this approach, but I don't know how
> > > important that down side is.
> > >
> > > As the example shows, if a package has a vendor directory we would trust
> > > it so nothing would need to be done. If we have to run "go mod
> > > download", however, this populates the dependency cache from the
> > > internet. This means that a package that requires this could not be
> > > built if the building machine is not on the net.
> >
> > (hands William a golden potato)
> >
> > Congratulations, you've just rediscovered why we do the things the way
> > we're doing them now...
>
> And, you dropped my question at the end of this message,
> so let me bring it back.
>
> > I don't know how important this is, because it is true for any go
> > package without a vendor directory no matter how you build it.
>
> This could also apply to a rust or nodejs package.
>
> The build process for all of these languages has the potential to reach
> out to the net to download dependencies.
>
> Iam talking mostly about Go, because that's what I work with, but I know
> it isn't the only language that does this.
>
> In the list of solutions Robin put on this thread, I prefer the third
> group. That would involve the fetch_extra phase which would for go
> basically be "cd ${S} || die; go mod download || die" along with an
> environment variable that would put modules under distfiles.
>
> If you have an idea for a better solution, I'm willing to hear it.
>

The "better solution" has already been provided -- repackage it.
All of your solutions solve only part of the problem, and usually
introduce more problems.  The current distfile fetching solution works
well for over 99% of the Gentoo packages today (this is not "the 99%"
but actual numbers, more below).  The problem is with Go, and Go needs
to be fixed (or worked around), and don't extend it to "theoretically
Rust or NodeJS" because they're nowhere close to how badly designed Go
ecosystem is.

No solution based on extended fetch phases is going to work.  However
complex you're going to invent it, it doesn't change that you end up
with live ebuilds, and as such they're never going to have KEYWORDS. 
All the added complexity to work arounds the usual problems with live
ebuilds isn't going to change that, not to mention being worth
the effort specifically for Go.

No change of rules on exported variables makes the problem disappear
either.  The problem of "PM can't handle packages with humongous number
of tiny files" is not worth addressing.  The problem that we need to
address today is "Go packages are *harmful* through using over 54000
tiny .mod distfiles".  This problem can be solved via repackaging,
and rather than wasting our times discussing bad solutions to the wrong
problem, we should be repackaging these packages *now*.

Now, for some numbers.  According to my greps (please correct me if I've
gotten this wrong), there 121 packages using go .mod files in ::gentoo.
This is 0.6% of all packages.  These 121 packages make up 44.7% of total
Manifest lines, or 46.6% total Manifest bytes (probably thanks to long
filenames).  This is 18 MiB of DIST lines that unconditionally waste
user's disk space and cause git history to grow larger with every
version bump.  And the worst part is, if I understand the process
correctly, every version bump will bring more and more .mod files
for old versions of packages that aren't even really used to build
the package!

Again, let me repeat.  We are talking about inventing super complex
solutions to make 121 bad packages suddenly look good.  Why isn't
anybody discussing fixing these packages instead?  Why did people think
it's a good idea to commit them as-is?

-- 
Best regards,
Michał Górny