Re: [gentoo-dev] Proposal to undeprecate EGO_SUM

[William Hubbs]

On Fri, Sep 30, 2022 at 10:07:44PM +0200, Arsen Arsenović wrote:
> Hey,
>
> On Friday, 30 September 2022 02:36:05 CEST William Hubbs wrote:
> > I don't know for certain about a vendor tarball, but I do know there
> > are instances where a vendor tarball wouldn't work.
> > app-containers/containerd is a good example of this, That is why the
> > vendor tarball idea was dropped.
> It is indeed not possible to verify vendor tarballs[1].  The proposed
> solution Go people had would also require network access.
>
> > Upstream doesn't need to provide a tarball, just an up-to-date
> > "vendor" directory at the top level of the project. Two examples that
> > do this are docker and kubernetes.
> Upstreams doing this sounds like a mess, because then they'd have to
> maintain multiple source trees in their repositories, if I understand
> what you mean.

Well, there isn't a lot of work involved in this for upstream, they just
run:

$ go mod vendor

at the top level of their project and keep that directory in sync in
their vcs. The down side is it can be big and some upstreams do not want
to do it.

>
> An alternative to vendor tarballs is modcache tarballs. These are
> absolutely massive (~20 times larger IIRC), though, they are verifiable.

The modcache tarballs are what I'm calling dependency tarballs, and yes
they are bigger than vendor tarballs and verifiable.
Also, the go-module eclass sets the GOMODCACHE environment variable to
point to the directory where the contents of the dependency tarball ends
up which makes it easy for the go tooling to just use the information in
that directory.

If we can get bug https://bugs.gentoo.org/833567 to happen in eapi 9,
that would solve all of this.

The next step after I got that to happen would be to put a shared go
module cache in, for example, "${DISTDIR}/go-mod", so that all go
modules from packages would be downloaded there, and they would be
consumed like all distfiles are.

> opinion: I see no way around it. Vendor tarballs are the way to go.  For
> trivial cases, this can likely be EGO_SUM, but it scales exceedingly
> poorly, to the point of the trivial case being a very small percentage
> of Go packages.  I proposed authenticated automation on Gentoo
> infrastructure as a solution to this, and implemented (a slow and
> unreliable) proof of concept (posted previously).  The obvious question
> of "how will proxy maintainers deal with this" is also relatively
> simple: giving them authorization for a subset of packages that they'd
> need to work on. This is an obvious increase in the barrier of entry for
> fresh proxy maintainers, but it's still likely less than needing
> maintainers to rework ebuilds to use vendor tarballs on dev.g.o.

Vendor tarballs are not complete.  The best example of this I see in the tree is
app-containers/containerd.  If you try to build that with a vendor tarball
instead of a dependency tarball, the build will break, but it works with
a dependency tarball.

William


>
>
> [1]: https://github.com/golang/go/issues/27348
> --
> Arsen Arsenović