Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[William Hubbs]

On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote:
> On 06/30/2015 08:35 AM, William Hubbs wrote:
> > All,
> >
> > we have digressed a bit, so I want to bring the discussion back to what
> > my main concerns are about this issue.
> >
> > 1. Should we bundle Go packages with Go software?
> >
> > If we do, except for the Go standard library which is part of
> > dev-lang/go, do we need to bother with installing Go sources and
> > packages at all?
> >
> > The down side of the whole bundling idea is that every consumer
> > on someone's system could potentially have a different version of the
> > Go package, which doesn't lend itself well to security concerns.
> >
> > This is why bundling is generally discouraged in Gentoo.
>
> Yes, as a general rule, bundling is sub-optimal. However, there are
> often exceptions to general rules like these, especially when there are
> competing concerns to contend with.
 
 I don't really see what the competing concerns are in this case.

> > Also, if we bundle, most of dev-go/* doesn't need to exist because these
> > libraries would be bundled into and statically linked into the software
> > that needs them.
>
> Some static libraries are commonly used enough that it might be
> reasonable to install them. Alternatively, we might invent a notion of
> having one ebuild execute another ebuild in order to install static
> dependencies into a temporary build directory.

Why do we need to worry about how many projects use a library? Upstream
has it as a library for good reason, so that multiple projects can use
 it. If upstream installs it as a library, that's how we should install
 it if we install it.

The problem I see with the argument about "commonly used enough" is the
vagueness of it. If we have two packages that use a library, it is
commonly used enough that it should be installed separately.

If we start out bundling libraries, especially libraries from different
upstreams than the package we are working on, that forces all go
maintainers to check all go ebuilds in the tree to see if multiple
bundling is going on and open bugs to create separate ebuilds for
libraries that were only used before by one package but now are used by
more than one.

> > 2. How should we bundle?
> >
> > This is where my concern about consul and some other ebuilds comes in.
> >
> > The way the consul ebuild is written (putting the commit hashes of
> > dependencies in SRC_URI) assumes that all of the dependencies will stay
> > on github. This makes the ebuild far less flexable than go itself is.
>
> Agreed. However, there's no rule which says that we have to force all
> ebuilds to fit into common templates.
 
We do when they deal with common issues; that's the whole point of
language-based eclasses, e.g. ruby* perl* and python*.

> > If we are going to bundle, I would rather have one tarball that includes
> > all of the sources for consul and the dependent libraries dropped on the
> > Gentoo mirrors. Such a tarball is very easy to create.
>
> I would prefer to use separate tarballs for each dependency, preferably
> with the commit hash encoded in the tarball name. This makes the ebuild
> dependencies transparent in the sense that the commit hashes of the
> dependencies are readily available. The "one big tarball" is opaque
> rather than transparent, and it will have a tendency bloat the mirrors.
> By keeping the dependencies in separate tarballs, we can easily do a
> revbump that updates a subset of the dependencies, without having to
> re-pack everything into a big bloated tarball.

I can agree with part of this; one big tarball is less transparent than
multiple tarballs.

Another thing to consider is,  with one big tarball, you can name the
extraction directory to match ${S}, which means there would be no need
for magic in the ebuilds to deal with putting extracted directories in
the right place.

All I'm saying is, if we are going to bundle, lets go all in and not
download multiple upstream packages in src_uri but put them in big
tarballs.

If we are not going to bundle, the best way to handle it is to not
bundle at all imo.

William