Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources

[Zac Medico]

On 06/30/2015 01:30 PM, William Hubbs wrote:
> On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote:
>> On 06/30/2015 08:35 AM, William Hubbs wrote:
>>> All,
>>>
>>> we have digressed a bit, so I want to bring the discussion back to what
>>> my main concerns are about this issue.
>>>
>>> 1. Should we bundle Go packages with Go software?
>>>
>>> If we do, except for the Go standard library which is part of
>>> dev-lang/go, do we need to bother with installing Go sources and
>>> packages at all?
>>>
>>> The down side of the whole bundling idea is that every consumer
>>> on someone's system could potentially have a different version of the
>>> Go package, which doesn't lend itself well to security concerns.
>>>
>>> This is why bundling is generally discouraged in Gentoo.
>>
>> Yes, as a general rule, bundling is sub-optimal. However, there are
>> often exceptions to general rules like these, especially when there are
>> competing concerns to contend with.
>
>  I don't really see what the competing concerns are in this case.

The competing concern is that un-bundling has some possibly undesirable
consequences, mainly that it means we'll be installing static libraries
that were only intended to be temporary build artifacts. It makes sense
to install them if there are multiple consumers, otherwise it doesn't
make much sense.

>>> Also, if we bundle, most of dev-go/* doesn't need to exist because these
>>> libraries would be bundled into and statically linked into the software
>>> that needs them.
>>
>> Some static libraries are commonly used enough that it might be
>> reasonable to install them. Alternatively, we might invent a notion of
>> having one ebuild execute another ebuild in order to install static
>> dependencies into a temporary build directory.
>
> Why do we need to worry about how many projects use a library?

If you want to clutter the tree with trivial ebuilds that only have a
single consumer, then I guess that's fine. It's not clear to me that
this is the best course of action, but I'm not going to try to stop you
if that's what you want to do.

>  Upstream
> has it as a library for good reason, so that multiple projects can use
>  it. If upstream installs it as a library, that's how we should install
>  it if we install it.

I don't think that consul upstream "installs it as a library." The last
time that I checked upstream's "build from source" instructions, the
consul binary was the only result of interest, so all of the temporary
build artifacts could simply be discarded after the consul binary had
been built.

> The problem I see with the argument about "commonly used enough" is the
> vagueness of it. If we have two packages that use a library, it is
> commonly used enough that it should be installed separately.

As soon as you have at least two consumers, then you have a good reason
to un-bundle a library. If there's only one consumer, then un-bundling
becomes questionable.

> If we start out bundling libraries, especially libraries from different
> upstreams than the package we are working on, that forces all go
> maintainers to check all go ebuilds in the tree to see if multiple
> bundling is going on and open bugs to create separate ebuilds for
> libraries that were only used before by one package but now are used by
> more than one.

Nobody is being forced to do anything. If a maintainer of a Go package
than bundles libraries is doing a version bump, then it would be a good
time for him to check if ebuilds have been created for any of those
bundled dependencies, and un-bundle them at that point.

>>> 2. How should we bundle?
>>>
>>> This is where my concern about consul and some other ebuilds comes in.
>>>
>>> The way the consul ebuild is written (putting the commit hashes of
>>> dependencies in SRC_URI) assumes that all of the dependencies will stay
>>> on github. This makes the ebuild far less flexable than go itself is.
>>
>> Agreed. However, there's no rule which says that we have to force all
>> ebuilds to fit into common templates.
>
> We do when they deal with common issues; that's the whole point of
> language-based eclasses, e.g. ruby* perl* and python*.

There can always be outliers that don't fit your existing templates.

>>> If we are going to bundle, I would rather have one tarball that includes
>>> all of the sources for consul and the dependent libraries dropped on the
>>> Gentoo mirrors. Such a tarball is very easy to create.
>>
>> I would prefer to use separate tarballs for each dependency, preferably
>> with the commit hash encoded in the tarball name. This makes the ebuild
>> dependencies transparent in the sense that the commit hashes of the
>> dependencies are readily available. The "one big tarball" is opaque
>> rather than transparent, and it will have a tendency bloat the mirrors.
>> By keeping the dependencies in separate tarballs, we can easily do a
>> revbump that updates a subset of the dependencies, without having to
>> re-pack everything into a big bloated tarball.
>
> I can agree with part of this; one big tarball is less transparent than
> multiple tarballs.
>
> Another thing to consider is,  with one big tarball, you can name the
> extraction directory to match ${S}, which means there would be no need
> for magic in the ebuilds to deal with putting extracted directories in
> the right place.

I think this so-called "magic" is a small price to pay for transparency,
and the ease with which one can do a revision bump while updating a
subset of the bundled dependencies (all the tarballs are automatically
generated by github, so I don't have to pack any of them myself).

> All I'm saying is, if we are going to bundle, lets go all in and not
> download multiple upstream packages in src_uri but put them in big
> tarballs.

I would prefer to create separate ebuilds for all of the dependencies,
than be forced to do this.

> If we are not going to bundle, the best way to handle it is to not
> bundle at all imo.

Even if there is only a single consumer of said libraries?
-- 
Thanks,
Zac