Re: [gentoo-dev] network sandbox challenge

[Alec Warner]

On Wed, Apr 1, 2020 at 5:14 AM Samuel Bernardo <
samuelbernardo.mail@×××××.com> wrote:

> Hi Robin,
> On 4/1/20 6:36 AM, Robin H. Johnson wrote:
>
> Normally we don't bundle dependencies, avoiding that problem entirely.
> The Go eclasses however are badly designed, committed against protest by
> paid corporate interests, and serve only to facilitate large-scale
> copyright infringement and security vulnerabilities. If you're looking
> for a consistent explanation of how they're supposed to work with the
> rest of Gentoo, you won't find one.
>
> mjo: Can you please substantiate your claims?
>
> It would have been nice to have heard your concerns during February, any
> of one the three times that William and I posted the go-module.eclass
> EGO_SUM development work for review on this mailing list. I don't see a
> single email from you during that entire period.
>
> The EGO_SUM support explicitly ensured that upstream distfiles (for each
> dependency) remained absolutely as upstream provided them, without
> merging the distfiles together or altering their content in way (I admit
> that the exact naming of the distfiles changed, because it was terrible,
> v0.0.0-20190311183353-d8887717615a.zip for example).
>
> Forgive my noobishness in this matter that let Alec to comment over my own
> statement.
>
> Alec pointed out some very important issues in go development that break
> copyright infringement and security vulnerabilities, but I'm sure that is
> not related to the good work done in go-module.eclass to surpass all go
> mess. npm is worst and I take from go-module as a good pattern to apply
> also into there.
>
I am antarus, not mjo (but more on that below!) I don't believe bundling
presents many challenges with regards to copyright infringement. As a
package maintainer you should know the licenses used in your packages. You
are required to reflect any licenses used in the LICENSE ebuild variable.
Obviously this becomes more work if you are using a bundle due to the fact
that bundling will include more code. In the golang ecosystem there is a
tool to help maintainers do this (
https://packages.gentoo.org/packages/dev-go/golicense). I get that with
bundling we cannot share the work from previous packages because packages
are not shared in a bundled environment but I expect the golicense tool to
have good coverage in practice. If the tool does the work, sharing the work
becomes moot.

I think licensing can be more challenging in other bundling scenarios where
tooling is not provided; but note that this is not significantly different
from the unbundled scenario in terms of license discovery. If I am
packaging a new program (A) and it depends on (B,C,D) I have two options. I
can either package [A,B,C,D] (normal gentoo way) or I can package [A] (with
B,C,D bundled). The intersection of the LICENSE variables is the same
effort for both here. The benefit of the multiple packages is that future
users of B,C,D can re-use the license discovery work and that isn't nothing.

> Going back to my overlay use case, will go-modules download all modules to
> distfiles directory? The naming convention will assure that there will be
> no modules repetition?
>
What about eclean-dist, will it work as expected for those modules
> dependencies?
>
> I think some of this answers would worth mention in documentation.
>
> Sorry for anything I wrongly stated and thank you very much for your help,
>
> Samuel
>
I've chosen this part to write my treatise on packaging, but rest assured
it's mostly intended as a response to mgorny and mjo; not specifically in
response to you.

The very long answer is that Gentoo was designed around a paradigm of
programs written primarily in C. In C programs you have the ability to link
to libraries which offer APIs and in the ideal case, each API is offered
via a unique SONAME[0]. Upstream packages were written and built in this
way (with dynamic linking). So in the case of package A, that uses
libraries B, C, and D; the result in many distributions is 4 packages
(A,B,C,D) and users who want A will get B, C, and D installed. This in fact
was a major selling point of package managers at the time because finding
these dependencies by hand and building and merging them all was painful.

Many applications break this trend; I don't think golang or nodejs are
particularly new (python and ruby have had (pip, venv) and rubygems[1] for
years, for example, which are similar bundling paradigms.) The struggle as
packagers and distribution managers is when upstream decides "my software
should be installed via a bundling solution (golang, node, pip, rubygems,
and so on)" we are left to decide both whether to map this to the ebuild
paradigm (no bundling of dependencies) or omit ebuilds entirely. In the
former case we are often left working at odds with upstream (who are
confused by our decomposition of their application) and in the latter case,
users often use the bundle anyway (e.g. they install the packages by hand
or use the ruby gems or whatever.) I assert this is somewhat of a false
choice. Bundling isn't all bad and we can learn from past mistakes[2] to
try to avoid problems.

Another challenge with bundling is that often bundling systems (bundler,
pip, venv, golang, etc.) specify specific versions, commits, or tags. This
is fine when bundling (because each bundle has its own version of a
dependency in the bundle) but when you are trying to share a system wide
package between N packages, you either need to SLOT the dependencies or
have a looser dependency specification. The fine-grained nature of the
upstream dependency specification can make this challenging[3].

Unbundling then made it easier for system operators to operate a system;
and you see this often in the security space. A security notice will come
out saying "foo-X-Y-Z is vulnerable, move to foo-Y.1." So operators want to
know "do I have foo-x-y-z installed?" When every package is in the package
manager this is a trivial question. When software is bundled inside of a
package, this visibility is lost. I haven't seen any tooling for Gentoo to
this problem.

In addition to the above, bundling can present exciting resource challenges
for some deployments. Imagine a common dep (CommonFoo-x-y-z) has a security
problem, so we must upgrade to CommonFoo-y-z. In the scenario where
CommonFoo is a dynamically linked package we can recompile it once[4] and
new consumers will just use the new dynamic shared object. In a bundling
scenario, we will be forced to rebuild[5] all consumers. This can take a
lot of time and resources depending on the deployment. Is the deployment
using a build farm? A binary packages host? How many disparate platforms
are in use?

Which is to say many people in Gentoo dislike bundling for various reasons;
many of them legitimate. I wish to present a narrative where bundling is an
engineering trade-off, rather than a decision that is settled engineering
law. This doesn't mean Gentoo needs to support all the bundling (clearly
most people don't want to) but not supporting it means that many packages
will not be in Gentoo at all (because unbundling is too costly) and so you
end up at this exciting discussion which happens every couple of years.

-A

[0] I understand this is not always true in practice, but let's assume
spherical cows momentarily.
[1] Gentoo has a rubygems-fakegem eclass that makes it pretty streamlined
to make an ebuild for a particular gem, but of course if my application
depends on 20 gems I still need to make 20 ebuilds in this scheme and merge
them all. Rubygems-fakegem is still pretty good though!
[2] On windows https://en.wikipedia.org/wiki/DLL_Hell was common. In the
.NET ecosystems assemblies addressed some of these problems.
[3] Similar to DLL hell but more generically:
https://en.wikipedia.org/wiki/Dependency_hell
[4] Practice of course, leads to all kinds of weird edge cases where
upgrading your shared lib causes dependencies to break for various reasons;
which is one reason why application authors like to bundle; because their
application ends up being perceived as more reliable and less finicky.
[5] The number of package rebuilds in Gentoo is a fairly common complaint,
from my personal observation. Obviously binary packages make this problem
worse (not better.) I dunno if its something the community should put more
effort into or not though; my expectation is that rebuilds are common and
making them more common is not a strategic problem; but I'm also not
compiling on some single core atom, so what do I know, eh? :)