Re: [gentoo-dev] Integrating Portage with other package managers

[Alec Warner]

On Wed, Mar 7, 2018 at 1:22 PM, R0b0t1 <r030t1@×××××.com> wrote:

> On Wed, Mar 7, 2018 at 11:52 AM, Alec Warner <antarus@g.o> wrote:
> > On Wed, Mar 7, 2018 at 11:51 AM, Michael Orlitzky <mjo@g.o>
> wrote:
> >>
> >> On 03/07/2018 11:06 AM, anoteros@××××××.io wrote:
> >> > Why should portage download some outdated second copy of the
> >> > sources for 'bar', rebuild it, and scatter it around the file system
> >> > where it cannot be used by other programs installed by cabal?
> >>
> >
> > I'm really not happy with the tone of this email, so I'm going to
> comment on
> > it a bit.
> >
>
> I can't help but agree with Mr. Orlitzky's sentiment. All language
> package managers suffer from the same sophomoric problems:
>

> 1) I usually don't know where things are downloaded from.
> 2) I can't integrate these changes with my distrbution (Gentoo,
> Ubuntu, Debian, Fedora, CentOS) safely without serious work.
> 3) I can't figure out easily what dependencies a package has. Usually
> I see if there are compile or runtime errors. Sometimes the
> dependencies are listed somewhere. If the dependency is not what is
> currently in e.g. Ubuntu's repository, I may have to maintain separate
> versions to be compatible.
> 4) Sometimes they aren't set up to be built at all. Let the magic
> package manager do everything for you. This works, except when your
> shared objects are not in the right places. (But it makes me feel
> dirty.)
>
> >>
> >> These other package managers don't solve any hard problems -- they're
> >> basically a fancy interface around wget and "git clone." Portage on the
> >> other hand has ~20 years of good ideas and hard work on the hard
> >> problems in package management. For example...
> >
> >
> > Portage is also full of not-good ideas; many of these we papered over
> with
> > PMS and EAPI to make
> > the actual API people use less horrific. Lets not preach from our ivory
> > tower here.
> >
>
> The magnitude of "not good" is, I would suggest, very different.
>
> Cabal is a pretty hilarious example. Have you ever tried to build it
> without using the release binaries? I suppose this is a second problem
> though, where people want to be "self-reliant" and instead just end up
> making things impossible to verify or make reproducible.
>
> For the longest time Cabal did not authenticate or verify the code it
> would run (as root). Very recently this was fixed, but I still feel
> bad any time I let it run, even if it's on a separate development
> system.
>

Gentoo just got signature checking enabled by default...in ~arch? I'm not
sure if that version of portage is stable yet.
Like I said, be careful how one preaches from the ivory tower.


>
> >>
> >> > It seems reasonable to me to 'hook' portage into these other package
> >> > managers, so that running 'emerge bar' would actually run 'cabal
> install
> >> > bar'
> >>
> >> Can "cabal install" build or even identify the C libraries that your
> >> Haskell package needs? No, because nobody ever thought of that, and it
> >> seems kind of hard now that the cabal build system has no ability to
> >> build non-Haskell packages, so no one is ever going to work on it.
> >>
> >> Can "cabal install" rebuild your Haskell packages when the ABI of some
> >> library changes? No, because "cabal install" doesn't have any idea
> >> what's installed on your system.
> >>
> >> Can "cabal install" uninstall a package? Nope, it has no idea what was
> >> done during the installation, and thus no idea what to undo.
> >>
> >> Can "cabal install" verify the integrity of your downloaded source code?
> >> No, because by design it fetches and runs code from complete strangers.
> >>
> >> Can "cabal install" use a local tarball to function without network
> >> access? Etc. We're dead in the water.
> >>
> >> Every other language-specific package manager has the same problems,
> >> because they're all written by people who didn't know anything about
> >> package management and then got bored when they realized that there's
> >> more to it than parsing a json file of dependencies.
> >>
> >
> > They are written by people who are not you, who have different problems
> than
> > you and often don't care about the above use cases.
> > It turns out this stuff exists because:
> >
> > 1) Upstream wants to push 1 single thing and have it work in all distros.
> > 2) pip / virtualenv / cargo / whatever work reasonable well.
> > 3) Rolling-based distros couldn't keep up with packaging.
> > 4) Snapshot-based distros (debian stable / ubuntu) were not designed with
> > this in mind as much; because packages were developed with a high
> velocity.
> >
>
>
This == upstream has a git tag (release X) and they sync the tag into pip
and they tell people to pip install X and their job is done.
Software, distributed.


> 1) What do you mean by this? Distributions are usually not binary
> compatible. This "works" by having each distribution customize and
> build a project by hand.
>

Right, and because "distributions" move a different speeds (Ubuntu, Debian,
Gentoo) and have different branches (testing, stable, unstable, ~arch,
pmask)
this is all even *more* overhead for developers to worry about. So they use
pip, and they don't (worry about it.)



> 2) Virtualenv works well, and cabal now has a local installation
> option. Still, these are not perfect.
> 3) Use a git ebuild, or target stable versions. If a project has so
> much churn that I can't keep up with it I will find something more
> stable.
>

I don't think Ubuntu ships git-ebuilds (because it doesn't have ebuilds) so
I'm stuck consuming some PPA (either one that upstream builds, or some
shady PPA run by who knows who that build my binaries.)

Easier to pip install!


> 4) True, but prefix would be a lot better at fixing that problem. If
> not that, something like virtualenv.
>
> In the case of either #3 or #4, the distribution developers prefer you
> use their package manager to install packages. You are only safe doing
> anything else in an environment like virtualenv, which does not exist
> for the vast majority of languages. This is why developers will pass
> around VM images, or devote an entire VM to development on a project.
> Languages and their packages are not designed to be compartmentalized
> and will trash your installation.


Containers are certainly awesome, yep.


> There is also (going along with #4):
> 5) The software may need to work on Windows. They could have used
> Cygwin or MSYS2. Anything else is an exercise in futility. People have
> come before you and done it better.
>

> Taken together these issues boil down to "the people rewriting package
> managers don't realize the problems they actually have" which agrees
> with what was said. As above, the typical workflows of these people
> involve lots of VMs or containers. Ask yourself "why?"
>

Because containers are awesome and are way easier to use.

-A


>
> >
> >>
> >> If you want to eliminate the duplication of effort, tell these people to
> >> use Gentoo Prefix instead of writing the N+1st crippled PM. Doing it the
> >> other way around won't work because we'd be replacing one good thing
> >> with 75 shitty things.
> >
> >
> > I agree that in theory they could have published ebuilds for Gentoo
> prefix
> > and it would have 'worked everywhere' but I think that boat sailed about
> 10
> > years ago.
> >
> > https://wiki.gentoo.org/wiki/Project:Perl/g-cpan is a project is in a
> > similar space and basically reads perl CPAN metadata to generate stub
> > ebuilds.
> > Portage tracks these stub ebuilds (and so for example, it tracks what
> these
> > cpan packages install and can remove them afterwards.)
> >
> > I think this is the most pragmatic approach I've seen used as its mostly
> an
> > adapter (to cpan) that just generates ebuilds.
> > Its plausible that with some careful eclass magic you might be able to
> make
> > the installed packages compatible with pip, cargo, etc.
> >
> > I think its more of a struggle to make it compatible with things like
> > virtualenv or pip --user though.
> >
>
> This might be a good way to relieve the amount of intervention
> required when repackaging code for an actual package manager. The
> information should be there. The other option is convincing people to
> package for multiple systems at once, which diffuses the effort to the
> point people tend to not mind.
>
> 1) Language package manager (usually used by Windows consumers).
> 2) .debs for Ubuntu/Debian.
> 3) .rpms for Fedora/CentOS.
> 4) Sometimes there's a Gentoo or Arch release.
>
> If you could sell #4 as a way to generate 1-3 it would likely be
> possible to reduce the proliferation of language specific package
> managers over time. Prefix would likely play an important role.
>
> Cheers,
>      R0b0t1
>
>