Re: [gentoo-dev] Re: better support for binary packages

[Philipp Riegger]

On Tue, 2009-05-26 at 08:48 +0000, Duncan wrote:
> Philipp Riegger <lists@××××××××××××.de> posted
> 1243321504.9661.14.camel@×××××××××××××××××××××××××××××××.de, excerpted
> below, on  Tue, 26 May 2009 09:05:03 +0200:
>
> > I don't see the connection between the email Fabio wrote and your
> > answer. Do you want to say, that you agree that he's doing what i
> > described and that it works the way i described it? I doubt it. If you
> > really care, could you answer my first email and state there the
> > problems you see with the approach and why you think this is making
> > Gentoo worse?
>
> I agree that an independent approach is the way to go.  Gentoo (or
> rather, its PMs, package managers, all three of them, of which portage is
> only one) doesn't do binaries really well, certainly, but I really don't
> see that changing within Gentoo itself, except at the expense of from-
> source, which it does quite well indeed.

Again, I see this as completely independent "features". Binary packages
would only support the building from source and binary packages could
never be created without ebuilds. This might even improve the quality of
the ebuilds in the tree, because they would get built from time to time,
build failures would get reported and all that.

> > But how would you make it worse? It already has the functionality to add
> > repositories for binpackages, why not improve it? Why leave it the way
> > it is?
>
> Sure, improve it, but what you are talking about isn't a simple
> improvement, but a massive rearchitecting.  That's not easily doable
> without a chance of focus.  It's that change of focus that would
> eventually kill the quality from-source support we have and that
> continues to develop, now.

Are you sure? How code would it take to look for a binary package in a
different path than now? 1 additional line or 1 line exchanged? How much
code would it take to create this packed bit vector out of the
USE-flags? 5-10 lines? Changing portage to _also_ support the new kind
of binary packages might be done in 1 day. And this is the first step.

> >> That said, I could envision an eselect like "binary profile" switcher,
> >> that subject to settings in a config file, changes USE flags, CFLAGS,
> >> gcc- configs an appropriate gcc version, etc, changing PKGDIR
> >> appropriately as well, so one could easily enough create as many
> >> "binary profiles" as desired and as the use case dictated.
>
> > Not sure, what the binary profile switcher really would change here.
> > What I was talking about were mostly USE-flags and packages, and I guess
> > your binary profile switcher doesn't change too much, there.
>
> Sure it does, but incrementally.  Basically, your proposal laid out a
> complicated tree based on metadata, a tree the package manager would have
> to understand and support, what I'm proposing is to allow the same thing,
> but not by adding all that complication to the package manager, but
> rather, by using a newly created application to switch among the branches
> of that tree on request.

Which might work, but would lead to a really strange package tree with
really big restrictions and disadvantages (no package sharing, no way of
finding out if an update is necessary,...).

> Portage (or other PM) would use its configured
> PKGDIR and wouldn't understand the tree, but it wouldn't need to, because
> the switcher would manage switching between the branches according to its
> configuration.

And there the flexibility goes. If you want to emerge PHP and there is a
binary package with the same USE-flage but also cli enabled or something
like that, it would really get complicated to inform you (the user) that
a simple and probably not too important change for you might save you an
hour of compilation.

> The result would be that in a large enough deployment to have build-
> servers, the build server(s) could build a particular set of CFLAGS/USE-
> flags/gcc-version/arch/whatever, then switch to another branch and build
> that set.  Individual package clients could simply point at the
> appropriate tree and get most of their packages, at least the common
> ones, that way.
>
> Now this wouldn't directly give you the flexibility of the package name
> changes you proposed, but it could do so indirectly, putting that
> information in the directory tree if configured to do so.  Clients may
> need to use the binprofile switcher as well, for individual packages they
> chose to build outside their normal USE profile, but the process could be
> automated once properly configured, using PM hooks as appropriate.

And as soon as you use PM hooks, you ask yourself, why you did it that
way, if not that change had made it easier, if it would have been better
to discuss the issue with portage/pkgcore/paludis developers and all
that.

Since I really like PM hooks, I'm not sure if they are really helpfull
for users. They are too cryptic. Maybe it would be wise to create some
eselect like thing for those hooks provide some basic scripts with that.


> For that matter, Gentoo already has three package managers, and binary
> support (or the lack thereof) has been deliberately left up to the
> package manager at this point -- it's NOT part of PMS.  It'd be equally
> possible to either take one of the current PMs and add your enhanced
> binary package scheme to it, or to start an independent forth package
> manager, and have it focus on binaries rather than from-source.  (It
> could even take the existing portage binpkgs and rename or otherwise
> manage them as necessary, thereby avoiding the from-source side entirely,
> if so desired.)
>
> > Well, the last is a little bit impossible outside of Gentoo
> > (I don't want to fork the tree) and I also don't want to fork portage,
> > so the first is complicated, too.
>
> You mention portage, but don't mention the other PMs at all.  As
> mentioned, binary packages aren't part of PMS (the Package Management
> Specification, the app-doc/pms package, the standard that allows PMs
> besides portage, currently, paludis and pkgcore) at all.  Gentoo really
> doesn't have an official binary package format at all (see PMS, Appendix
> B, Unspecified Items), only individual package managers like portage do,
> and at present they may conflict with each other.
>
> That's both a good and a bad thing.  As it's not specified, you're free
> to define it as necessary to meet your needs for any tool you devise to
> handle binary packages.  Of course, that means that it's just that tool's
> definition, and at least one package manager, portage, does happen to
> have binary packages and a working format definition for them.
>
> But that leaves you with maximum flexibility in creating whatever you
> want, either as a separate tool (or even independent Gentoo based
> distribution), or as an expanded PM, either forked from one of the
> current ones, or created independently.

Now, while flexibility migth be interesting for developers and power
users, I don't think it's always the right thing for users.

Why not talk about binary packages, work on a standard and write some
draft for PMS or another document (not like "you need this", but like
the special annexes from ada for example which leave it open to be
implemented or not). Now we have sabayon. We have the portage binary
packages. How many more do we need until we realize, that it only
duplicates work?

> > And all this layer thing Fabio was talking about. I did not try it and I
> > did not read the code, but I think it makes things much more
> > complicated. See also the discussion about mixing package managers
> > between Gentoo and Sabayon. I do not want these problems.
>
> But your proposal adds the complexity that would bring these problems,
> whether you want them or not.  Either way, the technical problems can be
> worked thru, it's simply a matter of coding for the most part, after all,
> but the problems are there and must be dealt with either way.  You prefer
> to have Gentoo do it as a whole.  I say no, leave Gentoo well enough
> alone in that regard, and let individual projects, either as Gentoo-based
> independent distributions, or at the application (either PM or switcher
> level) deal with the problems.  After all, Gentoo's a big organization
> and getting all of it together addressing a problem is a VERY hard task
> as I'm sure CiaranM among others can vouch for.  A much smaller PM or
> switcher app project is by definition of the smaller size, also much
> easier to steer and to get things actually done in.

While it might be difficult, it might just be worth it.

Bit it seems to be quite an uninteresting topic, since the people most
affected by it (Gentoo developers) did not join the conversation, yet.
Maybe I should take this to gentoo-server@ and gentoo-portage@, it might
fit there.

Philipp