[gentoo-dev] Re: better support for binary packages

[Duncan]

Philipp Riegger <lists@××××××××××××.de> posted
1243321504.9661.14.camel@×××××××××××××××××××××××××××××××.de, excerpted
below, on  Tue, 26 May 2009 09:05:03 +0200:

> I don't see the connection between the email Fabio wrote and your
> answer. Do you want to say, that you agree that he's doing what i
> described and that it works the way i described it? I doubt it. If you
> really care, could you answer my first email and state there the
> problems you see with the approach and why you think this is making
> Gentoo worse?

I agree that an independent approach is the way to go.  Gentoo (or 
rather, its PMs, package managers, all three of them, of which portage is 
only one) doesn't do binaries really well, certainly, but I really don't 
see that changing within Gentoo itself, except at the expense of from-
source, which it does quite well indeed.

> But how would you make it worse? It already has the functionality to add
> repositories for binpackages, why not improve it? Why leave it the way
> it is?

Sure, improve it, but what you are talking about isn't a simple 
improvement, but a massive rearchitecting.  That's not easily doable 
without a chance of focus.  It's that change of focus that would 
eventually kill the quality from-source support we have and that 
continues to develop, now.

>> That said, I could envision an eselect like "binary profile" switcher,
>> that subject to settings in a config file, changes USE flags, CFLAGS,
>> gcc- configs an appropriate gcc version, etc, changing PKGDIR
>> appropriately as well, so one could easily enough create as many
>> "binary profiles" as desired and as the use case dictated.

> Not sure, what the binary profile switcher really would change here.
> What I was talking about were mostly USE-flags and packages, and I guess
> your binary profile switcher doesn't change too much, there.

Sure it does, but incrementally.  Basically, your proposal laid out a 
complicated tree based on metadata, a tree the package manager would have 
to understand and support, what I'm proposing is to allow the same thing, 
but not by adding all that complication to the package manager, but 
rather, by using a newly created application to switch among the branches 
of that tree on request.  Portage (or other PM) would use its configured 
PKGDIR and wouldn't understand the tree, but it wouldn't need to, because 
the switcher would manage switching between the branches according to its 
configuration.

The result would be that in a large enough deployment to have build-
servers, the build server(s) could build a particular set of CFLAGS/USE-
flags/gcc-version/arch/whatever, then switch to another branch and build 
that set.  Individual package clients could simply point at the 
appropriate tree and get most of their packages, at least the common 
ones, that way.

Now this wouldn't directly give you the flexibility of the package name 
changes you proposed, but it could do so indirectly, putting that 
information in the directory tree if configured to do so.  Clients may 
need to use the binprofile switcher as well, for individual packages they 
chose to build outside their normal USE profile, but the process could be 
automated once properly configured, using PM hooks as appropriate.

> It would be ok to do all this stuff in an extra project, without Gentoo.

The proposal above wouldn't be without Gentoo.  It'd simply be a package 
level thing rather than a distribution level thing.  But either that or 
the independent distribution based on Gentoo route that lxnay has taken, 
either one works, without defocusing Gentoo on its meta-distrib-wide from-
source vision.

For that matter, Gentoo already has three package managers, and binary 
support (or the lack thereof) has been deliberately left up to the 
package manager at this point -- it's NOT part of PMS.  It'd be equally 
possible to either take one of the current PMs and add your enhanced 
binary package scheme to it, or to start an independent forth package 
manager, and have it focus on binaries rather than from-source.  (It 
could even take the existing portage binpkgs and rename or otherwise 
manage them as necessary, thereby avoiding the from-source side entirely, 
if so desired.)

> Well, the last is a little bit impossible outside of Gentoo
> (I don't want to fork the tree) and I also don't want to fork portage,
> so the first is complicated, too.

You mention portage, but don't mention the other PMs at all.  As 
mentioned, binary packages aren't part of PMS (the Package Management 
Specification, the app-doc/pms package, the standard that allows PMs 
besides portage, currently, paludis and pkgcore) at all.  Gentoo really 
doesn't have an official binary package format at all (see PMS, Appendix 
B, Unspecified Items), only individual package managers like portage do, 
and at present they may conflict with each other.

That's both a good and a bad thing.  As it's not specified, you're free 
to define it as necessary to meet your needs for any tool you devise to 
handle binary packages.  Of course, that means that it's just that tool's 
definition, and at least one package manager, portage, does happen to 
have binary packages and a working format definition for them.

But that leaves you with maximum flexibility in creating whatever you 
want, either as a separate tool (or even independent Gentoo based 
distribution), or as an expanded PM, either forked from one of the 
current ones, or created independently.
 
> And all this layer thing Fabio was talking about. I did not try it and I
> did not read the code, but I think it makes things much more
> complicated. See also the discussion about mixing package managers
> between Gentoo and Sabayon. I do not want these problems.

But your proposal adds the complexity that would bring these problems, 
whether you want them or not.  Either way, the technical problems can be 
worked thru, it's simply a matter of coding for the most part, after all, 
but the problems are there and must be dealt with either way.  You prefer 
to have Gentoo do it as a whole.  I say no, leave Gentoo well enough 
alone in that regard, and let individual projects, either as Gentoo-based 
independent distributions, or at the application (either PM or switcher 
level) deal with the problems.  After all, Gentoo's a big organization 
and getting all of it together addressing a problem is a VERY hard task 
as I'm sure CiaranM among others can vouch for.  A much smaller PM or 
switcher app project is by definition of the smaller size, also much 
easier to steer and to get things actually done in.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman