Re: [gentoo-dev] [RFC] Multiple ABI support through package appending/partial removal

[Brian Harring]

On Mon, Sep 24, 2012 at 12:09:49AM +0200, Micha?? G??rny wrote:
> Hello,
>
> Since my previous idea of DYNAMIC_SLOTS proved too complex to design
> and implement, I would like to offer an another idea, based partially
> on what Ciaran mentioned. Before I start getting into details, I'd like
> to know your opinions, and what possible problems am I missing. To keep
> it clean, I will focus on Python ABIs but other languages and multilib
> could be handled in a similar manner.
>
>
> The problem
> ===========
>
> Right now, building packages for multiple Python ABIs is done using
> USE_EXPAND-based useflags. This is a working solution but it requires
> rebuilding the package for all ABIs whenever the chosen ABI list
> changes.
>
> While it may be not that important for most of the Python packages, it
> becomes such when it comes to things like boost or -- if we'd extend
> that to multilib -- say, llvm. In that case, whenever a newly-installed
> package requests a specific ABI, user has to spend twice as much time
> to rebuild the same version.
>
>
> The general idea
> ================
>
> While not getting too deep into ebuild syntax, the core part
> of the idea is to mark some of the USE_EXPAND variables 'special'.
> In this particular example, such a special flag group would be
> 'PYTHON_TARGETS'.
>
> Now, let's consider user installs a new package with one
> python_targets_python2_7 enabled. The package is built and installed
> like usual but aside to regular vdb files an additional file
> is introduced, listing all the installed files as 'belonging'
> to python_targets_python2_7.
>
> If user enables python_targets_python3_2 on the same package, the PM
> doesn't trigger a full rebuild. Instead, it builds the package with
> the new flag being the only flag in PYTHON_TARGETS. The new files are
> installed over the installed package (and added to CONTENTS in vdb),
> and the files in install image are listed in vdb as 'belonging'
> to python_targets_python3_2.

What you're proposing would liter the ebuild/eclass with has_version 
checks; in brain dead simple cases, you can replace parts of the pkg 
as you're proposing there.

However if it installs scripts, things start getting more complex; 
needs to vary how it installs if it's overlaying part of itself.

This proposal also doesn't work in the phase of := slot deps either, 
not unless you've got a way to ensure, potentially weeks/months after 
the first build, that the node locks to the same slotting.


> Whenever files from two ABIs collide, package manager either replaces
> the installed files if the 'new' ABI is considered 'better' than
> the old one or preserves it. This follows the current behavior when
> multiple ABIs are built, and later builds overwrite files from earlier
> ones.

This is handwavey and kind of crackadled; the PM has no way of knowing 
which USE_EXPAND target is considered 'best', so in the case of 
multilib (say 64b and 32b subversion) there isn't any way to which svn 
binary should be there- 64b or 32b.  Best I can tell, your proposal 
winds up just being "last one to merge wins" which isn't acceptable. 


> At the point, the additional file contains something like
> (ugly pseudo-syntax):
>
>   /usr/lib64/python2.7/foo.py python_targets_python2_7
>   /usr/lib64/python3.2/foo.py python_targets_python3_2
>   /usr/share/doc/foo-1.2.3/README.bz2 python_targets_python2_7 \
>                                       python_targets_python3_2
>
> Now, if user requests disabling python_targets_python2_7
> on the package, the package manager may not rebuild it as well.
> Instead, it removes python_targets_python2_7 from the above list,
> and unmerges the files which don't belong into any other ABI.

If we're going to do sub-packaging... which is what you're attempting 
here... the VDB backend for it minimally cannot be a one off 
USE_EXPAND hack.  That'll just back us into a corner- which the vdb 
already does quite heavily.

Any subpackaging content tracking needs to be generic and usable.  
Really is that simple.


> Sadly, this will not 'downgrade' common files to another ABI
> but I believe that it is not really a killer-feature.
>
>
> Installing new packages and upgrading existing
> ==============================================
>
> Whenever a new package is to be built and multiple ABIs are requested,
> the package manager should split the build process between particular
> ABIs. Preferably, it should build all of them one-by-one, recording
> the 'belongs' entries from the image and then install them as a single
> package.

And how does the package know that it's being targetted at multiple 
ABIs?

Your proposal is built on the assumption ebuilds will happy overlay 
themselves in differing configurations w/out ever fucking up.

That's not the case frankly, and worse... for the cases where it 
doesn't fly, your proposal basically requires the PM to hide the 
"we're building/installing your ass multiple times" information from 
the ebuild, further compounding the issue.


> Whenever a package is to be upgraded, all ABIs have to rebuilt.
> The package manager can handle it as a regular package upgrade, not
> considering 'belongs' entries more than in a fresh package install.
>
> Whenever a package is removed completely, the 'belongs' entries need
> not to be considered at all.
>
>
> Backwards compatibility
> =======================
>
> The solution aims to be fully compatible with package managers
> not supporting it. They should see it as a regular package with
> selected useflags, and an additional opaque vdb file.
>
> When such a package manager attempts to rebuild or upgrade such
> package, the vdb file should be removed, thus not introducing any
> ambiguity for PMs supporting it. The package removal is unaffected
> at all.

*cough* you forgot about the saved environment.

To run the pkg_postinst of an install pkg, you need to run it within 
that saved environment.  That's hard law when it comes to ebuilds.

You're proposing generating multiple environments, not addressing 
which is used.  A rather *fatal* flaw there is the assumption that the 
ebuild/eclasses in the tree at the time of ABI #2 are going to be the 
same/compatible as what's in the tree at the time of ABI #1.  That 
potential alone makes env handling fucktons worse.

Just heading it off also, you cannot sanely slap together multiple env 
dumps and hope it works; minimally, the USE metadata, vars calculated 
from the run, etc, will collide and the last one to merge will be 
what's seen; whether that's right or wrong.


Bluntly; this feels like an attempt to duct tape multilib on; 
Literally, I've spent ~5m reading this proposal, then inlining the 
faults I see and there are multiple semi-fatal issues in it.

If you want to do multilib, aim for something that isn't a hack of 
existing PM behaviour; whatever we do has to work well, no insne edge 
cases, etc.

Keep in mind were this to land, it's not a one off feature; it lands, 
it stays as a core part of the pm/format from that point forward 
meaning fuckups in it bite us in the ass long term.

~harring