Re: [gentoo-dev] [RFC] Dynamic SLOTs

[Brian Harring]

On Mon, Jun 18, 2012 at 10:34:44AM +0200, Micha?? G??rny wrote:
> On Sun, 17 Jun 2012 09:26:55 +0200
> Micha?? G??rny <mgorny@g.o> wrote:
>
> > I'm attaching a reStructuredText version of the spec. You can view it
> > rendered as a gist[1]. But please keep the replies on the list, rather
> > than forking the gist.
> >
> > [1]:https://gist.github.com/2943774
>
> Updated version. I've introduced dynamic SLOT groups and updated all
> relevant information.
>
> I didn't break DYNAMIC_SLOTS in ebuild to multiple variables yet but it
> will be easy to change that if necessary.
>
> This fixes problem previously described as 9a (dependencies). However,
> I've added a new issue, 10c which marienz pointed out.

Bleh; wish your attachment had been text/plain for inline 
commenting; pardon any mangling...


> 3. Defining dynamic SLOT groups
> -------------------------------
>
> The list of supported dynamic SLOT groups should be declared
> in profile `make.defaults` as ``DYNAMIC_SLOT_GROUPS`` variable. That
> variable should contain whitespace-separated list of group names.
>
> For each group, the same file should have a variable named
> ``DYNAMIC_SLOTS_`` followed by the group name. That variable should list
> all possible dynamic SLOTs belonging to that group.

This is USE_EXPAND machinery; you should clarify why it's not being 
reused here, nor treat explicitly as such.

Also, this gets fugly when one starts talking about individual pkgs 
that have their own slotting, rather than global patterns like 
python/multilib.

Should consider that and address it...



> 4. Defining supported SLOTs in an ebuild
> ----------------------------------------
>
> An ebuild supporting building for multiple dynamic SLOTs has to declare
> the supported slots using ``DYNAMIC_SLOTS`` variable. The variable can
> be inherited from eclasses.
>
> For example, an ebuild supporting building for multiple Python ABIs
> would declare (either explicitly or implicitly through an eclass)::
>
> 	DYNAMIC_SLOTS='py2.6 py2.7 py3.1 py3.2'
>
> which would mean that when the package is built, one of the ``pyX.Y``
> SLOTs must be used. As all of the listed SLOTs belong to the same group,
> only of them may be used at a time.
>
> An ebuild may also declare dynamic SLOTs from multiple groups::
>
> 	DYNAMIC_SLOTS='py2.6 py2.7 lib32 lib64'
>
> In this case, both one of ``pyX.Y`` and ``libX`` SLOTs need to be
> declared.

The words "I'll implement that when hell freezes over" come to 
mind.  That's a mess requiring the PM to know all potential values 
and do interpolation on the fly; you're expecting the PM to be far, 
far more intelligent then it can be, and the results won't be 
pleasant (not counting the joys of writing such a resolver mind you).

Additionally, your notion breaks down if py3.3 supports lib64, but not 
lib32.

If the groupings are treated as USE_EXPAND (even if a segregated group 
of it), you can abuse the same REQUIRED_USE machinery to specify the 
allowed pigeon holes/slots, including arbitrary group combinations.

Still will be a bit harsh for the resolver I expect, but that's at 
least descriptive enough to handle the py3.3/lib64 scenario I 
mentioned, while being explicit data the PM can operate on (local to 
that ebuild) without having to do nastyness.

Note also that if one just dropped your notion of reinventing the 
wheel, and reused REQUIRED_USE logic for slots... well, strikes me 
that gives the flexibility you desire while folding it into existing 
slot machinery.  Haven't prototyped it, so may be cracktastic, but 
seems a bit more integrated than what you're proposing.


> 5. Building the ebuild against chosen SLOTs
> -------------------------------------------
>
> In ``pkg_*`` and ``src_*`` phases, the build environment is provided
> with currently enabled dynamic SLOTs via variables named
> ``DYNAMIC_SLOT_`` followed by dynamic SLOT group name. The ebuild must
> use this variable to adjust the build process accordingly which can be
> done either directly or via an eclass.
>
> For example, in an ebuild using dynamic SLOTs for Python ABIs, the check
> may look like::
>
> 	case ${DYNAMIC_SLOT_PYTHON} in
> 		py2.6)
> 			# ...
> 			;;
> 		py2.7)
> 			# ...
> 			;;
> 		# ...
> 	esac

For ebuilds where one can't reuse the same source (moreso, can't do 
the build of two different targets w/in that source, meaning you need 
two work trees), this breaks down- which I expect is common.  That 
workflow needs to be non sucky for ebuild devs- so... sort that angle 
please.


>
> 6. Relevance to binary and installed packages
> ---------------------------------------------
>
> It is necessary to store the dynamic SLOTs for which a package was built
> in the binary package and installed package metadata. The exact
> semantics are left to be implementation-specific. The implementation
> must ensure, however, that a multiple dynamic SLOT variants of the same
> package can be installed at the same time.
>
> For example, dynamic SLOT could be appended after package version::
>
> 	dev-python/foo-1.2.3+py2.6
> 	dev-libs/bar-1.0+lib32

As ciaran stated, this is predicated upon the vdb being opaque to 
ebuilds from this point forward.  That's a large assumption you need 
to validate before pushing this forward.

Once you have ensured that ebuilds treat the VDB as opaque... this 
section should be gutted.  Ebuild format shouldn't be dictating the 
VDB implementation (especially since the moment we *can*, I gurantee 
the PM authors will be redesigning the VDB to be non shitty, and the 
decision of how to handle the above is implementation specific).


> 7. Installed file collisions
> ----------------------------
>
> Due to the specifics of dynamic SLOT implementation, it is possible that
> files installed by various dynamic SLOT package variants collide.
> For example, each dynamic SLOT variant of a Python module may install
> a README file in the same docdir.
>
> The ebuilds must ensure that the colliding files are equivalent between
> various dynamic SLOT variants of the package. The package manager should
> overwrite those files whenever a new dynamic SLOT variant is installed,
> and must ensure that they will be removed when last dynamic SLOT variant
> of the package is uninstalled.
>
> The installed package metadata must ensure that the colliding files
> belong to all dynamic SLOT variants of the package, and their metadata
> is kept up-to-date.

Collisions should be whitelisted imo via a specific var; via that, it 
can be handled/tracked (easier on the PM, and easier on the ebuild dev 
since they have an explicit QA control to utilize).


> 8. Inter-package dependencies
> -----------------------------
>
> If an ebuild supports at least one dynamic SLOT from a particular
> dynamic SLOT group and its dependency does so, the package manager must
> implicitly depend on matching dynamic SLOT variant of the dependant
> package.
>
> If the dependant ebuild does not support a matching dynamic SLOT,
> the dependency must not be satisfied by that package.
>
> In other words, if ``dev-python/foo`` depends on ``dev-python/bar``
> and both of those ebuilds support dynamic SLOTs for Python ABIs, then
> the package manager must ensure that building ``dev-python/foo`` with
> ``py2.7`` dynamic SLOT will pull in ``dev-python/bar`` with ``py2.7``
> SLOT.

You're not giving any justification as to why this must be explicit, 
versus builtin via a new slot dep operator...

Just saying. :)


> 9. User interface
> -----------------
>
> The choice of dynamic SLOTs for ebuilds whenever not directly caused
> by implicit dependencies is to be implementation-defined.
>
> The package manager should provide an ability to control the default
> dynamic SLOT choice to external tools like eselect. For example,
> the default Python SLOT would be controlled by eselect-python.
>
> The package manager must provide user with an ability to explicitly
> choose dynamic SLOT when merging a package and to store such a SLOT
> specification in the world file.

This sounds strongly like you're requiring the manager to go 
interactive.  Note that the "store such a slot specification" bit is 
inaccurate also- you've thus far basically treated slotting as a 
separate beast from dynamic slotting.

On that subject, what *occurs* when an ebuild is already slotted, and 
has dynamic_slot crap in place?  Gcc for example, is slotted, and 
dynamic slotting is abusable for generating cross compilation 
toolchains- you skipped that entire interaction (a very, very non 
simple interaction I suspect).


> The package manager may support specifying more than a single exclusive
> dynamic SLOT variants. In this case, the package manager should split
> the request into merging multiple dynamic SLOT variants of the package.
>
> For example, the following call::
>
> 	emerge dev-python/foo+py2.6+py2.7
>
> may cause the package manager to either reject the request (because of
> colliding dynamic SLOTs) or build two dynamic SLOT variants
> of the package.

Re: "colliding dynamic slots", your proposals text, in current form, 
doesn't allow for any collisions.  To actually do what you describe, 
you'd need ^^ ( py2.6 py2.7 )


> 10. Unsolved problems
> ---------------------
>
> a) Large common parts of packages supporting dyn. SLOTs
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Often packages supporting dynamic SLOTs have large, common parts.
> In most common case, it is API documentation which gets generated
> at build time. Supporting multiple dynamic SLOTs would mean that
> documentation is going to be regenerated for every SLOT (or at least
> SLOTs supporting documentation generation).
>
> This get worse when various versions of the SLOT introduce tiny,
> irrelevant differences in the generated files. As the spec does disallow
> installing colliding files with different contents, this has to be
> handled some other way.
>
> Possible solutions:
>
> 1. Let each dynamic SLOT install own copy of files (i.e. separate docs
>    for each version),
> 2. Move the common files into separate ebuild,
> 3. Introduce a special dynamic SLOT which would install just the common
>    files, and depend on it (requires adding support for explicit dynamic
>    SLOT dependencies).

3 is sounding painful.

> b) Packages installing multiple kinds of dynamic SLOTs
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Example: packages installing bindings for multiple languages. The usual
> way of defining dynamic SLOTs, i.e.::
>
> 	DYNAMIC_SLOTS='py2.6 py2.7 ruby1.8 ruby1.9'
>
> would mean that each time both a Python version, and a Ruby version has
> to be chosen. This would have to be backed up at least with USE flags
> and it gets more ugly...
>
> An interesting solution would be to assume that only one of the listed
> SLOTs is to be used at a time. This would mean that each build involves
> building just one of the bindings for a particular language of interest.
> Sadly, this would require making the spec more complex.
>
> A straightforward solution is to simply split the package.

/me thinks you need to take a hard look at subversion bindings; that's 
not "go compile the lib and install it", that's generally "compile the 
lib once with variants in the binding" which won't match into your 
scheem all that well best I can tell.


> c) Packages depending on utilities which don't need matching SLOTs
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The implicit dependency part assumes that all dependencies expect
> matching SLOTs. However, this is not always true.
>
> For example, a package building Python module may depend on another
> Python package yet call only a utility provided in that package.
> In that case, it is more likely that the utility would actually need
> to be pulled in with the currently selected system Python ABI rather
> than one used for the requested package.
>
> This issue could be probably solved through introducing explicit dynamic
> SLOT dependency atoms.

If you want to do implicit... frankly, you need to create a working 
prototype demonstrating it.  That's a high bar requirement I realize, 
but I'm stating it since this is the sort of nasty resolution problem 
where actually *doing* it and seeing how it flies, flexes, etc, is 
critical for proving it's a viable solution for people to rely upon.

Most likely in creating such a beast, you'll wind up simplifying the 
proposal due to sharp edges encountered; that simplification should 
occur w/in the proposal itself rather than having it get passed, then 
people start going "well fuck me sideways, this isn't actually sane 
nor particularly implementable".

Just saying; prototype is required there, frankly for most of this 
proposal in general.

~harring