Re: [gentoo-dev] [RFC] Dynamic SLOTs

[Michał Górny]

On Mon, 18 Jun 2012 19:42:55 -0700
Brian Harring <ferringb@×××××.com> wrote:

> Bleh; wish your attachment had been text/plain for inline
> commenting; pardon any mangling...

Sorry, forgot to change the type before sending.

> > 3. Defining dynamic SLOT groups
> > -------------------------------
> >
> > The list of supported dynamic SLOT groups should be declared
> > in profile `make.defaults` as ``DYNAMIC_SLOT_GROUPS`` variable. That
> > variable should contain whitespace-separated list of group names.
> >
> > For each group, the same file should have a variable named
> > ``DYNAMIC_SLOTS_`` followed by the group name. That variable should
> > list all possible dynamic SLOTs belonging to that group.
>
> This is USE_EXPAND machinery; you should clarify why it's not being
> reused here, nor treat explicitly as such.

I tried to avoid reusing USE flags partially since that could lead to
potential problems which I could miss. It's basically fine to use flags
here as long as we ensure that only one from a group is enabled for one
ebuild, and they are reused like the rest of the text states.

> Also, this gets fugly when one starts talking about individual pkgs
> that have their own slotting, rather than global patterns like
> python/multilib.
>
> Should consider that and address it...

Could you provide an example use case? I guess we could assume there
could be dyn. SLOTs/flags not mentioned in the global list, and that's
where developer has to explicitly express their relations.

> > 4. Defining supported SLOTs in an ebuild
> > ----------------------------------------
> >
> > An ebuild supporting building for multiple dynamic SLOTs has to
> > declare the supported slots using ``DYNAMIC_SLOTS`` variable. The
> > variable can be inherited from eclasses.
> >
> > For example, an ebuild supporting building for multiple Python ABIs
> > would declare (either explicitly or implicitly through an eclass)::
> >
> > 	DYNAMIC_SLOTS='py2.6 py2.7 py3.1 py3.2'
> >
> > which would mean that when the package is built, one of the
> > ``pyX.Y`` SLOTs must be used. As all of the listed SLOTs belong to
> > the same group, only of them may be used at a time.
> >
> > An ebuild may also declare dynamic SLOTs from multiple groups::
> >
> > 	DYNAMIC_SLOTS='py2.6 py2.7 lib32 lib64'
> >
> > In this case, both one of ``pyX.Y`` and ``libX`` SLOTs need to be
> > declared.
>
> The words "I'll implement that when hell freezes over" come to
> mind.  That's a mess requiring the PM to know all potential values
> and do interpolation on the fly; you're expecting the PM to be far,
> far more intelligent then it can be, and the results won't be
> pleasant (not counting the joys of writing such a resolver mind you).

Doesn't the previous section exactly state where all possible values
are to be listed?

> Additionally, your notion breaks down if py3.3 supports lib64, but
> not lib32.

Doesn't sound like a likely case.

> If the groupings are treated as USE_EXPAND (even if a segregated
> group of it), you can abuse the same REQUIRED_USE machinery to
> specify the allowed pigeon holes/slots, including arbitrary group
> combinations.
>
> Still will be a bit harsh for the resolver I expect, but that's at
> least descriptive enough to handle the py3.3/lib64 scenario I
> mentioned, while being explicit data the PM can operate on (local to
> that ebuild) without having to do nastyness.
>
> Note also that if one just dropped your notion of reinventing the
> wheel, and reused REQUIRED_USE logic for slots... well, strikes me
> that gives the flexibility you desire while folding it into existing
> slot machinery.  Haven't prototyped it, so may be cracktastic, but
> seems a bit more integrated than what you're proposing.

Agreed.

> > 5. Building the ebuild against chosen SLOTs
> > -------------------------------------------
> >
> > In ``pkg_*`` and ``src_*`` phases, the build environment is provided
> > with currently enabled dynamic SLOTs via variables named
> > ``DYNAMIC_SLOT_`` followed by dynamic SLOT group name. The ebuild
> > must use this variable to adjust the build process accordingly
> > which can be done either directly or via an eclass.
> >
> > For example, in an ebuild using dynamic SLOTs for Python ABIs, the
> > check may look like::
> >
> > 	case ${DYNAMIC_SLOT_PYTHON} in
> > 		py2.6)
> > 			# ...
> > 			;;
> > 		py2.7)
> > 			# ...
> > 			;;
> > 		# ...
> > 	esac
>
> For ebuilds where one can't reuse the same source (moreso, can't do
> the build of two different targets w/in that source, meaning you need
> two work trees), this breaks down- which I expect is common.  That
> workflow needs to be non sucky for ebuild devs- so... sort that angle
> please.

Yes, argparse is a good example. I was still thinking how to handle
that. Probably will require conditionals in SRC_URI and so on, so
probably one more argument for USE_EXPAND.

> > 6. Relevance to binary and installed packages
> > ---------------------------------------------
> >
> > It is necessary to store the dynamic SLOTs for which a package was
> > built in the binary package and installed package metadata. The
> > exact semantics are left to be implementation-specific. The
> > implementation must ensure, however, that a multiple dynamic SLOT
> > variants of the same package can be installed at the same time.
> >
> > For example, dynamic SLOT could be appended after package version::
> >
> > 	dev-python/foo-1.2.3+py2.6
> > 	dev-libs/bar-1.0+lib32
>
> As ciaran stated, this is predicated upon the vdb being opaque to
> ebuilds from this point forward.  That's a large assumption you need
> to validate before pushing this forward.

Right now, I'm mostly trying to check whether we can work out a good,
working solution for the issue.

> Once you have ensured that ebuilds treat the VDB as opaque... this
> section should be gutted.  Ebuild format shouldn't be dictating the
> VDB implementation (especially since the moment we *can*, I gurantee
> the PM authors will be redesigning the VDB to be non shitty, and the
> decision of how to handle the above is implementation specific).

The implementation is not dictated. It is just an example to simplify
understanding the text.

> > 7. Installed file collisions
> > ----------------------------
> >
> > Due to the specifics of dynamic SLOT implementation, it is possible
> > that files installed by various dynamic SLOT package variants
> > collide. For example, each dynamic SLOT variant of a Python module
> > may install a README file in the same docdir.
> >
> > The ebuilds must ensure that the colliding files are equivalent
> > between various dynamic SLOT variants of the package. The package
> > manager should overwrite those files whenever a new dynamic SLOT
> > variant is installed, and must ensure that they will be removed
> > when last dynamic SLOT variant of the package is uninstalled.
> >
> > The installed package metadata must ensure that the colliding files
> > belong to all dynamic SLOT variants of the package, and their
> > metadata is kept up-to-date.
>
> Collisions should be whitelisted imo via a specific var; via that, it
> can be handled/tracked (easier on the PM, and easier on the ebuild
> dev since they have an explicit QA control to utilize).

Possibly.

> > 8. Inter-package dependencies
> > -----------------------------
> >
> > If an ebuild supports at least one dynamic SLOT from a particular
> > dynamic SLOT group and its dependency does so, the package manager
> > must implicitly depend on matching dynamic SLOT variant of the
> > dependant package.
> >
> > If the dependant ebuild does not support a matching dynamic SLOT,
> > the dependency must not be satisfied by that package.
> >
> > In other words, if ``dev-python/foo`` depends on ``dev-python/bar``
> > and both of those ebuilds support dynamic SLOTs for Python ABIs,
> > then the package manager must ensure that building
> > ``dev-python/foo`` with ``py2.7`` dynamic SLOT will pull in
> > ``dev-python/bar`` with ``py2.7`` SLOT.
>
> You're not giving any justification as to why this must be explicit,
> versus builtin via a new slot dep operator...

The reason is very simple: if we require a new slot dep operator, we
require devs to check all the ebuilds in the tree and add that operator
everywhere as necessary. This also includes checking all the reverse
dependencies whenever dependent package starts supporting dynSLOTs.

In other words, I'll certainly lose Ruby support on this one. And
probably most of other devs will not support this too.

> > 9. User interface
> > -----------------
> >
> > The choice of dynamic SLOTs for ebuilds whenever not directly caused
> > by implicit dependencies is to be implementation-defined.
> >
> > The package manager should provide an ability to control the default
> > dynamic SLOT choice to external tools like eselect. For example,
> > the default Python SLOT would be controlled by eselect-python.
> >
> > The package manager must provide user with an ability to explicitly
> > choose dynamic SLOT when merging a package and to store such a SLOT
> > specification in the world file.
>
> This sounds strongly like you're requiring the manager to go
> interactive.

As in s/choose/specify/?

> Note that the "store such a slot specification" bit is
> inaccurate also- you've thus far basically treated slotting as a
> separate beast from dynamic slotting.
>
> On that subject, what *occurs* when an ebuild is already slotted, and
> has dynamic_slot crap in place?  Gcc for example, is slotted, and
> dynamic slotting is abusable for generating cross compilation
> toolchains- you skipped that entire interaction (a very, very non
> simple interaction I suspect).

I don't think one does collide with the other at any point. The main
reasoning for dynamic SLOTs is to allow creating multiple pseudo-SLOTs
with one package version. They shouldn't interact at all with regular
SLOTs which are bound to versions.

> > The package manager may support specifying more than a single
> > exclusive dynamic SLOT variants. In this case, the package manager
> > should split the request into merging multiple dynamic SLOT
> > variants of the package.
> >
> > For example, the following call::
> >
> > 	emerge dev-python/foo+py2.6+py2.7
> >
> > may cause the package manager to either reject the request (because
> > of colliding dynamic SLOTs) or build two dynamic SLOT variants
> > of the package.
>
> Re: "colliding dynamic slots", your proposals text, in current form,
> doesn't allow for any collisions.  To actually do what you describe,
> you'd need ^^ ( py2.6 py2.7 )

The groups imply implicit collisions as the 'definitions' part states.
I didn't see a reason to repeat that in every ebuild.

> > 10. Unsolved problems
> > ---------------------
> >
> > a) Large common parts of packages supporting dyn. SLOTs
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > Often packages supporting dynamic SLOTs have large, common parts.
> > In most common case, it is API documentation which gets generated
> > at build time. Supporting multiple dynamic SLOTs would mean that
> > documentation is going to be regenerated for every SLOT (or at least
> > SLOTs supporting documentation generation).
> >
> > This get worse when various versions of the SLOT introduce tiny,
> > irrelevant differences in the generated files. As the spec does
> > disallow installing colliding files with different contents, this
> > has to be handled some other way.
> >
> > Possible solutions:
> >
> > 1. Let each dynamic SLOT install own copy of files (i.e. separate
> > docs for each version),
> > 2. Move the common files into separate ebuild,
> > 3. Introduce a special dynamic SLOT which would install just the
> > common files, and depend on it (requires adding support for
> > explicit dynamic SLOT dependencies).
>
> 3 is sounding painful.
>
> > b) Packages installing multiple kinds of dynamic SLOTs
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > Example: packages installing bindings for multiple languages. The
> > usual way of defining dynamic SLOTs, i.e.::
> >
> > 	DYNAMIC_SLOTS='py2.6 py2.7 ruby1.8 ruby1.9'
> >
> > would mean that each time both a Python version, and a Ruby version
> > has to be chosen. This would have to be backed up at least with USE
> > flags and it gets more ugly...
> >
> > An interesting solution would be to assume that only one of the
> > listed SLOTs is to be used at a time. This would mean that each
> > build involves building just one of the bindings for a particular
> > language of interest. Sadly, this would require making the spec
> > more complex.
> >
> > A straightforward solution is to simply split the package.
>
> /me thinks you need to take a hard look at subversion bindings;
> that's not "go compile the lib and install it", that's generally
> "compile the lib once with variants in the binding" which won't match
> into your scheem all that well best I can tell.
>
>
> > c) Packages depending on utilities which don't need matching SLOTs
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > The implicit dependency part assumes that all dependencies expect
> > matching SLOTs. However, this is not always true.
> >
> > For example, a package building Python module may depend on another
> > Python package yet call only a utility provided in that package.
> > In that case, it is more likely that the utility would actually need
> > to be pulled in with the currently selected system Python ABI rather
> > than one used for the requested package.
> >
> > This issue could be probably solved through introducing explicit
> > dynamic SLOT dependency atoms.
>
> If you want to do implicit... frankly, you need to create a working
> prototype demonstrating it.  That's a high bar requirement I realize,
> but I'm stating it since this is the sort of nasty resolution problem
> where actually *doing* it and seeing how it flies, flexes, etc, is
> critical for proving it's a viable solution for people to rely upon.
>
> Most likely in creating such a beast, you'll wind up simplifying the
> proposal due to sharp edges encountered; that simplification should
> occur w/in the proposal itself rather than having it get passed, then
> people start going "well fuck me sideways, this isn't actually sane
> nor particularly implementable".
>
> Just saying; prototype is required there, frankly for most of this
> proposal in general.

Honestly, I'm not giving it much hope. And certainly I'd like to avoid
hacking around if it's a no-go anyway.

In pseudo-code, I thought about a logic like the following. Feel free
to correct me if this couldn't work.

(where 'slots' here are short for dynamic SLOTs)

current_slots = ('py2.7', 'lib32')

for d in dependencies:
  for p in matching_packages(d):
    try:
      sub_slots = ()

      for s in current_slots:
        sg = slot_group_for(s)
        if p.supports_any_slot(sg.possible_slots):
          if p.supports_slot(s):
            sub_slots += (s,)
          else:
            raise DynSlotNotSupported()

      parse_package_deps(p, sub_slots)
      break
    except DynSlotNotSupported:
      continue # try next package
  else:
    raise NoMatchingPackages()

-- 
Best regards,
Michał Górny