[gentoo-dev] [RFC] python-r1 + distutils-r1, v2 (+ comparison)

[Michał Górny]

Hello,

This is the second and hopefully last version of python-r1 +
distutils-r1 eclasses before committing. I would also like to shortly
point out the goals and the differences between various python eclasses
in Gentoo.

Both are attached. For those who prefer hosted form:
https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/python-r1.eclass
https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/distutils-r1.eclass

Changes from the previous version:
- support for DOCS and HTML_DOCS (alike EAPI 4);
- minimal support for passing arguments to setup.py (through
  distutils-r1_python_compile & distutils-r1_python_install);
- EPYTHON values adjusted to match python.eclass;
- scripts are -${EPYTHON} suffixed -- i.e. foo-python2.7,
  foo-pypy-c1.8;
- a wrapper script[1] is used to choose the correct script instead of
  constant symlink. Thus, EPYTHON & eselect-python are respected,
- PYTHON_COMPAT can be an array only.

What still needs considering:
- a nicer way of passing setup.py arguments (mysetuppyargs=()?,
  esetuppy wrapper?);
- Prefix support.

[1]:https://bitbucket.org/mgorny/python-exec


Now, for the differences part.

Goals
-----

python.eclass aims to support every single Python package out there,
including rare corner cases supported in-eclass. It supports both
packages supporting one and multiple Python implementations. distutils
support is provided by second eclass, and both provide phase functions
(python.eclass providing default ph.f. wrappers).

python-distutils-ng aims to support majority of Python packages using
distutils. With a bit of hackery it can support non-distutils packages,
but the use is limited to the common cases. In an uncommon case, it's
not flexible enough. It supports Python packages supporting multiple
implementations only.

python-r1 aims mostly to provide tools to support majority of Python
packages. It tries to be simple & flexible, thus allowing handling
various corner cases without special branches of code in the eclass.
It doesn't export any phase functions, nor set dependencies by default
(just provides a variable with them).

Right now, it supports packages supporting multiple implementations
only; if necessary, the support will be extended -- either through
additional code if that wouldn't add too much complexity, or through
additional eclass.

It is accompanied by distutils-r1 which provides a simple overlay for
the very common case of distutils packages. This eclass exports phase
functions and sets dependencies implicitly. It also handles renaming
the distutils-installed scripts and linking the python-exec wrapper.


Python targets
--------------

python.eclass uses implicit Python target specifications. It provides
no ebuild-transparent way of enabling/disabling them.

python-distutils-ng and python-r1 use USE flags to explicitly list
enabled Python implementations. Both eclasses use the same values for
PYTHON_TARGETS USE_EXPAND.


Installed scripts
-----------------

python-distutils-ng rework the installed scripts creating copies for
Python implementations and changing the shebang. The created copies
don't differ any other way. The old script name is then symlinked to
the version for default implementation.

python.eclass & python-r1 instead let distutils rework the scripts
and just rename them before merging. A wrapper is used to choose
the correct version, respecting EPYTHON & eselect-python.

python.eclass installs the files to separate installation images
and merges them. python-r1 installs them to the main image directly,
renaming the installed scripts between successive implementations.

python.eclass creates a wrapper script for each package. The script is
written in python. python-r1 instead installs the compiled wrapper
once (in an ebuild), and symlinks it for the packages. The wrapper is
written in much simpler C.

The implementation suffixes for all three eclasses are different:
- python.eclass uses -2.6, -2.7 for Python, -2.7-pypy-1.8,
  -2.7-pypy-1.9 for pypy and -2.5-jython for jython (enjoy!);
- p-d-ng uses Python target names (-python2_6, -python2_7, -pypy1_8),
- p-r1 uses $EPYTHON values (-python2.6, -python2.7, -pypy-c1.8 (sic!)).


Dependency on Python
--------------------

python.eclass has a few variables necessary to set dependencies
on Python implementation, including internal sub-syntax (and an even
more complete generator sub-syntax in progress overlay). I suspect that
most of the possible variants can be achieved using it, just please
don't make me try learning it all.

Additionally, packages supporting multiple Python implementations are
required to specify their supported Python versions twice -- with
PYTHON_DEPEND and RESTRICT_PYTHON_ABIS.

USE flag dependencies are specified using three variables, and I'd like
to avoid getting anywhere deeper.

p-d-ng has a very simple dependency setup. It has a variable for
listing supported Python versions and a variable to make Python
dependencies conditional under the 'python' flag. Lately, it has been
added a method of listing requested USE flags.

p-d-ng does not provide any support for more complex dependency
specifications, nor a way to disable adding default Python dependencies.

python-r1 does not append any dependencies by default, and instead
exports them in a variable. The variable can be used to easily express
simple cases, and avoids polluting the ebuild in more complex cases. It
also supports providing USE dependencies for the implementation.

distutils-r1 adds the Python dependencies to RDEPEND & DEPEND;
additionally, it adds a dependency on python-exec.


Dependencies on other packages
------------------------------

As python.eclass (as of gx86 state) does not express enabled
implementations explicitly, it is not possible to request those
implementations from other packages. The dependencies on other Python
packages are thus simple.

p-d-ng does not provide any simpler way of writing dependencies
on other Python packages using the eclass. Thus, the necessary USE
dependencies have to be written by hand.

python-r1 does provide a simple PYTHON_USEDEP variable which can be
used in dependency atoms to request the same Python implementations
being enabled.


Phase functions for distutils
-----------------------------

distutils.eclass allows specifying additional arguments to setup.py.
src_install() installs default documentation files (*unconditionally*)
and ${DOCS}, it also does installation image magic.

p-d-ng has mixed unconditional and user-overridable stuff. Copying
sources is done unconditionally; redoing scripts can be disabled using
a separate variable. setup.py invocations are overridable.

d-r1 splits each phase into 'global' and 'per-implementation' stages
which are both overridable. The default stages apply PATCHES (and user
patches), copy sources, run setup.py, install DOCS & HTML_DOCS and link
python-exec wrappers.



Thank you for reading up to this. I'd appreciate any comments, ideas,
and especially any API suggestions while the eclass still ain't live.

-- 
Best regards,
Michał Górny