Re: [gentoo-dev] [RFC] python-r1 + distutils-r1, v2 (+ comparison)

[Michał Górny]

On Fri, 12 Oct 2012 13:35:44 +0300
Reinis Danne <rei4dan@×××××.com> wrote:

> On Mon, Oct 08, 2012 at 05:16:05PM +0200, Michał Górny wrote:
> > Hello,
> >
> > This is the second and hopefully last version of python-r1 +
> > distutils-r1 eclasses before committing. I would also like to shortly
> > point out the goals and the differences between various python eclasses
> > in Gentoo.
> >
> > Both are attached. For those who prefer hosted form:
> > https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/python-r1.eclass
> > https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/distutils-r1.eclass
> >
> > Changes from the previous version:
> > - support for DOCS and HTML_DOCS (alike EAPI 4);
> > - minimal support for passing arguments to setup.py (through
> >   distutils-r1_python_compile & distutils-r1_python_install);
> > - EPYTHON values adjusted to match python.eclass;
> > - scripts are -${EPYTHON} suffixed -- i.e. foo-python2.7,
> >   foo-pypy-c1.8;
> > - a wrapper script[1] is used to choose the correct script instead of
> >   constant symlink. Thus, EPYTHON & eselect-python are respected,
> > - PYTHON_COMPAT can be an array only.
> >
> > What still needs considering:
> > - a nicer way of passing setup.py arguments (mysetuppyargs=()?,
> >   esetuppy wrapper?);
> > - Prefix support.
> >
> > [1]:https://bitbucket.org/mgorny/python-exec
> >
> >
> > Now, for the differences part.
> >
> > Goals
> > -----
> >
> > python.eclass aims to support every single Python package out there,
> > including rare corner cases supported in-eclass. It supports both
> > packages supporting one and multiple Python implementations. distutils
> > support is provided by second eclass, and both provide phase functions
> > (python.eclass providing default ph.f. wrappers).
> >
> > python-distutils-ng aims to support majority of Python packages using
> > distutils. With a bit of hackery it can support non-distutils packages,
> > but the use is limited to the common cases. In an uncommon case, it's
> > not flexible enough. It supports Python packages supporting multiple
> > implementations only.
> >
> > python-r1 aims mostly to provide tools to support majority of Python
> > packages. It tries to be simple & flexible, thus allowing handling
> > various corner cases without special branches of code in the eclass.
> > It doesn't export any phase functions, nor set dependencies by default
> > (just provides a variable with them).
> >
> > Right now, it supports packages supporting multiple implementations
> > only; if necessary, the support will be extended -- either through
> > additional code if that wouldn't add too much complexity, or through
> > additional eclass.
> >
> > It is accompanied by distutils-r1 which provides a simple overlay for
> > the very common case of distutils packages. This eclass exports phase
> > functions and sets dependencies implicitly. It also handles renaming
> > the distutils-installed scripts and linking the python-exec wrapper.
> >
> >
> > Python targets
> > --------------
> >
> > python.eclass uses implicit Python target specifications. It provides
> > no ebuild-transparent way of enabling/disabling them.
> >
> > python-distutils-ng and python-r1 use USE flags to explicitly list
> > enabled Python implementations. Both eclasses use the same values for
> > PYTHON_TARGETS USE_EXPAND.
> >
> >
> > Installed scripts
> > -----------------
> >
> > python-distutils-ng rework the installed scripts creating copies for
> > Python implementations and changing the shebang. The created copies
> > don't differ any other way. The old script name is then symlinked to
> > the version for default implementation.
> >
> > python.eclass & python-r1 instead let distutils rework the scripts
> > and just rename them before merging. A wrapper is used to choose
> > the correct version, respecting EPYTHON & eselect-python.
> >
> > python.eclass installs the files to separate installation images
> > and merges them. python-r1 installs them to the main image directly,
> > renaming the installed scripts between successive implementations.
> >
> > python.eclass creates a wrapper script for each package. The script is
> > written in python. python-r1 instead installs the compiled wrapper
> > once (in an ebuild), and symlinks it for the packages. The wrapper is
> > written in much simpler C.
> >
> > The implementation suffixes for all three eclasses are different:
> > - python.eclass uses -2.6, -2.7 for Python, -2.7-pypy-1.8,
> >   -2.7-pypy-1.9 for pypy and -2.5-jython for jython (enjoy!);
> > - p-d-ng uses Python target names (-python2_6, -python2_7, -pypy1_8),
> > - p-r1 uses $EPYTHON values (-python2.6, -python2.7, -pypy-c1.8 (sic!)).
> >
> >
> > Dependency on Python
> > --------------------
> >
> > python.eclass has a few variables necessary to set dependencies
> > on Python implementation, including internal sub-syntax (and an even
> > more complete generator sub-syntax in progress overlay). I suspect that
> > most of the possible variants can be achieved using it, just please
> > don't make me try learning it all.
> >
> > Additionally, packages supporting multiple Python implementations are
> > required to specify their supported Python versions twice -- with
> > PYTHON_DEPEND and RESTRICT_PYTHON_ABIS.
> >
> > USE flag dependencies are specified using three variables, and I'd like
> > to avoid getting anywhere deeper.
> >
> > p-d-ng has a very simple dependency setup. It has a variable for
> > listing supported Python versions and a variable to make Python
> > dependencies conditional under the 'python' flag. Lately, it has been
> > added a method of listing requested USE flags.
> >
> > p-d-ng does not provide any support for more complex dependency
> > specifications, nor a way to disable adding default Python dependencies.
> >
> > python-r1 does not append any dependencies by default, and instead
> > exports them in a variable. The variable can be used to easily express
> > simple cases, and avoids polluting the ebuild in more complex cases. It
> > also supports providing USE dependencies for the implementation.
> >
> > distutils-r1 adds the Python dependencies to RDEPEND & DEPEND;
> > additionally, it adds a dependency on python-exec.
> >
> >
> > Dependencies on other packages
> > ------------------------------
> >
> > As python.eclass (as of gx86 state) does not express enabled
> > implementations explicitly, it is not possible to request those
> > implementations from other packages. The dependencies on other Python
> > packages are thus simple.
> >
> > p-d-ng does not provide any simpler way of writing dependencies
> > on other Python packages using the eclass. Thus, the necessary USE
> > dependencies have to be written by hand.
> >
> > python-r1 does provide a simple PYTHON_USEDEP variable which can be
> > used in dependency atoms to request the same Python implementations
> > being enabled.
> >
> >
> > Phase functions for distutils
> > -----------------------------
> >
> > distutils.eclass allows specifying additional arguments to setup.py.
> > src_install() installs default documentation files (*unconditionally*)
> > and ${DOCS}, it also does installation image magic.
> >
> > p-d-ng has mixed unconditional and user-overridable stuff. Copying
> > sources is done unconditionally; redoing scripts can be disabled using
> > a separate variable. setup.py invocations are overridable.
> >
> > d-r1 splits each phase into 'global' and 'per-implementation' stages
> > which are both overridable. The default stages apply PATCHES (and user
> > patches), copy sources, run setup.py, install DOCS & HTML_DOCS and link
> > python-exec wrappers.
> >
> >
> >
> > Thank you for reading up to this. I'd appreciate any comments, ideas,
> > and especially any API suggestions while the eclass still ain't live.
>
> Hi!
>
> Do you have any examples of ebuild conversion for non-distutils
> using ebuild? I was reading trough these threads but could find
> only examples for distutils-r1.

Well, yes, in the earlier thread there was one non-distutils package
requested as an example. It's autotools but the rule is similar. Here
it is:
http://git.overlays.gentoo.org/gitweb/?p=dev/mgorny.git;a=blob;f=dev-python/pygobject/pygobject-3.2.2.ebuild;h=289eacee0a0f52744a16e80d7e91d7bc5987b6a2;hb=7baa032fc4f3bf158d76c187920057c7da57f41d

Basically, you have two options:
1. Use just python-r1, get your own phase functions and use
python_foreach_impl(), If you're using out-of-source builds (which is
quite likely with cmake), it will be probably beneficial to make
cmake-utils.eclass support BUILD_DIR (without prefix).
2. Use distutils-r1 and override python_compile(), python_install()
(and probably python_prepare_all()). Well, assuming it will all play
together nice.

> I have a package which requires for each supported python
> version to know python executable, include dir and library to
> build python bindings (uses cmake) and location for
> site-packages so I can manually install them.
>
> Of these there is function only for executable
> (python_get_PYTHON). I hope there is a reasonable way to provide
> the rest of them without the need for me to resort on guessing
> the locations.

It should be pretty straightforward to add the remaining getters
to the eclass. But if you don't mind, I'd rather get on to that after
the initial eclass is committed since it's a specific case rather than
a common one.

-- 
Best regards,
Michał Górny