[gentoo-dev-announce] New Python eclasses -- a summary and reminder

[Michał Górny]

Hello, fellow developers.

It seems that most of the new Python eclass development has been kept
in gentoo-python mailing list and most developers hasn't been properly
introduced to them and lack necessary information to use them. For that
reason, I'd like to quickly summarize what has changed and how Python
ebuilds shall be written nowadays.

First of all, I would like to apologize for the official eclass
guides [1] being a bit out-of-date. I was short on time lately and I
tried to use the remaining bits of it to work on ebuilds and eclasses.
For the most recent documentation, please use eclass manpages [2].

Secondly, I'd like to make it clear that the old python.eclass is
'almost' deprecated. We're in process of converting the in-tree
packages to use the new eclasses but that's a lot of work [3].

When committing new packages, you are strongly encouraged to use
the new eclasses. If you'd like to convert existing package, feel free
to -- but please make sure to do so properly, since the new eclasses
aren't supposed to be 1:1-compatible with python.eclass.

The new eclasses (except for python-any-r1) require EAPI=5. That's
mostly due to USE_EXPAND USE-deps being useless in PMS variant of
EAPI<5. That should not be a problem for most of the packages since
EAPI=5 portage is stable already and we're expecting users to migrate
to an EAPI=5 profile.

Now, a few quick details you may want to know about the new eclasses
and which may confuse you since it's different to what python.eclass
used to do. It's a bit late but I hope it helps you.

[1]:http://www.gentoo.org/proj/en/Python/#doc_chap5
[2]:http://devmanual.gentoo.org/eclass-reference/index.html
[3]:http://dev.gentoo.org/~mgorny/python-r1/conversion.xhtml


The eclasses
------------

There are 5 eclasses to choose from. You always inherit only one
of the eclasses, the 'parent' ones will be pulled in for you.

1) distutils-r1.eclass -- for *all* packages which use the distutils
build system. Replacement for distutils.eclass.

2) python-r1.eclass -- for packages which are being installed for
multiple Python implementations at the same time. Replacement for
python.eclass with SUPPORT_PYTHON_ABIS=1.

3) python-single-r1.eclass -- for packages which can be built for
one Python implementation at a time only. Replacement for python.eclass
without SUPPORT_PYTHON_ABIS.

4) python-any-r1.eclass -- for packages which need only simple,
build-time dependency on Python. Doesn't export USE flags and only uses
'|| ( python:X.Y python:X.Z )' kind of deps. Usually replaces
non-eclass deps on Python.

5) python-utils-r1.eclass -- helper functions. Inheriting it directly
sounds a bit fragile.

So, if you package uses distutils (calls setup.py), you always inherit
distutils-r1 only. Otherwise, you inherit 2-4 of your choice.



Variable and function substitutions
-----------------------------------

PYTHON_DEPEND, RESTRICT_PYTHON_ABIS -> PYTHON_COMPAT (an array!)
SUPPORT_PYTHON_ABIS -> depends on eclass
PYTHON_USE_WITH* -> hand-written USE-dep in PYTHON_REQ_USE

DISTUTILS_SRC_TEST -> manual IUSE, DEPEND, python_test()
PYTHON_CFLAGS -> has to be done by hand

PYTHON_MODNAME -> removed, not necessary anymore
DISTUTILS_USE_SEPARATE_SOURCE_DIRECTORIES -> DISTUTILS_IN_SOURCE_BUILD
DISTUTILS_GLOBAL_OPTIONS -> mydistutilsargs (an array)
DOCS -> an array now, replaces defaults (instead of appending)
added HTML_DOCS, PATCHES

$(PYTHON) -> ${PYTHON} (path) or ${EPYTHON} (basename)
${PYTHON_ABI} -> use ${EPYTHON} instead

[outside of distutils-r1, for distutils-r1 see below]
python_execute_function -> python_foreach_impl
python_execute_function -f -> python_export_best; ...
python_mod_{optimize,cleanup} -> removed

$(python_get_sitedir) and friends -> now contain ${EPREFIX} as well
insinto $(python_get_...) -> python_domodule, python_doinclude, ...

python_convert_shebangs:
- on already-installed script -> python_replicate_script,
- on to-be-installed scripts -> python_foreach_impl python_doscript ...,
- in python-single-r1 -> python_fix_shebang "${D}"...

Former two create impl-variants and wrapper. The latter converts only.

I think that's all common cases.


PYTHON_COMPAT
-------------

Eclasses 1-4 use PYTHON_COMPAT to denote supported Python
implementations. You have to list all the implementations there.
There's no 'but my package supports even Python 4!'

Please test packages before adding something there. Grep bugzie
for bugs, run test suites, *check dependencies*. If in doubt, add only
python2.7 and 3.2 (if both work). We'll add other versions if somebody
needs them.



distutils-r1
------------

Most importantly, please don't migrate packages 1:1 from the old
eclass. The new eclass design is based on python-distutils-ng. Feel
free to look at some of dev-python/ packages to get a feeling how to
solve some of the common issues.

The packages which can provide solutions for most of the common
problems (by having those problems) include:

- dev-python/matplotlib -- py2-only deps in py2+3 package (please
  always open bugs for those), weird config command,

- dev-python/numpy -- a lot of horrible, random stuff,

- dev-python/pygments -- nosetests and 2to3 conversion,

- dev-python/unittest2 -- separate sources for py2 & py3, tests having
  problems with parallel builds.

A few quick notes:

1) src_prepare() to src_install() are replaced each with two sub-phases:

- python_prepare_all() to python_install_all() -- which are run once
  for the whole build process. There you put the common stuff like
  applying patches, building and installing docs.

  python_prepare_all() and python_install_all() (only the two) have
  default functions so remember to call them
  (as distutils-r1_python_prepare_all).

- python_prepare() to python_install() -- which are run repeatedly for
  each Python implementation being built. There you put the specific
  stuff like building and installing the sources, running the tests.

  python_compile() and python_install() have defaults, and those
  defaults shall be usually called (distutils-r1_python_compile). Use
  of python_prepare() should be avoided as that implies in-source
  build.

You declare src_*() phases only in special cases (like disabling
parallel build for only one phase, see dev-python/nose). Moving 1:1
distutils.eclass python_execute_function snippets is simply wrong.

As in, WRONG:

  src_test() {
    testing() {
      footest
    }
    python_foreach_impl testing
  }

CORRECT:

  python_test() {
    footest || die "Tests fail with ${EPYTHON}"
  }


2) out-of-source builds + parallel builds usually mean fun but trouble.

When working on an ebuild, it's best to set PYTHON_TARGETS=python2_7
temporarily and get it to work like this. Then you try it with all
targets enabled and built in parallel.

If you can't fix the package to work properly with that kind of build,
try DISTUTILS_IN_SOURCE_BUILD=1 first. That usually solves most
of the issues, with the cost of keeping multiple source copies.

As a last resort, use DISTUTILS_NO_PARALLEL_BUILD=1. You may have to
use it to avoid excessive use of memory or disk space. If you have to
use it for any other reason, please report a bug upstream. If possible,
please try to set it only during specific src_*() phases as it helps
a lot in making builds faster.


3) non-standard distutils ebuilds

The distutils-r1 eclass was designed to handle most common case
of distutils use gracefully -- the Python packages. Therefore, it
exports phase functions and adds python to DEP and RDEP by default.

If you have a package with Python bindings where distutils is only
flag-conditional, you want to set DISTUTILS_OPTIONAL=1. Then add
${PYTHON_DEPS} to your DEP+RDEP yourself (as with python-r1) and call
the phases in 'use python &&'. Or just split the package, it avoids
unnecessary rebuilds and makes linking easier/more correct.

If you have a distutils package which can be installed for one Python
implementation only (like python-single-r1 or distutils.eclass without
SUPPORT_PYTHON_ABIS), set DISTUTILS_SINGLE_IMPL=1. This will switch
the code to use python-single-r1 instead of python-r1.

Both of the above variables have to be set before inheriting
distutils-r1.



python-r1 and python-single-r1
------------------------------

This is closer to 1:1 from python.eclass. A few notes though:

1) use python-r1 *only* if your package and all of its Python deps
support being built for multiple Python implementations. Otherwise,
use python-single-r1.

As in, WRONG:

  PYTHON_COMPAT=( python2_7 ) # hahaha, tricked him!
  inherit python-r1

CORRECT:

  PYTHON_COMPAT=( python{2_6,2_7} )
  inherit python-single-r1


2) no Python deps are added by default. Use:

  RDEPEND="${PYTHON_DEPS}"

or:

  RDEPEND="python? ( ${PYTHON_DEPS} )"


3) Python package deps need USE-deps to enforce same implementations
being enabled. That is:

  DEPEND="dev-python/setuptools[${PYTHON_USEDEP}]"

If necessary to hack py3 package with py2 deps (please keep a bug open
in that case!):

  RDEPEND="gtk? ( dev-python/pygtk[$(python_gen_usedep python2*)] )"
  REQUIRED_USE="gtk? ( || ( $(python_gen_useflags python2*) ) )"


4) Please try to use out-of-source builds and avoid copying sources
unnecessarily. python_foreach_impl() exports BUILD_DIR to help you
with that. autotools-utils and cmake-utils respect that.

  inherit autotools-utils

  src_configure() {
    local myeconfargs=(
      --with-foo
      #...
    )

    # it builds the package in implementation-specific dir!
    python_foreach_impl autotools-utils_src_configure
  }

  src_compile() {
    python_foreach_impl autotools-utils_src_compile
  }

If you really have to use an in-source build, python_copy_sources()
and run_in_build_dir() functions are at your service:

  src_prepare() {
    python_copy_sources
  }

  src_configure() {
    python_foreach_impl run_in_build_dir econf ...
  }


5) python-r1 doesn't export pkg_setup(). The implementation data gets
set in python_foreach_impl or by python_export_best.

python-single-r1 exports pkg_setup(). It finds out which implementation
was selected and exports all data for it.


6) The generic idea of 'converting shebangs' is discouraged.

Packages supporting multiple implementations are supposed to create
impl-variants of *all* the scripts. This is only way to enforce that
PYTHON_COMPAT is respected properly.

In distutils-r1 this is done automatically. In python-r1 packages,
you can either:

a) install scripts to temporary locations and use python_doscript()
to install them. Note that this is done for a single implementation
only (so you put it in python_foreach_impl).

It installs the script as 'foo-${EPYTHON}' and symlinks the wrapper
to 'foo'. If your build system tries to write to 'foo' then, this may
end up really bad.

b) use python_replicate_script() on an installed script. It will copy
the script into variants and then symlink the wrapper. All in one run,
don't use python_foreach_impl here.

In python-single-r1, you can mangle shebangs to match the selected
Python implementation. Use python_fix_shebang for that.

Please note that the shebang conversion function is very strict
in order to avoid mistakenly breaking something. If it fails with your
shebang, please let us know.



python-any-r1
-------------

This eclass is something special. Unlike previous eclasses, it does not
set any USE flags, so no rebuilds are forced when user changes enabled
Python implementations. It can be used to handle build-time deps
on Python.

Alike with other eclasses, you need to set PYTHON_COMPAT and use
PYTHON_DEPS to get proper dependencies.

  # Build system needs python2.6+
  PYTHON_COMPAT=( python{2_6,2_7} )
  inherit python-any-r1

  DEPEND="${PYTHON_DEPS}"

It adds '|| ( python:{2_7,2_6} )'. pkg_setup() finds out best installed
one and uses it.

-- 
Best regards,
Michał Górny