[gentoo-dev] [RFC] dev-python/setuptools deps in distutils-r1 packages

[Michał Górny]

Hi,

TL;DR: should we depend on setuptools by default?  Alternatively, should
we add distutils_enable_setuptools API to provide at least partial
validity checks.


The problem
===========
The vast majority of Python packages nowadays uses setuptools as their
build system.  According to a cheap grep, 1633 out of 2415 packages
using distutils-r1 depends on it -- and I suspect the vast majority of
those that do not are simply missing the dependency.

Are there valid cases for not using setuptools?  Notably, packages
needed to bootstrap setuptools aren't using it.  Plus some simple
packages that really don't need its features.

There are also packages that use setuptools but have distutils fallback.
We don't like them because switching between build systems involves file
<-> directory replacement that isn't handled cleanly by our package
managers.  Therefore, we want to always force one build system in them.

The setuptools dependency is so common it's easy to miss.  Notably, it's
a prerequisite of specifying package dependencies, so it's normally not
listed in dependencies.

Finally, while most of the time setuptools is just BDEPEND, there are
cases when it's RDEPEND as well.  The most common is the use
of entry_points -- and again, upstreams don't mention it explicitly.

All that considered, I think we should work on providing a better API
for depending on setuptools, to reduce the number of missing
dependencies and make it possible to automatically test for them
(reminder: PMS doesn't permit inspecting *DEPEND).


Variant 1: automatic dependency on setuptools
=============================================
Basically, we add a new trinary pre-inherit variable:

DISTUTILS_USE_SETUPTOOLS=no
  -> no deps
DISTUTILS_USE_SETUPTOOLS=bdepend
  -> add to BDEPEND (the default)
DISTUTILS_USE_SETUPTOOLS=rdepend
  -> add to BDEPEND+RDEPEND

This is roughly 'erring on the safe side'.  The default will work for
the majority of packages.  We will have to disable it for setuptools
bootstrap deps, and devs will be able to adjust it to correct values
as they update ebuilds.  For the time being, existing *DEPEND
on setuptools will avoid breaking stuff.

This will also enable me to add extra QA checks to esetup.py.  It should
be able to reasonably detect incorrect value and report it.  This will
imply some 'false positives' for packages that use the old method of
specifying setuptools in RDEPEND but that's a minor hassle.

Pros:
- works out of the box for the majority of packages
- enables full-range QA checking

Cons:
- pre-inherit variable
- some (harmless) false positives on existing packages


Variant 2: distutils_enable_setuptools
======================================
The alternative method is to add another function
to the distutils_enable* series:

distutils_enable_setuptools [-r]

The basic form adds it to BDEPEND, -r B+RDEPEND.  Of course, no dep is
present by default.  The main difference from setting deps explicitly is
that it permits us to do a minimal QA check between pure BDEPEND
and B+RDEPEND for now.

When all ebuilds are migrated from explicit dependencies to this method,
we can also start detecting missing deps completely.  However, that
presumes we require using this function rather than explicit deps.

Pros:
- no pre-inherit variables

Cons:
- only partial QA check possible for the time being
- requires migrating all ebuilds long-term


Variant 3: leave as-is, add minimal install-qa-check.d
======================================================
We can just continue adding deps manually, and add a minimal install-qa-
check.d that greps installed scripts for entry_points usage.  This will
let us detect missing RDEPEND on setuptools but not BDEPEND.

Pros:
- no changes to ebuilds

Cons:
- only partial QA check possible


WDYT?
=====
Both options have their pros and cons.  I think V1 is the best since it
avoids a common mistake, and gives full range QA check.  It also doesn't
interfere with existing deps.  V2 neatly fits into the recent series but
still requires users to remember to call it, and we can't report missing
calls until we clean up all ebuilds.  V3 provides only minimal QA
improvement without changing the eclass.

WDYT?  Do you have any other ideas?

-- 
Best regards,
Michał Górny