Re: [gentoo-dev] [RFC] dev-python/setuptools deps in distutils-r1 packages

[Joonas Niilola]

Hey,


On 11/25/19 7:38 PM, Michał Górny wrote:
> Hi,
>
> TL;DR: should we depend on setuptools by default?  Alternatively, should
> we add distutils_enable_setuptools API to provide at least partial
> validity checks.
>
>
> The problem
> ===========
> The vast majority of Python packages nowadays uses setuptools as their
> build system.  According to a cheap grep, 1633 out of 2415 packages
> using distutils-r1 depends on it -- and I suspect the vast majority of
> those that do not are simply missing the dependency.

Truthfully I thought distutils already (B)DEPENDed on setuptools, so at
least I've left it out "on purpose". I was recently made aware it
doesn't. No bug reports are made about it, so I think it's safe to
assume setuptools is already present in every system?


>
> Are there valid cases for not using setuptools?  Notably, packages
> needed to bootstrap setuptools aren't using it.  Plus some simple
> packages that really don't need its features.
>
> There are also packages that use setuptools but have distutils fallback.
> We don't like them because switching between build systems involves file
> <-> directory replacement that isn't handled cleanly by our package
> managers.  Therefore, we want to always force one build system in them.
>
> The setuptools dependency is so common it's easy to miss.  Notably, it's
> a prerequisite of specifying package dependencies, so it's normally not
> listed in dependencies.
>
> Finally, while most of the time setuptools is just BDEPEND, there are
> cases when it's RDEPEND as well.  The most common is the use
> of entry_points -- and again, upstreams don't mention it explicitly.
>
> All that considered, I think we should work on providing a better API
> for depending on setuptools, to reduce the number of missing
> dependencies and make it possible to automatically test for them
> (reminder: PMS doesn't permit inspecting *DEPEND).
>
>
> Variant 1: automatic dependency on setuptools
> =============================================
> Basically, we add a new trinary pre-inherit variable:
>
> DISTUTILS_USE_SETUPTOOLS=no
>   -> no deps
> DISTUTILS_USE_SETUPTOOLS=bdepend
>   -> add to BDEPEND (the default)
> DISTUTILS_USE_SETUPTOOLS=rdepend
>   -> add to BDEPEND+RDEPEND
>
> This is roughly 'erring on the safe side'.  The default will work for
> the majority of packages.  We will have to disable it for setuptools
> bootstrap deps, and devs will be able to adjust it to correct values
> as they update ebuilds.  For the time being, existing *DEPEND
> on setuptools will avoid breaking stuff.
>
> This will also enable me to add extra QA checks to esetup.py.  It should
> be able to reasonably detect incorrect value and report it.  This will
> imply some 'false positives' for packages that use the old method of
> specifying setuptools in RDEPEND but that's a minor hassle.
>
> Pros:
> - works out of the box for the majority of packages
> - enables full-range QA checking
>
> Cons:
> - pre-inherit variable
> - some (harmless) false positives on existing packages

I'm a fan of this solution, unless it causes a lot of new CI noise. I
think people are getting fed up with recent changes already, starting to
ignore whatever CI bot says, unless it's fatal.

No harm in double-defining setuptools in BDEPEND right...? (via eclass
and ebuild)


>
>
> Variant 2: distutils_enable_setuptools
> ======================================
> The alternative method is to add another function
> to the distutils_enable* series:
>
> distutils_enable_setuptools [-r]
>
> The basic form adds it to BDEPEND, -r B+RDEPEND.  Of course, no dep is
> present by default.  The main difference from setting deps explicitly is
> that it permits us to do a minimal QA check between pure BDEPEND
> and B+RDEPEND for now.
>
> When all ebuilds are migrated from explicit dependencies to this method,
> we can also start detecting missing deps completely.  However, that
> presumes we require using this function rather than explicit deps.
>
> Pros:
> - no pre-inherit variables
>
> Cons:
> - only partial QA check possible for the time being
> - requires migrating all ebuilds long-term

I don't like this at this point. This would just bring unwanted noise
and lots of editing in ebuilds. V1 and V3 are better IMHO. This could be
implemented in distutils-r2.eclass, though.


>
>
> Variant 3: leave as-is, add minimal install-qa-check.d
> ======================================================
> We can just continue adding deps manually, and add a minimal install-qa-
> check.d that greps installed scripts for entry_points usage.  This will
> let us detect missing RDEPEND on setuptools but not BDEPEND.
>
> Pros:
> - no changes to ebuilds
>
> Cons:
> - only partial QA check possible

The old way, is also good.


>
>
> WDYT?
> =====
> Both options have their pros and cons.  I think V1 is the best since it
> avoids a common mistake, and gives full range QA check.  It also doesn't
> interfere with existing deps.  V2 neatly fits into the recent series but
> still requires users to remember to call it, and we can't report missing
> calls until we clean up all ebuilds.  V3 provides only minimal QA
> improvement without changing the eclass.
>
> WDYT?  Do you have any other ideas?

As I said, I don't even know how one ends up with a system without
setuptools, so I don't view this "superimportant". Still I like your
suggestions 1 & 3 in handling it in future. However if you're going to
implement CI checks with this, I don't believe the timing is right
currently.


-- juippis