Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Alec Warner]

On Sun, Feb 19, 2023 at 9:32 AM Michał Górny <mgorny@g.o> wrote:
>
> Signed-off-by: Michał Górny <mgorny@g.o>
> ---
>  glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 glep-9999.ebuild
>
> diff --git a/glep-9999.ebuild b/glep-9999.ebuild
> new file mode 100644
> index 0000000..9ee18ca
> --- /dev/null
> +++ b/glep-9999.ebuild
> @@ -0,0 +1,132 @@
> +---
> +GLEP: 9999
> +Title: TEST_SUITE_PRESENT variable
> +Author: Michał Górny <mgorny@g.o>
> +Type: Standards Track
> +Status: Draft
> +Version: 1
> +Created: 2023-02-19
> +Last-Modified: 2023-02-19
> +Post-History: 2023-02-19
> +Content-Type: text/x-rst
> +---
> +
> +
> +Abstract
> +========
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> +the package features a test suite.  It can be set either by the ebuild,
> +the eclass or the default ``src_test`` implementation, and afterwards
> +included in the package manager logs.  This can aid in analyzing
> +the results of automated package testing.
> +
> +
> +Motivation
> +==========
> +
> +The deployment of new Python targets in Gentoo currently involves
> +testing of a large number of Gentoo packages against the said target.
> +This is currently done manually for the most part.  It can be
> +particularly time consuming if multiple individuals repeatedly test
> +the same package only to determine that it remains incompatible with
> +the new interpreter.
> +
> +The Python team wanted to explore the use of automation to aid this
> +testing.  Unfortunately, this faces a major problem: for the vast
> +of majority of packages, the incompatibilities with new Python versions
> +do not exhibit during the installation and can only be detected through
> +running the test suite.  The results of automated testing are therefore
> +only meaningful if the package features a test phase.
> +
> +For packages using ``distutils-r1`` eclass, the presence of test suite
> +can usually be easily determined through grepping for
> +``distutils_enable_tests`` call or an explicit ``python_test()``
> +function.  Even then, it seems sensible to work towards a more generic
> +approach to tell whether a package had a test suite or not,
> +and therefore whether a particular successful automated testing result
> +means that the package actually passed tests or only confirmed that
> +the Python files were copied successfully.
> +
> +An explicit indication whether a test suite was present can be presented
> +by the package manager as part of logs, along with the result of running
> +the test phase.  Afterwards, these logs can be used to determine which
> +packages were actually tested.

So for my own edification:
 A package has no test suite: src_test always returns true; and the
output is useless for automation purposes.
 A package has a test suite: src_test may return true; and the output
is useless because it's not distinguishable from the first case.
 A package has a test suite: src_test returns false (because the tests
failed.) This is currently the case where all the value out of
src_test is gained for automation.

Here this proposal intends to distinguish between the first two cases;
by basically annotating packages so that if src_test passes and
TEST_SUITE_DETECTED=true, we can take a positive inference; and if
TEST_SUITE_DETECTED=no or indeterminate, we cannot take an inference
either way.

If so +1 to this.

-A

> +
> +
> +Specification
> +=============
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> +by a ``src_test()`` implementation to indicate whether the package
> +featured a test suite.  It can take three values:
> +
> +- ``yes`` indicating that a test suite was run
> +- ``indeterminate`` indicating that it was not possible to clearly
> +  determine whether the test suite was present or not (this could be
> +  a case e.g. when a generic test command is run and it does not
> +  indicate whether any tests were found)
> +- ``no`` indicating that no test suite was run
> +
> +This variable *should* be set by eclasses defining the ``src_test()``
> +phase.  If the package in question is using ``src_test()`` defined
> +by an eclass that does not declare it explicitly, the PM must assume
> +``indeterminate``.
> +
> +The variable *may* be set by an ebuild defining the ``src_test()``
> +phase.  If the ebuild does not define it explicitly, the PM must assume
> +``yes``.
> +
> +The default ``src_test()`` implementation as defined by the PMS sets
> +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> +target, and to ``no`` if neither of the targets is found.
> +
> +
> +Rationale
> +=========
> +
> +The use of ternary flag makes it possible to clearly represent all three
> +possible outcomes while navigating the defaults defined in the GLEP.
> +The flag is set in ``src_test()``, so that runtime conditions (such
> +as the results obtained from the actual test runner) can be used to
> +determine the actual value.
> +
> +The defaults were defined based on the following assumptions:
> +
> +1. The presence of ``check`` target is common in autotools projects but
> +   it does not guarantee that the target actually does anything, let
> +   alone run a proper test suite.  However, the lack of any test target
> +   clearly indicates that no tests were run.
> +
> +2. Eclass ``src_test`` implementations can be very generic and succeed
> +   without actually performing any testing.  It is therefore reasonable
> +   to default to ``indeterminate`` result when they are used,
> +   and recommend them to explicitly override the variable.
> +
> +3. Explicit ``src_test`` declared in ebuild can generally be assumed
> +   to actually run tests, with the exception of declaring the function
> +   to prevent ``default_src_test`` from running.  It therefore makes
> +   sense to default to ``yes`` but allow ebuilds to override the value
> +   in the latter case.
> +
> +
> +Backwards Compatibility
> +=======================
> +
> +This GLEP is entirely optional.  The package managers not implementing
> +it will treat the variable as a regular bash variable set by the test
> +phase and ignore it.
> +
> +
> +Reference Implementation
> +========================
> +
> +TODO
> +
> +
> +Copyright
> +=========
> +
> +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
> +International License.  To view a copy of this license, visit
> +https://creativecommons.org/licenses/by-sa/4.0/.
> --
> 2.39.2
>
>