Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Alec Warner]

On Sun, Feb 19, 2023 at 3:11 PM Maciej Barć <xgqt@g.o> wrote:
>
> What if developer configured an ebuild in a way that it downloads the
> test suite/files/data with USE=test?
>
> IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should
> be true (exists).

This is what I was afraid of with the name; it's not only about a
testsuite being there or not, it's about trusting that:
 - There is a test suite.
 - It was run.
 - It passed, and passing is valuable.

Just existing is not sufficient, because we cannot gain any
information from an src_test that just downloaded but didn't run the
suite. That is not valuable information.

-A

>
> W dniu 19.02.2023 o 18:32, Michał Górny pisze:
> > Signed-off-by: Michał Górny <mgorny@g.o>
> > ---
> >   glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 132 insertions(+)
> >   create mode 100644 glep-9999.ebuild
> >
> > diff --git a/glep-9999.ebuild b/glep-9999.ebuild
> > new file mode 100644
> > index 0000000..9ee18ca
> > --- /dev/null
> > +++ b/glep-9999.ebuild
> > @@ -0,0 +1,132 @@
> > +---
> > +GLEP: 9999
> > +Title: TEST_SUITE_PRESENT variable
> > +Author: Michał Górny <mgorny@g.o>
> > +Type: Standards Track
> > +Status: Draft
> > +Version: 1
> > +Created: 2023-02-19
> > +Last-Modified: 2023-02-19
> > +Post-History: 2023-02-19
> > +Content-Type: text/x-rst
> > +---
> > +
> > +
> > +Abstract
> > +========
> > +
> > +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> > +the package features a test suite.  It can be set either by the ebuild,
> > +the eclass or the default ``src_test`` implementation, and afterwards
> > +included in the package manager logs.  This can aid in analyzing
> > +the results of automated package testing.
> > +
> > +
> > +Motivation
> > +==========
> > +
> > +The deployment of new Python targets in Gentoo currently involves
> > +testing of a large number of Gentoo packages against the said target.
> > +This is currently done manually for the most part.  It can be
> > +particularly time consuming if multiple individuals repeatedly test
> > +the same package only to determine that it remains incompatible with
> > +the new interpreter.
> > +
> > +The Python team wanted to explore the use of automation to aid this
> > +testing.  Unfortunately, this faces a major problem: for the vast
> > +of majority of packages, the incompatibilities with new Python versions
> > +do not exhibit during the installation and can only be detected through
> > +running the test suite.  The results of automated testing are therefore
> > +only meaningful if the package features a test phase.
> > +
> > +For packages using ``distutils-r1`` eclass, the presence of test suite
> > +can usually be easily determined through grepping for
> > +``distutils_enable_tests`` call or an explicit ``python_test()``
> > +function.  Even then, it seems sensible to work towards a more generic
> > +approach to tell whether a package had a test suite or not,
> > +and therefore whether a particular successful automated testing result
> > +means that the package actually passed tests or only confirmed that
> > +the Python files were copied successfully.
> > +
> > +An explicit indication whether a test suite was present can be presented
> > +by the package manager as part of logs, along with the result of running
> > +the test phase.  Afterwards, these logs can be used to determine which
> > +packages were actually tested.
> > +
> > +
> > +Specification
> > +=============
> > +
> > +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> > +by a ``src_test()`` implementation to indicate whether the package
> > +featured a test suite.  It can take three values:
> > +
> > +- ``yes`` indicating that a test suite was run
> > +- ``indeterminate`` indicating that it was not possible to clearly
> > +  determine whether the test suite was present or not (this could be
> > +  a case e.g. when a generic test command is run and it does not
> > +  indicate whether any tests were found)
> > +- ``no`` indicating that no test suite was run
> > +
> > +This variable *should* be set by eclasses defining the ``src_test()``
> > +phase.  If the package in question is using ``src_test()`` defined
> > +by an eclass that does not declare it explicitly, the PM must assume
> > +``indeterminate``.
> > +
> > +The variable *may* be set by an ebuild defining the ``src_test()``
> > +phase.  If the ebuild does not define it explicitly, the PM must assume
> > +``yes``.
> > +
> > +The default ``src_test()`` implementation as defined by the PMS sets
> > +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> > +target, and to ``no`` if neither of the targets is found.
> > +
> > +
> > +Rationale
> > +=========
> > +
> > +The use of ternary flag makes it possible to clearly represent all three
> > +possible outcomes while navigating the defaults defined in the GLEP.
> > +The flag is set in ``src_test()``, so that runtime conditions (such
> > +as the results obtained from the actual test runner) can be used to
> > +determine the actual value.
> > +
> > +The defaults were defined based on the following assumptions:
> > +
> > +1. The presence of ``check`` target is common in autotools projects but
> > +   it does not guarantee that the target actually does anything, let
> > +   alone run a proper test suite.  However, the lack of any test target
> > +   clearly indicates that no tests were run.
> > +
> > +2. Eclass ``src_test`` implementations can be very generic and succeed
> > +   without actually performing any testing.  It is therefore reasonable
> > +   to default to ``indeterminate`` result when they are used,
> > +   and recommend them to explicitly override the variable.
> > +
> > +3. Explicit ``src_test`` declared in ebuild can generally be assumed
> > +   to actually run tests, with the exception of declaring the function
> > +   to prevent ``default_src_test`` from running.  It therefore makes
> > +   sense to default to ``yes`` but allow ebuilds to override the value
> > +   in the latter case.
> > +
> > +
> > +Backwards Compatibility
> > +=======================
> > +
> > +This GLEP is entirely optional.  The package managers not implementing
> > +it will treat the variable as a regular bash variable set by the test
> > +phase and ignore it.
> > +
> > +
> > +Reference Implementation
> > +========================
> > +
> > +TODO
> > +
> > +
> > +Copyright
> > +=========
> > +
> > +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
> > +International License.  To view a copy of this license, visit
> > +https://creativecommons.org/licenses/by-sa/4.0/.
>
> --
> Have a great day!
>
> ~ Maciej XGQT Barć
>
> xgqt@g.o
> Gentoo Linux developer
> (dotnet, emacs, math, ml, nim, scheme, sci)
> https://wiki.gentoo.org/wiki/User:Xgqt
> 9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C