[gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@g.o>
---
 glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 132 insertions(+)
 create mode 100644 glep-9999.ebuild

diff --git a/glep-9999.ebuild b/glep-9999.ebuild
new file mode 100644
index 0000000..9ee18ca
--- /dev/null
+++ b/glep-9999.ebuild
@@ -0,0 +1,132 @@
+---
+GLEP: 9999
+Title: TEST_SUITE_PRESENT variable
+Author: Michał Górny <mgorny@g.o>
+Type: Standards Track
+Status: Draft
+Version: 1
+Created: 2023-02-19
+Last-Modified: 2023-02-19
+Post-History: 2023-02-19
+Content-Type: text/x-rst
+---
+
+
+Abstract
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite.  It can be set either by the ebuild,
+the eclass or the default ``src_test`` implementation, and afterwards
+included in the package manager logs.  This can aid in analyzing
+the results of automated package testing.
+
+
+Motivation
+==========
+
+The deployment of new Python targets in Gentoo currently involves
+testing of a large number of Gentoo packages against the said target.
+This is currently done manually for the most part.  It can be
+particularly time consuming if multiple individuals repeatedly test
+the same package only to determine that it remains incompatible with
+the new interpreter.
+
+The Python team wanted to explore the use of automation to aid this
+testing.  Unfortunately, this faces a major problem: for the vast
+of majority of packages, the incompatibilities with new Python versions
+do not exhibit during the installation and can only be detected through
+running the test suite.  The results of automated testing are therefore
+only meaningful if the package features a test phase.
+
+For packages using ``distutils-r1`` eclass, the presence of test suite
+can usually be easily determined through grepping for
+``distutils_enable_tests`` call or an explicit ``python_test()``
+function.  Even then, it seems sensible to work towards a more generic
+approach to tell whether a package had a test suite or not,
+and therefore whether a particular successful automated testing result
+means that the package actually passed tests or only confirmed that
+the Python files were copied successfully.
+
+An explicit indication whether a test suite was present can be presented
+by the package manager as part of logs, along with the result of running
+the test phase.  Afterwards, these logs can be used to determine which
+packages were actually tested.
+
+
+Specification
+=============
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
+by a ``src_test()`` implementation to indicate whether the package
+featured a test suite.  It can take three values:
+
+- ``yes`` indicating that a test suite was run
+- ``indeterminate`` indicating that it was not possible to clearly
+  determine whether the test suite was present or not (this could be
+  a case e.g. when a generic test command is run and it does not
+  indicate whether any tests were found)
+- ``no`` indicating that no test suite was run
+
+This variable *should* be set by eclasses defining the ``src_test()``
+phase.  If the package in question is using ``src_test()`` defined
+by an eclass that does not declare it explicitly, the PM must assume
+``indeterminate``.
+
+The variable *may* be set by an ebuild defining the ``src_test()``
+phase.  If the ebuild does not define it explicitly, the PM must assume
+``yes``.
+
+The default ``src_test()`` implementation as defined by the PMS sets
+the value to ``indeterminate`` if it runs a ``check`` or ``test``
+target, and to ``no`` if neither of the targets is found.
+
+
+Rationale
+=========
+
+The use of ternary flag makes it possible to clearly represent all three
+possible outcomes while navigating the defaults defined in the GLEP.
+The flag is set in ``src_test()``, so that runtime conditions (such
+as the results obtained from the actual test runner) can be used to
+determine the actual value.
+
+The defaults were defined based on the following assumptions:
+
+1. The presence of ``check`` target is common in autotools projects but
+   it does not guarantee that the target actually does anything, let
+   alone run a proper test suite.  However, the lack of any test target
+   clearly indicates that no tests were run.
+
+2. Eclass ``src_test`` implementations can be very generic and succeed
+   without actually performing any testing.  It is therefore reasonable
+   to default to ``indeterminate`` result when they are used,
+   and recommend them to explicitly override the variable.
+
+3. Explicit ``src_test`` declared in ebuild can generally be assumed
+   to actually run tests, with the exception of declaring the function
+   to prevent ``default_src_test`` from running.  It therefore makes
+   sense to default to ``yes`` but allow ebuilds to override the value
+   in the latter case.
+
+
+Backwards Compatibility
+=======================
+
+This GLEP is entirely optional.  The package managers not implementing
+it will treat the variable as a regular bash variable set by the test
+phase and ignore it.
+
+
+Reference Implementation
+========================
+
+TODO
+
+
+Copyright
+=========
+
+This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
+International License.  To view a copy of this license, visit
+https://creativecommons.org/licenses/by-sa/4.0/.
-- 
2.39.2