Re: [gentoo-dev] Stabilizations and src_test

[Kent Fredric]

On Sun, 12 Apr 2020 10:43:07 +0200
Agostino Sarubbo <ago@g.o> wrote:

> In case of 'yes', the arch team member must compile with FEATURE="test" and he
> is allowed to block the stabilization in case of test-failure.
>
> In case there will be a test-failure there are two choices:
> 1) The maintainer fixes the test failure;
> 2) The maintainer does not want to take care, so he can simply remove the
> blocker and set "src_test pass" to no.
>
> Both of the above are fine for me.
>
> Obviously, if there are already test-failure bugs open, the flag "src_test
> pass" should be set to 'no' when the stabilization bugs is filled.
>
> I'm sure that this way would help to avoid wasting time on both sides.
>
> What do you think about?

Agree mostly, but this problem is in two places.

Ebuilds themselves need more mechanisms to communicate this, as if an
ebuild has problematic tests, then they're likely to be problematic
every time, and remembering to set the appropriate bugzilla flags
becomes a weakness.

In this vein, there's lots of similar inadequacies you find if you
follow this thread.

Like for instance, I maintain a lot of packages where the ability for
the tests to work is *environment specific*, and there's ultimately
different *kinds* of testing, and the audiences who run each varies.

Take perl packages for instance. There's a few where the test suite
*cannot* work, because they need either network IO or access to
physical devices to function.

But simply disabling tests because of this produces a gross weakness,
because Perl code cannot even be considered to be compatible with the
target perl until you actually execute it, because those
incompatibilities are ultimately "compiler errors", and you only see
them when you spin up a runtime, unlike code with a dedicated compile
stage.

Subsequently, I've taken to bolting on ebuild-internal hacks that
simply enumerate the modules and forces a perl instance to try loading
them, to at least provide *some* degree of assurance that the code
*probably* works.

But ultimately this means the way I see it, every ebuild could have
*multiple* test targets, and could have a control variable indicating
which targets to execute by default, and consumer knobs could expand,
or reduce, the testing based on preference.

We could even have something like say:

TEST_TARGETS="+build integration"
TEST_RESTRICT="network-sandbox? ( !root? ( integration ))"


Things like say, mysql, which has a standard test suite that takes days
to run, might be well served by having controls like this, so consumers
who want to have *some* basic assurance code works, without paying the
price for a comprehensive smoke, can do so, instead of opting to choose
"tests are hard, lets not do them".

Just I've been cautious about talking about this, because this is such
a can of worms, and leads into the debate about TDEPEND, and how we
stuff everything into USE flags...

And I really would *hate* for testing control variables to be USE
flags, where --changed-use rampantly introduces unnecessary havoc

( And the eventual fact will be there will be code that tries to depend
on specific values of these test control flags, and code that wants to
omit dependencies based on them, and portage complexity goes cray cray )