Re: [gentoo-dev] Stabilizations and src_test

[Michał Górny]

On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote:
> Hello all,
>
> If you work on the stabilization workflow you may have noticed that:
>
> - There are people that rant if you don't run src_test against their packages;
> - There are people that rant if you open a test failure bug against their
> packages and you block the stabilization.
>
> So, unless there will be a clear policy about that, I'd suggest to introduce a
> way to establish if a package is expected to pass src_test or not.
>
> A simple way to achieve this goal would be:
> introduce a new bugzilla custom flag (like "Runtime testing required" we
> already have) maybe called "src_test pass" or similar, that, by default is yes
> and the maintainer should set it to no when needed.
>
> In case of 'yes', the arch team member must compile with FEATURE="test" and he
> is allowed to block the stabilization in case of test-failure.
>
> In case there will be a test-failure there are two choices:
> 1) The maintainer fixes the test failure;
> 2) The maintainer does not want to take care, so he can simply remove the
> blocker and set "src_test pass" to no.
>
> Both of the above are fine for me.
>
> Obviously, if there are already test-failure bugs open, the flag "src_test
> pass" should be set to 'no' when the stabilization bugs is filled.
>

This is not a problem that can be solved by a binary flag.

If package's test suite is entirely broken and unmaintained, the package
should use RESTRICT=test and not silently ask arch teams to ignore it.

If package's test suite is only slightly broken, then I'd prefer saying
'please report but ignore *these* test failures' because I can't fix
them right now but they don't seem major.  Skipping the test suite
entirely is not a solution because it doesn't disambiguate between 'few
tests fail' and 'every single test explodes'.

-- 
Best regards,
Michał Górny