Re: [gentoo-dev] Stabilizations and src_test

[Patrick McLean]

On Sun, 12 Apr 2020 11:21:29 +0200
Michał Górny <mgorny@g.o> wrote:

> On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote:
> > Hello all,
> >
> > If you work on the stabilization workflow you may have noticed that:
> >
> > - There are people that rant if you don't run src_test against their packages;
> > - There are people that rant if you open a test failure bug against their
> > packages and you block the stabilization.
> >
> > So, unless there will be a clear policy about that, I'd suggest to introduce a
> > way to establish if a package is expected to pass src_test or not.
> >
> > A simple way to achieve this goal would be:
> > introduce a new bugzilla custom flag (like "Runtime testing required" we
> > already have) maybe called "src_test pass" or similar, that, by default is yes
> > and the maintainer should set it to no when needed.
> >
> > In case of 'yes', the arch team member must compile with FEATURE="test" and he
> > is allowed to block the stabilization in case of test-failure.
> >
> > In case there will be a test-failure there are two choices:
> > 1) The maintainer fixes the test failure;
> > 2) The maintainer does not want to take care, so he can simply remove the
> > blocker and set "src_test pass" to no.
> >
> > Both of the above are fine for me.
> >
> > Obviously, if there are already test-failure bugs open, the flag "src_test
> > pass" should be set to 'no' when the stabilization bugs is filled.
> >
>
> This is not a problem that can be solved by a binary flag.
>
> If package's test suite is entirely broken and unmaintained, the package
> should use RESTRICT=test and not silently ask arch teams to ignore it.
>
> If package's test suite is only slightly broken, then I'd prefer saying
> 'please report but ignore *these* test failures' because I can't fix
> them right now but they don't seem major.  Skipping the test suite
> entirely is not a solution because it doesn't disambiguate between 'few
> tests fail' and 'every single test explodes'.
>

I agree with this, the metric for marking an ebuild as "tests
don't work, please ignore them is RESTRICT=test". Usually for packages
that have a *few* tests that fail and don't seem major, I would suggest
trying to patch out the tests, or ask the test harness to skip the
known bad tests. This way the tinderbox will (hopefully) still succeed.