Re: [gentoo-project] Call for agenda items - Council meeting 2016-08-14

[Kent Fredric]

On Tue, 9 Aug 2016 01:59:35 -0400
Rich Freeman <rich0@g.o> wrote:

> While I think your proposal is a great one, I think this is actually
> the biggest limitation.  A lot of our packages (most?) don't actually
> have tests that can be run on every build (most don't have tests, some
> have tests that take forever to run or can't be used on a clean
> install).

IMHO, That's not "ideal", but we don't need idealism to be useful here.

Tests passing give one kind of useful kind of quality test.

But "hey, it compiles" gives useful data in itself.

By easy counter example, "it doesn't compile" is in itself useful
information ( and the predominant supply of bugs filed are compilation
failures ).

Hell, sometimes I hit a compile failure and I just go "eeh, I'll look
into it next week". How many people are doing the same?

The beauty of the automated datapoint is it doesn't have to be "awesome
quality" to be useful, its just guidance for further investigation.
 
> While runtime testing doesn't HAVE to be extensive, we do want
> somebody to at least take a glance at it.

Indeed, I'm not hugely in favour of abolishing manual stabilization
entirely, but sometimes it just gets to a point where its a bit beyond
a joke with requests languishing untouched for months.

If there was even data saying "hey, look, its obvious this isn't ready
for stabilization", we could *remove* or otherwise mark for
postponement stabilization requests that were failing due to
crowd-source metrics.

This means it can also be used to focus existing stabilization efforts
to reduce the number of things being thrown in the face of manual
stabilizers.

>
> If everything you're proposing is just on top of what we're already
> doing, then we have the issue that people aren't keeping up with the
> current workload, and even if that report is ultra-nice it is actually
> one more step than we have today.  The workload would only go down if
> a machine could look at the report and stabilize things without input
> at least some of the time.

Indeed, it would require the crowd service to be automated, and the
relevant usage of the data as automated as possible, and humans would
only go looking at the data when interested.

For instance, when somebody manually files a stable request, some
watcher could run off and scour the reports in a given window and
comment "Warning: Above threshold failure rates for target in last
n-days, proceed with caution", and it would only enhance the existing
stabilization workflow.