Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Kent Fredric]

On 4 October 2013 05:11, "Paweł Hajdan, Jr." <phajdan.jr@g.o> wrote:

> Even then, no amount of testing guarantees lack of problems.


Indeed, but which is a better assurance, "5 testers tested this combination
and nothing bad happened", or "5000 people tested this combination and
nothing bad happened".

Now, if you were to see "no people have successfully built combination X",
that in itself is interesting, even if you don't have actual failure
reports of that combination.

Also, if "5 testers tested this combination and nothing bad happened" is
combined with "however, we have 200 similar installation failures reported
for this combination", you've got some context for research you need to do
to understand why those failures exist ( even if none of them managed to
file a bug report ).

Essentially, I'm saying we need to lower the thresholds to providing
reliable feedback about what is happening with packages in the field, ie:
Diego's smoke boxes are very very useful, but thats *one* person. Imagine
if we can get 500+ people running similar smoke operations with a
manageable feedback system.

That would give us fare more assurance than the arch testers are likely to
be able to provide.


-- 
Kent