Re: [gentoo-dev] A feedback about the CI bug reporting system

[Michał Górny]

On Fri, 2020-11-06 at 08:21 +0100, Agostino Sarubbo wrote:
> Hello all,
>
> 6 months have been passed after the CI system started to file bug reports.
> ~ 4700 bugs have been submitted
>
> We _know_ that atm is not possible to set a specific summary, instead a
> generic summary is used in case of compile failures and test failures.
> There are also some documented limitations.

I do disagree with your presumption that this needs to be automated.
The whole point behind providing a service is that you should be ready
to dedicate *your* time into the service.  However, we keep feeling that
you assume that your time is too precious, and it is better to waste
a little bit of everybody else's time.  This is why Toralf's effort is
much more appreciated.

> If there aren't much commits, usually you get the bug after 30 minutes after
> the commit and this looks to be nice.
>
> Since there are conflicting opinions I would like to know if you find it
> useful or not.
>

I am concerned about new test scenarios being run and QA warnings being
reported without proper research and documentation.  It's not exactly
helpful to spam people with hundreds of bugs without a single proper
document explaining how to resolve issues.  This just provokes people to
do apply bad workarounds, not to mention wasting everyone's time.

The compiler-rt bugs were one example of both of my points being valid.
 You've started running a specific scenario without researching
the underlying problems first, and you have missed entirely that you're
reporting a lots of bugs for the same root issue.  People have gotten
hundreds of bugs with no clear explanation how to fix them.  The only
reason we didn't get hundreds of horrible 'fixes' is that the problem
was probably too hard to workaround.

DISTUTILS_USE_SETUPTOOLS is yet another example of a minor catastrophe.
 Sure, I could've added more information to the check message.  However,
there is a major difference between people slowly getting QA warnings
from a new check and people getting hundreds of bugs telling them to fix
these warnings.  This is a difference between involving a thinking
person, and a semi-automated process that doesn't account for obvious
mistakes.

AR bugs is another class of controversy.  There is one thing to ensure
that build systems respect AR for toolchain work, there is another to
assume that it's fine not to have 'ar' archiver on the system.  And yes,
I do realize there are other people involved in this bad idea that
apparently now you need an arch-specific toolchain to unpack
the archives inside Debian packages.

I am also concerned about the sheer number of bugs caused by missing
dependencies.  Sure, I do find it valuable to get reports of missing
dependencies and fix them in my free time.  But I disagree that
stabilizations and keywordings should keep being blocked on problems
that are unlikely to ever hit real user systems.

To summarize, what your tinderboxing effort lacks is really a human
touch.  You seem to have set the goal to file as many bugs as possible
automatically.  I disagree with that, as I would like this effort to
focus on helping developers, not pursuing them.  This requires a human
touch, not a machine lord.

-- 
Best regards,
Michał Górny