Re: [gentoo-project] [RFC] Making "stabilization" a prerequisite to become a Gentoo Developer

[Daniel Campbell]

On Thu, Oct 05, 2017 at 02:32:59PM -0700, Rich Freeman wrote:
> On Thu, Oct 5, 2017 at 2:17 PM, Daniel Campbell <zlg@g.o> wrote:
> >
> > I think your idea is noble, but it comes down to "do QA and Arch testers
> > care enough to make their job easier and invite others?" I suspect there
> > are territorial behaviors at work that prevent people from learning and
> > improving their packages through testing. As long as our culture is
> > insular, this problem will remain.
> >
>
> I don't really think this is the problem.  If somebody wants to chime
> in and say that they have been dying to arch test but some arch team
> is keeping them away please do so, but my sense is that there just
> isn't a lot of interest in it.
>
> Face it, arch testing is tedious.  There isn't any mystery as to how
> it works.  You install a package, use it a bit, and generally make
> sure it works.  Then you commit a keyword change.
>
> Now, perhaps the whole process could be radically transformed, such as
> using automated testing, just compile testing, and improved reporting
> tools/etc.  The issue there is that this is a completely different set
> of skills than what current arch testers are actually doing.  It is
> also one of those areas that a few people talk about but few put much
> time into.  Most of the people who do talk about it aren't arch
> testers (nor do I imply that they should be, or that arch testers
> should be doing this work - I'm just pointing out that blaming arch
> testers misses the mark).

To clarify, I'm not placing "blame" per se. It's not productive and
reduces morale. We all seem in agreement that it's a problem in need of
solving, though, so my first thought was "what are arch testers doing to
solve their problem?" If anyone took that as blame, it wasn't meant to
be.

>
> The QA team really has nothing to do with the arch testing process.

Why not? If someone stabilizes a package and it breaks stable somewhere,
you can be sure QA will jump on the developer who caused the problem.
They may not be *directly* involved, but QA is necessarily part of just
about all development in the main tree. No value judgment on whether
that's good or bad, just a simple fact.

>
> Now, maybe there are some barriers, such as arch teams that don't
> really allow non-dev arch testing.  Back in the original amd64 arch
> test days where the concept was pioneered the typical process was:
>
> 1.  Non-dev trains as arch tester.
> 2.  Non-dev takes a test to be certified as an arch tester.  This was
> administered by the arch team - no recruiters/etc involved.  The arch
> tester status was just an arch team thing - it didn't have any formal
> Gentoo-level recognition really.
> 3.  Arch tester reviews stabilization/testing bugs and marks those
> they have tested.
> 4.  Arch team members monitor bugs tagged as having been tested.  They
> immediately commit changes without further testing.
>
> For this to work you really need #4 - if arch testers just submit bugs
> for them to end up in a backlog while a dev re-tests them then there
> is little benefit to the process.  Back in the original amd64 AT days
> it was rare for more than a few hours to pass between an AT marking a
> bug as tested, and the commit being made.  I'm not sure why it wasn't
> automated - probably just laziness.
>
> In any case, for any of this to work people need to WANT to arch test.
> I doubt that trying to force people to arch test will help.

Agreed. I just took a look at the wiki page for amd64 testing [1] and
there are a few pieces of documentation already. I recall mgorny doing
an audit of projects to see what they needed; were arch projects part of
that? Looks like we have a starting point already. Maybe we should be
asking *why* people don't want to arch test.

I don't arch test because (a) I don't use stable and (b) I don't want to
mess something up. There are also too many ways to do arch testing and
it's not clear which method is most reliable. chroots can be problematic
since they have bound mounts to the host system and the interaction
between systems can present false positives for issues, and are fragile
to maintain. A VM comes with its own set of problems (drivers, I/O,
admin overhead), and containers are a world all their own.

Perhaps arch testing would be more attractive if there was One True Way™
to setup an arch testing system and higher level tools to facilitate bug
interaction.

How do our downstream forks handle arch testing? Do they have the same
problems we do?
>
> -- Rich
>

[1]: https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers