Re: [gentoo-dev] [RFC] QA Team's role

[Mark Loeser]

Stuart Herbert <stuart@g.o> said:
> On Sun, 2006-02-26 at 17:22 -0500, Mark Loeser wrote:
> > * In case of emergency, or if package maintainers refuse to cooperate,
> >   the QA team may take action themselves to fix the problem.
>
> I'd like to see this say
>
>   * In case of emergency, or after a failed appeal by the package
>     maintainer to the council, the QA team may take action themselves
>     to fix the problem, if the package maintainer(s) are unavailable
>     or refuse to co-operate.
>
> That still gives the QA team the powers it needs for an enforcement
> role, which is all that it needs.

Your change seems to imply that the QA team must wait for the council's
okay to go forth and fix the package, rather the QA team able to act on
its own.  If that is the case, I don't see how we would ever be able to
get things done when someone disagrees with us.

> > * In the event that a developer still insists that a package does not
> >   break QA standards, an appeal can be made at the next council meeting. The
> >   package should be dealt with per QA's request until such a time that a
> >   decision is made by the council.
>
> I'd like to see an alternative wording:
>
>   * In the event that a developer still insists that a package does not
>     break QA standards, or that the QA standards are somehow defective,
>     an appeal can be made at the next council meeting.
>
> If it is an emergency, then the QA team is still free to intervene
> before the council meeting.  But, where it isn't an emergency, there is
> no need for immediate action, and so there should be no action until the
> appeal has been heard.

Again, then we are going to get into the argument of the definition of
an emergency and never be able to get anything done.  We really hope
problems never come down to this, which is why we left it worded this
way.

> The QA team has no monopoly on what is right or wrong in Gentoo, and
> neither do package maintainers.  If there is a disagreement that leads
> to an appeal, the onus should be on the QA team to have to present their
> case to the council, as they are the ones pushing for change.

Again, this is taking away any power that the QA team might have, and
gets us stuck in the current situation where we can't do anything when
someone disagrees with us.  We are hoping that most people are not going
to see us as idiots running around trying to change everything just because
we don't like it.  It is expected that people will trust us to some degree,
and we are giving a way for people to challenge our decisions if they
believe we are wrong.

>
> > * Just because a particular QA violation has yet to cause an issue does
> >   not change the fact that it is still a QA violation.
>
> I'd support the above if the following statement was also included:
>
>   * QA standards, and the actions required to resolve them, must be
>     pragmatic and proportional to the impact on actual end-users of
>     Gentoo.
>
> Gentoo would be best served by a QA team that spent its time tackling
> issues that are urgent and important to end-users (quadrant 1 & 2
> activities).  A QA team that gets bogged down in the thick of thin
> things is a symptom of a team that has become self-serving.

This is not quantifiable at all and will just get us into arguments with
people that disagree with us that there is a problem present.

>
> > * The QA team will maintain a list of current "QA Standards".  The list
> >   is not meant by any means to be a comprehensive document, but rather a
> >   dynamic document that will be updated as new problems are discovered.
>
> These QA standards are policy changes that affect the whole tree.  The
> QA team does *not* own QA policy - we all do, as contributors to Gentoo.
> I'm asking the council to agree an adequate process for the approval of
> QA standards by a wider group than just the QA team.  Maybe changes
> could be batched up into a GLEP, say once a quarter, with the option of
> fast-tracking GLEPs in between for emergency QA policy changes?  This
> would allow the QA team to perform effectively, and have the added
> benefit of ensuring that the wider developer community knew what was
> coming in advance, and could "buy in" to the changes.

Again, this bogs us down in useless process if someone wants to argue a
change that clearly breaks things, but isn't documented.  It is
impossible to document every single thing that is a problem, and we are
asking for some freedom to be able to work on issues that fall under QA.

> There's a secondary issue here for me.  None of our tools are perfect;
> we all have to work pragmatically within the capabilities of what we
> have.  Some of the real QA issues in the tree arise from the limitations
> of our tools.  I'd like to see the following incorporated into the QA
> team's mandate.
>
>   * If the QA team wishes to push standards about specific aspects of
>     our tools, then those standards must be endorsed by the leads of
>     those tools.

So the Portage team will have to agree with us on every single problem
that is a QA issue?  This seems like something we can leave alone until
it doesn't work, at which point we bring it before the council to figure
out how we can best handle this situation.  Saying that another team
must approve all QA checks just seems silly.

> Why?  Because the QA team has no monopoly on what is right and wrong
> with the tree.  If any proposed QA standard cannot pass a simple litmus
> test of the endorsement of the leads of the tools, how can it be fit for
> enforcement against the wider development community?  What is a package
> maintainer to do?  He/she can't go back to the tools team in question
> and ask for a fix; all they will get is that the tools team in question
> doesn't agree with the QA team, and doesn't support that particular
> standard.  Better to have QA standards that are strongly supported than
> to have QA standards supported by just the one team.

All of this seems to assume that the QA team is going to abuse its
power.  So far we have had very few problems when we ask people to
fix issues that we have found for their packages.  Is this going to
always be true?  No, and someone needs to be able to get problems fixed
instead of just leaving things the way they are, and we want to fill
that role.

> I'd like to see the QA team's primary role stated as "educational"
> first, then watchdog, then intervener in that order (emergencies
> not-withstanding).

Which is what we plan on doing, and have been doing so far.

> With that in mind, I suggest that the QA standards must include
> educational information about the problem(s) that the standard
> addresses, before they can be approved.  This is in no way to challenge
> the legitimacy of the agreed standards.  It's to help all developers do
> better QA on their own work (everyone does a better job if they
> understand the reasons why).  It also serves the dual purpose of helping
> the next generation of QA testers when the current team members have
> moved on.  This could be Ciaran's opportunity to see TheDoc become an
> "offical" document.

We are working on a document, and hope to document all of the problems
and why they are problems.  We don't plan on saying something is just
wrong and not give an explanation.


-- 
Mark Loeser   -   Gentoo Developer (cpp gcc-porting qa toolchain x86)
email         -   halcy0n AT gentoo DOT org
                  mark AT halcy0n DOT com
web           -   http://dev.gentoo.org/~halcy0n/
                  http://www.halcy0n.com