Re: [gentoo-dev] [RFC] QA Team's role

[Stuart Herbert]

Hi Mark,

Thanks for posting this.  I've a few suggestions to make (see below).  I
support all the other points in your proposal.

On Sun, 2006-02-26 at 17:22 -0500, Mark Loeser wrote:
> * In case of emergency, or if package maintainers refuse to cooperate,
>   the QA team may take action themselves to fix the problem.

I'd like to see this say

  * In case of emergency, or after a failed appeal by the package
    maintainer to the council, the QA team may take action themselves 
    to fix the problem, if the package maintainer(s) are unavailable
    or refuse to co-operate.

That still gives the QA team the powers it needs for an enforcement
role, which is all that it needs.

> * In the event that a developer still insists that a package does not
>   break QA standards, an appeal can be made at the next council meeting. The
>   package should be dealt with per QA's request until such a time that a
>   decision is made by the council.

I'd like to see an alternative wording:

  * In the event that a developer still insists that a package does not
    break QA standards, or that the QA standards are somehow defective, 
    an appeal can be made at the next council meeting.

If it is an emergency, then the QA team is still free to intervene
before the council meeting.  But, where it isn't an emergency, there is
no need for immediate action, and so there should be no action until the
appeal has been heard.  

The original wording is, imho, unfairly unbalanced.  What will happen
under the original wording is that the QA team is free to force their
demands up front, and most developers will go with the flow rather than
take the trouble to take the issue to the council.  

The QA team has no monopoly on what is right or wrong in Gentoo, and
neither do package maintainers.  If there is a disagreement that leads
to an appeal, the onus should be on the QA team to have to present their
case to the council, as they are the ones pushing for change.

> * Just because a particular QA violation has yet to cause an issue does
>   not change the fact that it is still a QA violation.

I'd support the above if the following statement was also included:

  * QA standards, and the actions required to resolve them, must be
    pragmatic and proportional to the impact on actual end-users of
    Gentoo.

Gentoo would be best served by a QA team that spent its time tackling
issues that are urgent and important to end-users (quadrant 1 & 2
activities).  A QA team that gets bogged down in the thick of thin
things is a symptom of a team that has become self-serving.

> * The QA team will maintain a list of current "QA Standards".  The list
>   is not meant by any means to be a comprehensive document, but rather a
>   dynamic document that will be updated as new problems are discovered.

These QA standards are policy changes that affect the whole tree.  The
QA team does *not* own QA policy - we all do, as contributors to Gentoo.
I'm asking the council to agree an adequate process for the approval of
QA standards by a wider group than just the QA team.  Maybe changes
could be batched up into a GLEP, say once a quarter, with the option of
fast-tracking GLEPs in between for emergency QA policy changes?  This
would allow the QA team to perform effectively, and have the added
benefit of ensuring that the wider developer community knew what was
coming in advance, and could "buy in" to the changes.

There's a secondary issue here for me.  None of our tools are perfect;
we all have to work pragmatically within the capabilities of what we
have.  Some of the real QA issues in the tree arise from the limitations
of our tools.  I'd like to see the following incorporated into the QA
team's mandate.

  * If the QA team wishes to push standards about specific aspects of
    our tools, then those standards must be endorsed by the leads of
    those tools.

Why?  Because the QA team has no monopoly on what is right and wrong
with the tree.  If any proposed QA standard cannot pass a simple litmus
test of the endorsement of the leads of the tools, how can it be fit for
enforcement against the wider development community?  What is a package
maintainer to do?  He/she can't go back to the tools team in question
and ask for a fix; all they will get is that the tools team in question
doesn't agree with the QA team, and doesn't support that particular
standard.  Better to have QA standards that are strongly supported than
to have QA standards supported by just the one team.

I'd like to see the QA team's primary role stated as "educational"
first, then watchdog, then intervener in that order (emergencies
not-withstanding).

With that in mind, I suggest that the QA standards must include
educational information about the problem(s) that the standard
addresses, before they can be approved.  This is in no way to challenge
the legitimacy of the agreed standards.  It's to help all developers do
better QA on their own work (everyone does a better job if they
understand the reasons why).  It also serves the dual purpose of helping
the next generation of QA testers when the current team members have
moved on.  This could be Ciaran's opportunity to see TheDoc become an
"offical" document.  There are few things worse than standards
autocratically and inflexibily applied long after the original
supporters have left, and no-one else can remember why they were
necessary in the first place.

There *may* be arguments about how much effort it takes to produce this
educational material, and I'm sympathetic to that.  But for me, what it
boils down to is this: QA is something that we all need to do.  I'd
rather have a QA team that's busy teaching the rest of us do things
better than spending all its time cleaning up after a developer
community that becomes more dazed and confused and left behind as time
goes on.  And it also scales much better :)

This document does nothing to address the QA of anything other than the
package tree.  If the scope of the team is limited to just the package
tree, then I'd like to see the team named appropriately - tree-qa
perhaps - because they would be a general, all-ranging QA team.

Best regards,
Stu
-- 
Stuart Herbert                                         stuart@g.o
Gentoo Developer                                  http://www.gentoo.org/
                                          http://blog.stuartherbert.com/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--