Re: [gentoo-dev] [RFC] QA Team's role

[Daniel Goller]

On Sunday 26 February 2006 18:34, Mark Loeser wrote:
> Stuart Herbert <stuart@g.o> said:
> > On Sun, 2006-02-26 at 17:22 -0500, Mark Loeser wrote:
> > > * In case of emergency, or if package maintainers refuse to cooperate,
> > >   the QA team may take action themselves to fix the problem.
> >
> > I'd like to see this say
> >
> >   * In case of emergency, or after a failed appeal by the package
> >     maintainer to the council, the QA team may take action themselves
> >     to fix the problem, if the package maintainer(s) are unavailable
> >     or refuse to co-operate.
> >
> > That still gives the QA team the powers it needs for an enforcement
> > role, which is all that it needs.
>
> Your change seems to imply that the QA team must wait for the council's
> okay to go forth and fix the package, rather the QA team able to act on
> its own.  If that is the case, I don't see how we would ever be able to
> get things done when someone disagrees with us.

give the QA team the power to fix things, likely they act on something being 
broken rather than impose style changes, i do not think we want to start 
discussions and appeals while broken packages remain broken

>
> > > * In the event that a developer still insists that a package does not
> > >   break QA standards, an appeal can be made at the next council
> > > meeting. The package should be dealt with per QA's request until such a
> > > time that a decision is made by the council.
> >
> > I'd like to see an alternative wording:
> >
> >   * In the event that a developer still insists that a package does not
> >     break QA standards, or that the QA standards are somehow defective,
> >     an appeal can be made at the next council meeting.
> >
> > If it is an emergency, then the QA team is still free to intervene
> > before the council meeting.  But, where it isn't an emergency, there is
> > no need for immediate action, and so there should be no action until the
> > appeal has been heard.
>
> Again, then we are going to get into the argument of the definition of
> an emergency and never be able to get anything done.  We really hope
> problems never come down to this, which is why we left it worded this
> way.

again, give the QA team the benefit of the doubt, if they are willing to fix 
it before the maintainer, gentoo only has to gain, while hopefully everyone 
learns what not to repeat in the future
>
> > The QA team has no monopoly on what is right or wrong in Gentoo, and
> > neither do package maintainers.  If there is a disagreement that leads
> > to an appeal, the onus should be on the QA team to have to present their
> > case to the council, as they are the ones pushing for change.
>
> Again, this is taking away any power that the QA team might have, and
> gets us stuck in the current situation where we can't do anything when
> someone disagrees with us.  We are hoping that most people are not going
> to see us as idiots running around trying to change everything just because
> we don't like it.  It is expected that people will trust us to some degree,
> and we are giving a way for people to challenge our decisions if they
> believe we are wrong.

again, the QA team should have final say, and their decision should only be 
appealable after the fact, not stallable with appeals before the fact
leave the tree as is (unbroken/unfixed) until the council could review any 
appeal

>
> > > * Just because a particular QA violation has yet to cause an issue does
> > >   not change the fact that it is still a QA violation.
> >
> > I'd support the above if the following statement was also included:
> >
> >   * QA standards, and the actions required to resolve them, must be
> >     pragmatic and proportional to the impact on actual end-users of
> >     Gentoo.
> >
> > Gentoo would be best served by a QA team that spent its time tackling
> > issues that are urgent and important to end-users (quadrant 1 & 2
> > activities).  A QA team that gets bogged down in the thick of thin
> > things is a symptom of a team that has become self-serving.
>
> This is not quantifiable at all and will just get us into arguments with
> people that disagree with us that there is a problem present.
>
> > > * The QA team will maintain a list of current "QA Standards".  The list
> > >   is not meant by any means to be a comprehensive document, but rather
> > > a dynamic document that will be updated as new problems are discovered.
> >
> > These QA standards are policy changes that affect the whole tree.  The
> > QA team does *not* own QA policy - we all do, as contributors to Gentoo.
> > I'm asking the council to agree an adequate process for the approval of
> > QA standards by a wider group than just the QA team.  Maybe changes
> > could be batched up into a GLEP, say once a quarter, with the option of
> > fast-tracking GLEPs in between for emergency QA policy changes?  This
> > would allow the QA team to perform effectively, and have the added
> > benefit of ensuring that the wider developer community knew what was
> > coming in advance, and could "buy in" to the changes.
>
> Again, this bogs us down in useless process if someone wants to argue a
> change that clearly breaks things, but isn't documented.  It is
> impossible to document every single thing that is a problem, and we are
> asking for some freedom to be able to work on issues that fall under QA.

the list of documented "Don't"s should be an open list, not an ultimate and 
final list, just because i mess up in a way noone ever thought of documenting 
before, should not give me the chance to be ignorant to the fact that i 
indeed broke a package/the tree

>
> > There's a secondary issue here for me.  None of our tools are perfect;
> > we all have to work pragmatically within the capabilities of what we
> > have.  Some of the real QA issues in the tree arise from the limitations
> > of our tools.  I'd like to see the following incorporated into the QA
> > team's mandate.
> >
> >   * If the QA team wishes to push standards about specific aspects of
> >     our tools, then those standards must be endorsed by the leads of
> >     those tools.
>
> So the Portage team will have to agree with us on every single problem
> that is a QA issue?  This seems like something we can leave alone until
> it doesn't work, at which point we bring it before the council to figure
> out how we can best handle this situation.  Saying that another team
> must approve all QA checks just seems silly.
>
> > Why?  Because the QA team has no monopoly on what is right and wrong
> > with the tree.  If any proposed QA standard cannot pass a simple litmus
> > test of the endorsement of the leads of the tools, how can it be fit for
> > enforcement against the wider development community?  What is a package
> > maintainer to do?  He/she can't go back to the tools team in question
> > and ask for a fix; all they will get is that the tools team in question
> > doesn't agree with the QA team, and doesn't support that particular
> > standard.  Better to have QA standards that are strongly supported than
> > to have QA standards supported by just the one team.
>
> All of this seems to assume that the QA team is going to abuse its
> power.  So far we have had very few problems when we ask people to
> fix issues that we have found for their packages.  Is this going to
> always be true?  No, and someone needs to be able to get problems fixed
> instead of just leaving things the way they are, and we want to fill
> that role.

agreed, most objections seem to aim at preventing abuse of power rather than 
commenting on how a QA team can work reliably

>
> > I'd like to see the QA team's primary role stated as "educational"
> > first, then watchdog, then intervener in that order (emergencies
> > not-withstanding).
>
> Which is what we plan on doing, and have been doing so far.

just make sure the QA team can't be paralyzed with stubborn devs filing appeal 
after appeal, give them the power to act too, not just 
educate/watch/intervene in security issues

>
> > With that in mind, I suggest that the QA standards must include
> > educational information about the problem(s) that the standard
> > addresses, before they can be approved.  This is in no way to challenge
> > the legitimacy of the agreed standards.  It's to help all developers do
> > better QA on their own work (everyone does a better job if they
> > understand the reasons why).  It also serves the dual purpose of helping
> > the next generation of QA testers when the current team members have
> > moved on.  This could be Ciaran's opportunity to see TheDoc become an
> > "offical" document.
>
> We are working on a document, and hope to document all of the problems
> and why they are problems.  We don't plan on saying something is just
> wrong and not give an explanation.

all in all gentoo can only gain from an active, able to act QA team