Re: [gentoo-dev] QA Roles v2

[Daniel Ostrow]

Mark Loeser <halcy0n@g.o> said:
> Here is my updated version after some feedback from people:
>
> * The QA team's purpose is to provide cross-team assistance in keeping
>   the tree in a good state. This is done primarily by finding and pointing
>   out issues to maintainers and, where necessary, taking direct action.
> * In case of emergency, or if package maintainers refuse to cooperate,
>   the QA team may take action themselves to fix the problem.
> * The QA team may also offer to fix obvious typos and similar minor
>   issues, and silence from the package maintainers can be taken as agreement in
>   such situations.
> * In the event that a developer still insists that a package does not
>   break QA standards, an appeal can be made at the next council meeting. The
>   package should be dealt with per QA's request until such a time that a
>   decision is made by the council.
> * In the case of disagreement on policy among QA members, the majority
>   of established QA members must agree with the action.
> * Just because a particular QA violation has yet to cause an issue does
>   not change the fact that it is still a QA violation.
> * If a particular developer persistently causes breakage, the QA team
>   may request that devrel re-evaluates that developer's commit rights.
>   Evidence of past breakages will be presented with this request to
>   devrel.
> * The QA team will maintain a list of current "QA Standards" with
>   explanations as to why they are problems, and how to fix the problem.  The
>   list is not meant by any means to be a comprehensive document, but rather a
>   dynamic document that will be updated as new problems are discovered.  The QA
>   team will also do their best to ensure all developer tools are in line with
>   the current QA standards.

I'm happy enough to send the above to the Council. I think the only issue 
will be with bullet point 4. I know that the QA team as a whole like the 
wording this way leaving the onus on the package maintainer to prove the 
merrits of their package rather then having the onus on the QA team to 
prove fault. Personally I also like the wording this way (in most cases).

In the cases where a clear technical solution presents itself to the
problems the package presents that does not change the intended behavior
(unless said behavior is what is broken) and the maintainer still
refuses to apply the neceesary changes I think that the QA team should
be cleared to make them. This is all under the understanding that the
maintainer has the right to appeal in order to revert said changes.

The tougher call comes when the severity of a QA violation comes into
question. If the QA team presents a problem to a maintainer that they
believe is severe enough to warrant a package's removal and no technical
solution has presented itself to either the QA team or the maintainer to
work around the issue I think that the QA team should have the right to
hard mask the package pending an appeal. In such cases I'd almost say
that an appeal should be automatically triggered so that a decision
about the true severity of the QA issue can be ironed out. I certainly 
hope that there will be few enough of these that the council will not end
up bogged down in policy decisions and QA conflict mediation.

The above also has to be done on a case by case basis, if hardmasking a
package would cause wide tree breakage itself then another choice has to
be made.

Concurrent with the above what I'd like to see is the QA team showing a
willingness to help maintainers find workarounds to particualarly sticky
violations. I'm not saying that they should have to learn the packages
inside out or that they should not be allowed to act until a solution is
found just that they should put a certain level of effort into helping
find (or concoct) a solution. This is not to say that they are not doing
this now, however, as has been said in the prior incarnation of this
thread I also believe that the stickier problems are likely to arise
because the maintainer in question did not see a better solution, so
part of QA's roll should be to help educate the developer community.

All in all the one thing I'd like to aviod is QA bugs getting closed as
invalid (one of the events that lead up to this thread). I know there 
is a sense of territory with the ebuilds one maintains, but I'd really 
like to see an effort made to allow the QA team to explain themselves.
If a bug gets opened up on one of your pacakges and it is unclear to you
why something is a QA issue either comment on the bug asking th QA team
to explain (and I consider pointing someone to existing offial documentation
so long as it contains an explanation of what is wrong and how to
generally fix such issues to be a valid explanation) or ping them
online.

I really hope that noone thinks the QA team is out to get them. They are 
here to make the experiance better for all of us as a whole (even if
that sometimes means that the experiance for a particular package or two
needs to be a little worse). Tree QA is something that we have never had
before, at least not really (don't mean to trivialize the work that
Mr_Bones_ does). It is something that I believe we need so lets all help
with the transition instead of fighting it.

Thanks,

-- 
Daniel Ostrow
Gentoo Foundation Board of Trustees
Gentoo/{PPC,PPC64,DevRel}
dostrow@g.o