[gentoo-qa] Autorepoman/QA Goals

[gentoo-qa] Autorepoman/QA Goals

[Alec Joseph Warner]

So all 3 of the people subscribed to this list have probably heard this 
already, but I re-iterate here for those who don't spend their lives 
staring at IRC backlogs.

The current QA team is definately short on overall goals.  We have a lot 
of people that are passionate about QA, and they all do their own thing 
to contribute.  However I think we can do better.

So step one is listing QA violations.  Sure Repoman complains about 
most, if not all of them and Mr Bones likes to squish the usurpers that 
break dependencies in the tree.  However I think some of them just pop 
out at people and they have no clue how to fix them, or the fix isn't 
easy, or they just don't know how to fix it so they ignore it.

Just as was discussed about making people work on Enterprise Gentoo, we 
can't make poeple do QA who don't necessary care about it.  And thats 
not to say QA isn't important, or whatnot.  I think a lot of people view 
it as a PITA, and so they ignore any problems.

Step 2 is improving our QA tool.  Sven has written Autorepoman.  I have 
half the code, and partly this mail is to ask Sven to release the other 
half.  Sven, I'd personally like to work on it, I know you have very 
little time, and it may not be the dream tool you meant it to be.  If 
necessarily I may make an attempt to write my own tool, if you are wary 
about releasing the code, although I'd prefer not to throw away code 
you've already written.

The current repoman sucks, is not extensible and worse, is within 
portage itself, bound to their release cycle, which in the past has been 
pretty slow ( although it should be speeding up ).  For now I volunteer 
maintainership of the tool, primarily because in the 2.1 branch there 
are a lot of changes and whomever maintains it will probably have to 
have close ties with the portage team.  The main problem with taking 
repoman out of portage and into QA maintainership is that when devs use 
~arch versions of portage, Repoman tends to get castrated by the API 
changes.

Step 3: Posting QA violations publically.  By Publically, I mean to the 
world.  We would need a box to run the tool on and generate a webpage, 
much like autorepoman does now ( although hopefully a bit prettier ). 
Someone also suggested an internal ( lets it be dev-only or qa-only ) 
pig tree, that also lists who last touched each file that had a QA 
violation.  This is a toss up but may prove useful ( or foolhardy ) to keep.

Step 4: Having a tool to track QA violations, per-dev, per-herd, 
per-project.  I was thinking a record similar to like a GLSA, where we 
simply record who/what/where/when/why/how and store it somewhere.  Then 
we can run reports on what projects sucked, what projects were good, 
what devs suck, what devs were good, herds..etc...

Step 5: Helping devs help themselves.  Most of the things in Repoman's 
complaint list are automagically fixable.  Repoman complains that the 
ebuild is in cvs but the digest isn't, add the digest to cvs 
automagically...or ebuild is missing HOMEPAGE, add HOMEPAGE="none", or 
whatnot.  Automation, This goes hand in hand with super sekrit goal #6

Step 6:  QA must be Unobtrusive.  We were joking in IRC that the title 
should be "QA we take the fun out of developing."  And while funny, 
thats a bad attitude to have towards the situation.  To that end we need 
to be unobtrusive to developers to gain their support of QA.  We need to 
make good QA easy for devs to accomplish so they aren't hounded by 
errors every time they commit and say fuck it QA sucks.  I think a lot 
of that has to do with tools that do the work for them, and while some 
may argue that this will allow bad habits ( Oh I didn't know X was a QA 
violation because repoman magically fixed it for me every time ), I 
think partly #1 will mitigate that issue.

Lastly of course, we need to come up with a system of punishment to go 
along with #4, in case a dev/project/herd is having issues keeping up 
with QA.  This is the most difficult issue IMHO, mostly because there 
arne't a lot of avenues to go down, and DevRel seems quite stagnant 
recently, although that mail on -dev from fmccor looks promising :) 
Specifically we probably need some kind of group to review QA records 
and enforce said punishments themselves, or through devrel as a liason 
type deal.

Lots of stuff, I know solar had some ideas relating to arch teams, I've 
cc'd a few people because I am not sure if they are subscribed ( sorry 
if I ended up sending duplicate mails to you three ).

-Alec Warner (antarus)
-- 
gentoo-qa@gentoo.org mailing list

Re: [gentoo-qa] Autorepoman/QA Goals

[Karl Trygve Kalleberg]

Alec Joseph Warner wrote:
> So all 3 of the people subscribed to this list have probably heard this
> already, but I re-iterate here for those who don't spend their lives
> staring at IRC backlogs.
> 
> The current QA team is definately short on overall goals.  We have a lot
> of people that are passionate about QA, and they all do their own thing
> to contribute.  However I think we can do better.

All the points you raise are valid, and have all been discussed before.
I can remember most of these issues going back to '01/'02. The reason
they aren't solved yet isn't because it's mentally difficult, or because
the ideas are stupid. It's because (1) it's a lot of work to put
everything together, (2) focus has been spent on dictating policies and
arguing on the lists instead of producing code, (3) nobody has cared
enough about the problem to actually write tools that developers can accept.

Don't let my comment be a discouragement. It is not. I'm just saying
that the best way out of this predicament is to just write the tools.
Writing policies is useful for documentation, but useless for checking.
Whoever picks up this project should carefully stay of of the limelight
to avoid the usual flamefests. Historically, that's how the QA people
got burnt out before, trying to force through policies, and never got
around to the kick-ass tools we wanted.


Cheers,

-- Karl T
-- 
gentoo-qa@gentoo.org mailing list