Gentoo Archives: gentoo-dev-announce

From: "Jorge Manuel B. S. Vicetto" <jmbsvicetto@g.o>
To: gentoo-dev <gentoo-dev@l.g.o>
Cc: gentoo-dev-announce@g.o
Subject: [gentoo-dev-announce] Automatic testing on Gentoo
Date: Tue, 10 May 2011 20:19:40
Hash: SHA1


Another issue that was raised in the discussion with the arch teams,
even though it predates the arch teams resources thread as we've talked
about it on FOSDEM 2011 and even before, is getting more automatic
testing done on Gentoo.

I'm bcc'ing a few teams on this thread as it involves them and hopefully
might interest them as well.

Both Release Engineering and QA teams would like to have more automatic
testing to find breakages and to help track "when" things break and more
importantly *why* they break.

To avoid misunderstandings, we already have testing and even automated
testing being done on Gentoo. The "first line" of testing is done by
developers using repoman and or the PM's QA tools. We also have
individual developers and the QA team hopefully checking commits and
everyone testing their packages.

Furtermore, the current weekly automatic stage building has helped
identify some issues with the tree. The tinderbox work done by Patrick
and Diego, as well as others, has also helped finding broken packages
and or identifying packages affected by major changes before they hit
the tree. The use of repoman, pcheck and or paludis quality assurance
tools in the past and present to generate reports about tree issues,
like Michael's (mr_bones) emails have also helped identifying and
addressing issues.

Recently, we've got a new site to check the results of some tests with the possibility to add more scripts
to provide / run even more tests.

So, why "more testing"? For starters, more *automatic* testing. Then
more testing as reports from testing can help greatly in identifying
when things break and why they break. As someone that looks over the
automatic stage building for amd64 and x86, and that has to talk to
teams / developers when things break, having more, more in depth and
regular automatic testing would help my (releng) job. The work for the
live-dvd would also be easier if the builds were "automated" and the job
wasn't "restarted" every N months. Furthermore, creating a framework for
developers to be able to schedule testing for proposed changes, in
particular for substantial changes, might (should?) help improve the
user's experience.

I hope you agree with "more testing" by now, but what testing? It's good
to test something, but "what" do we want to test and "how" do we want to

I think we should try to have at least the following categories of tests:

 * Portage (overlays?) QA tests
	tests with the existing QA tools to check the consistency of
dependencies and the quality of ebuilds / eclasses.

 * (on demand?) package (stable / unstable) revdep rebuild (tinderbox)
	framework to schedule testing of proposed changes and check their impact

 * Weekly (?) stable / unstable stage / ISO arch builds
	the automatic stage building, including new specs for the testing tree
as we currently only test the stable tree.

 * (schedule?) specific tailored stage4 builds
	testing of specific tailored "real world" images (web server, intranet
server, generic desktop, GNOME desktop, KDE desktop, etc).

 * Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds
	automatic creation of the live-DVD to test a very broad set of packages

 * automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI /
GUI / log parsing ?)
	framework to test the built stages / install media and ensure it works
as expected

I don't have a framework for conducting some of these tests, including
the stage/iso validation, but some of them can use the existing tools
like the stage building and the tree QA tests.

Do you have any suggestions about the automatic testing? Do you know of
other tests or tools that we can and should use to improve QA on Gentoo?

- -- 

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla -