Gentoo Logo
Gentoo Spaceship




Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev-announce
Navigation:
Lists: gentoo-dev-announce: < Prev By Thread Next > < Prev By Date Next >
Headers:
To: gentoo-dev <gentoo-dev@g.o>
From: "Jorge Manuel B. S. Vicetto" <jmbsvicetto@g.o>
Subject: Automatic testing on Gentoo
Date: Tue, 10 May 2011 20:13:43 +0000
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi.

Another issue that was raised in the discussion with the arch teams,
even though it predates the arch teams resources thread as we've talked
about it on FOSDEM 2011 and even before, is getting more automatic
testing done on Gentoo.

I'm bcc'ing a few teams on this thread as it involves them and hopefully
might interest them as well.

Both Release Engineering and QA teams would like to have more automatic
testing to find breakages and to help track "when" things break and more
importantly *why* they break.

To avoid misunderstandings, we already have testing and even automated
testing being done on Gentoo. The "first line" of testing is done by
developers using repoman and or the PM's QA tools. We also have
individual developers and the QA team hopefully checking commits and
everyone testing their packages.

Furtermore, the current weekly automatic stage building has helped
identify some issues with the tree. The tinderbox work done by Patrick
and Diego, as well as others, has also helped finding broken packages
and or identifying packages affected by major changes before they hit
the tree. The use of repoman, pcheck and or paludis quality assurance
tools in the past and present to generate reports about tree issues,
like Michael's (mr_bones) emails have also helped identifying and
addressing issues.

Recently, we've got a new site to check the results of some tests
http://qa-reports.gentoo.org/ with the possibility to add more scripts
to provide / run even more tests.

So, why "more testing"? For starters, more *automatic* testing. Then
more testing as reports from testing can help greatly in identifying
when things break and why they break. As someone that looks over the
automatic stage building for amd64 and x86, and that has to talk to
teams / developers when things break, having more, more in depth and
regular automatic testing would help my (releng) job. The work for the
live-dvd would also be easier if the builds were "automated" and the job
wasn't "restarted" every N months. Furthermore, creating a framework for
developers to be able to schedule testing for proposed changes, in
particular for substantial changes, might (should?) help improve the
user's experience.

I hope you agree with "more testing" by now, but what testing? It's good
to test something, but "what" do we want to test and "how" do we want to
test?


I think we should try to have at least the following categories of tests:

 * Portage (overlays?) QA tests
	tests with the existing QA tools to check the consistency of
dependencies and the quality of ebuilds / eclasses.

 * (on demand?) package (stable / unstable) revdep rebuild (tinderbox)
	framework to schedule testing of proposed changes and check their impact

 * Weekly (?) stable / unstable stage / ISO arch builds
	the automatic stage building, including new specs for the testing tree
as we currently only test the stable tree.

 * (schedule?) specific tailored stage4 builds
	testing of specific tailored "real world" images (web server, intranet
server, generic desktop, GNOME desktop, KDE desktop, etc).

 * Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds
	automatic creation of the live-DVD to test a very broad set of packages

 * automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI /
GUI / log parsing ?)
	framework to test the built stages / install media and ensure it works
as expected


I don't have a framework for conducting some of these tests, including
the stage/iso validation, but some of them can use the existing tools
like the stage building and the tree QA tests.

Do you have any suggestions about the automatic testing? Do you know of
other tests or tools that we can and should use to improve QA on Gentoo?

- -- 
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJNyZx3AAoJEC8ZTXQF1qEPqoIQAKxIUHJItLX2HCgqbjmOMOTu
P7Losyu6bAi9ndtyRGYwlmEHSRBgbrkHyllx2GCMj6vR20HBYWUiUaFn3QIghLLq
2d1Z75zzL6FN9IQvAM8BgQWEj7+Fe24MdOhx8knQmXzZn4jffzxeI/Clw/YzfxWd
7uVNWh2x48+/susjLhrkpmbQfcvuSuwK/qzhMsfJcbL5G0rHweoXtOI6L2fvLd/8
VxwtNPRm0lemB2DSifN5zmDiWe7Z1Tb+qnb7XZrj4KgJB154dbnpIirqW6eilYz7
zDVzGtjRm5MdRHzNxcHZ0M1XqR0N9BcwBBsqyh2Qhr6y8W8BX7gnqC/OuT+2LPOi
HzvZ4sbGq2uq6/Fqjnyv9yWtqVNDjlJI2WjuZSsmZJaPVr/zSUptPfJEO/Qdla98
6aC7zdZucQAG8ai6KccttsaVv2N9Q5YAmZygBsiMjBZqNMfb8vsxN8VtDattd16Y
ICnYBIyAxkazI94dp0dAuX429c+9+jTYZVMmGSbMQ8I/jFayEkpvim9wmCtIG+nx
aySk+CKUpBFxF+nAttO0NEnM5oNtoNNx8k4VtMLRVyUG/LDK7z4p1OGocGZ1uELq
+0aNNrY3qmDK4Yq0ID5bhp/gppn7PGrJBvm7zrqXUk7lVqs3NJHFSz4NLNIp41le
o0qGl3+8Mhbns1mljpmx
=sWpj
-----END PGP SIGNATURE-----


Navigation:
Lists: gentoo-dev-announce: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: Arch teams Resources (round 2) - General issues
Next by thread:
New Staffer: Francisco Blas Izquierdo Riera (klondike)
Previous by date:
Re: Arch teams Resources (round 2) - General issues
Next by date:
New Staffer: Francisco Blas Izquierdo Riera (klondike)


Updated Jul 14, 2012

Summary: Archive of the gentoo-dev-announce mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.