Gentoo Archives: gentoo-dev-announce

From: "Jorge Manuel B. S. Vicetto" <jmbsvicetto@g.o>
To: gentoo-dev <gentoo-dev@l.g.o>
Cc: gentoo-dev-announce@g.o
Subject: [gentoo-dev-announce] Automatic testing on Gentoo
Date: Tue, 10 May 2011 20:19:40
Message-Id: 4DC99C77.1050105@gentoo.org
1 -----BEGIN PGP SIGNED MESSAGE-----
2 Hash: SHA1
3
4 Hi.
5
6 Another issue that was raised in the discussion with the arch teams,
7 even though it predates the arch teams resources thread as we've talked
8 about it on FOSDEM 2011 and even before, is getting more automatic
9 testing done on Gentoo.
10
11 I'm bcc'ing a few teams on this thread as it involves them and hopefully
12 might interest them as well.
13
14 Both Release Engineering and QA teams would like to have more automatic
15 testing to find breakages and to help track "when" things break and more
16 importantly *why* they break.
17
18 To avoid misunderstandings, we already have testing and even automated
19 testing being done on Gentoo. The "first line" of testing is done by
20 developers using repoman and or the PM's QA tools. We also have
21 individual developers and the QA team hopefully checking commits and
22 everyone testing their packages.
23
24 Furtermore, the current weekly automatic stage building has helped
25 identify some issues with the tree. The tinderbox work done by Patrick
26 and Diego, as well as others, has also helped finding broken packages
27 and or identifying packages affected by major changes before they hit
28 the tree. The use of repoman, pcheck and or paludis quality assurance
29 tools in the past and present to generate reports about tree issues,
30 like Michael's (mr_bones) emails have also helped identifying and
31 addressing issues.
32
33 Recently, we've got a new site to check the results of some tests
34 http://qa-reports.gentoo.org/ with the possibility to add more scripts
35 to provide / run even more tests.
36
37 So, why "more testing"? For starters, more *automatic* testing. Then
38 more testing as reports from testing can help greatly in identifying
39 when things break and why they break. As someone that looks over the
40 automatic stage building for amd64 and x86, and that has to talk to
41 teams / developers when things break, having more, more in depth and
42 regular automatic testing would help my (releng) job. The work for the
43 live-dvd would also be easier if the builds were "automated" and the job
44 wasn't "restarted" every N months. Furthermore, creating a framework for
45 developers to be able to schedule testing for proposed changes, in
46 particular for substantial changes, might (should?) help improve the
47 user's experience.
48
49 I hope you agree with "more testing" by now, but what testing? It's good
50 to test something, but "what" do we want to test and "how" do we want to
51 test?
52
53
54 I think we should try to have at least the following categories of tests:
55
56 * Portage (overlays?) QA tests
57 tests with the existing QA tools to check the consistency of
58 dependencies and the quality of ebuilds / eclasses.
59
60 * (on demand?) package (stable / unstable) revdep rebuild (tinderbox)
61 framework to schedule testing of proposed changes and check their impact
62
63 * Weekly (?) stable / unstable stage / ISO arch builds
64 the automatic stage building, including new specs for the testing tree
65 as we currently only test the stable tree.
66
67 * (schedule?) specific tailored stage4 builds
68 testing of specific tailored "real world" images (web server, intranet
69 server, generic desktop, GNOME desktop, KDE desktop, etc).
70
71 * Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds
72 automatic creation of the live-DVD to test a very broad set of packages
73
74 * automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI /
75 GUI / log parsing ?)
76 framework to test the built stages / install media and ensure it works
77 as expected
78
79
80 I don't have a framework for conducting some of these tests, including
81 the stage/iso validation, but some of them can use the existing tools
82 like the stage building and the tree QA tests.
83
84 Do you have any suggestions about the automatic testing? Do you know of
85 other tests or tools that we can and should use to improve QA on Gentoo?
86
87 - --
88 Regards,
89
90 Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
91 Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
92 -----BEGIN PGP SIGNATURE-----
93 Version: GnuPG v2.0.17 (GNU/Linux)
94 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
95
96 iQIcBAEBAgAGBQJNyZx3AAoJEC8ZTXQF1qEPqoIQAKxIUHJItLX2HCgqbjmOMOTu
97 P7Losyu6bAi9ndtyRGYwlmEHSRBgbrkHyllx2GCMj6vR20HBYWUiUaFn3QIghLLq
98 2d1Z75zzL6FN9IQvAM8BgQWEj7+Fe24MdOhx8knQmXzZn4jffzxeI/Clw/YzfxWd
99 7uVNWh2x48+/susjLhrkpmbQfcvuSuwK/qzhMsfJcbL5G0rHweoXtOI6L2fvLd/8
100 VxwtNPRm0lemB2DSifN5zmDiWe7Z1Tb+qnb7XZrj4KgJB154dbnpIirqW6eilYz7
101 zDVzGtjRm5MdRHzNxcHZ0M1XqR0N9BcwBBsqyh2Qhr6y8W8BX7gnqC/OuT+2LPOi
102 HzvZ4sbGq2uq6/Fqjnyv9yWtqVNDjlJI2WjuZSsmZJaPVr/zSUptPfJEO/Qdla98
103 6aC7zdZucQAG8ai6KccttsaVv2N9Q5YAmZygBsiMjBZqNMfb8vsxN8VtDattd16Y
104 ICnYBIyAxkazI94dp0dAuX429c+9+jTYZVMmGSbMQ8I/jFayEkpvim9wmCtIG+nx
105 aySk+CKUpBFxF+nAttO0NEnM5oNtoNNx8k4VtMLRVyUG/LDK7z4p1OGocGZ1uELq
106 +0aNNrY3qmDK4Yq0ID5bhp/gppn7PGrJBvm7zrqXUk7lVqs3NJHFSz4NLNIp41le
107 o0qGl3+8Mhbns1mljpmx
108 =sWpj
109 -----END PGP SIGNATURE-----