1 |
-----BEGIN PGP SIGNED MESSAGE----- |
2 |
Hash: SHA1 |
3 |
|
4 |
Hi. |
5 |
|
6 |
Another issue that was raised in the discussion with the arch teams, |
7 |
even though it predates the arch teams resources thread as we've talked |
8 |
about it on FOSDEM 2011 and even before, is getting more automatic |
9 |
testing done on Gentoo. |
10 |
|
11 |
I'm bcc'ing a few teams on this thread as it involves them and hopefully |
12 |
might interest them as well. |
13 |
|
14 |
Both Release Engineering and QA teams would like to have more automatic |
15 |
testing to find breakages and to help track "when" things break and more |
16 |
importantly *why* they break. |
17 |
|
18 |
To avoid misunderstandings, we already have testing and even automated |
19 |
testing being done on Gentoo. The "first line" of testing is done by |
20 |
developers using repoman and or the PM's QA tools. We also have |
21 |
individual developers and the QA team hopefully checking commits and |
22 |
everyone testing their packages. |
23 |
|
24 |
Furtermore, the current weekly automatic stage building has helped |
25 |
identify some issues with the tree. The tinderbox work done by Patrick |
26 |
and Diego, as well as others, has also helped finding broken packages |
27 |
and or identifying packages affected by major changes before they hit |
28 |
the tree. The use of repoman, pcheck and or paludis quality assurance |
29 |
tools in the past and present to generate reports about tree issues, |
30 |
like Michael's (mr_bones) emails have also helped identifying and |
31 |
addressing issues. |
32 |
|
33 |
Recently, we've got a new site to check the results of some tests |
34 |
http://qa-reports.gentoo.org/ with the possibility to add more scripts |
35 |
to provide / run even more tests. |
36 |
|
37 |
So, why "more testing"? For starters, more *automatic* testing. Then |
38 |
more testing as reports from testing can help greatly in identifying |
39 |
when things break and why they break. As someone that looks over the |
40 |
automatic stage building for amd64 and x86, and that has to talk to |
41 |
teams / developers when things break, having more, more in depth and |
42 |
regular automatic testing would help my (releng) job. The work for the |
43 |
live-dvd would also be easier if the builds were "automated" and the job |
44 |
wasn't "restarted" every N months. Furthermore, creating a framework for |
45 |
developers to be able to schedule testing for proposed changes, in |
46 |
particular for substantial changes, might (should?) help improve the |
47 |
user's experience. |
48 |
|
49 |
I hope you agree with "more testing" by now, but what testing? It's good |
50 |
to test something, but "what" do we want to test and "how" do we want to |
51 |
test? |
52 |
|
53 |
|
54 |
I think we should try to have at least the following categories of tests: |
55 |
|
56 |
* Portage (overlays?) QA tests |
57 |
tests with the existing QA tools to check the consistency of |
58 |
dependencies and the quality of ebuilds / eclasses. |
59 |
|
60 |
* (on demand?) package (stable / unstable) revdep rebuild (tinderbox) |
61 |
framework to schedule testing of proposed changes and check their impact |
62 |
|
63 |
* Weekly (?) stable / unstable stage / ISO arch builds |
64 |
the automatic stage building, including new specs for the testing tree |
65 |
as we currently only test the stable tree. |
66 |
|
67 |
* (schedule?) specific tailored stage4 builds |
68 |
testing of specific tailored "real world" images (web server, intranet |
69 |
server, generic desktop, GNOME desktop, KDE desktop, etc). |
70 |
|
71 |
* Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds |
72 |
automatic creation of the live-DVD to test a very broad set of packages |
73 |
|
74 |
* automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI / |
75 |
GUI / log parsing ?) |
76 |
framework to test the built stages / install media and ensure it works |
77 |
as expected |
78 |
|
79 |
|
80 |
I don't have a framework for conducting some of these tests, including |
81 |
the stage/iso validation, but some of them can use the existing tools |
82 |
like the stage building and the tree QA tests. |
83 |
|
84 |
Do you have any suggestions about the automatic testing? Do you know of |
85 |
other tests or tools that we can and should use to improve QA on Gentoo? |
86 |
|
87 |
- -- |
88 |
Regards, |
89 |
|
90 |
Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org |
91 |
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng |
92 |
-----BEGIN PGP SIGNATURE----- |
93 |
Version: GnuPG v2.0.17 (GNU/Linux) |
94 |
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ |
95 |
|
96 |
iQIcBAEBAgAGBQJNyZx3AAoJEC8ZTXQF1qEPqoIQAKxIUHJItLX2HCgqbjmOMOTu |
97 |
P7Losyu6bAi9ndtyRGYwlmEHSRBgbrkHyllx2GCMj6vR20HBYWUiUaFn3QIghLLq |
98 |
2d1Z75zzL6FN9IQvAM8BgQWEj7+Fe24MdOhx8knQmXzZn4jffzxeI/Clw/YzfxWd |
99 |
7uVNWh2x48+/susjLhrkpmbQfcvuSuwK/qzhMsfJcbL5G0rHweoXtOI6L2fvLd/8 |
100 |
VxwtNPRm0lemB2DSifN5zmDiWe7Z1Tb+qnb7XZrj4KgJB154dbnpIirqW6eilYz7 |
101 |
zDVzGtjRm5MdRHzNxcHZ0M1XqR0N9BcwBBsqyh2Qhr6y8W8BX7gnqC/OuT+2LPOi |
102 |
HzvZ4sbGq2uq6/Fqjnyv9yWtqVNDjlJI2WjuZSsmZJaPVr/zSUptPfJEO/Qdla98 |
103 |
6aC7zdZucQAG8ai6KccttsaVv2N9Q5YAmZygBsiMjBZqNMfb8vsxN8VtDattd16Y |
104 |
ICnYBIyAxkazI94dp0dAuX429c+9+jTYZVMmGSbMQ8I/jFayEkpvim9wmCtIG+nx |
105 |
aySk+CKUpBFxF+nAttO0NEnM5oNtoNNx8k4VtMLRVyUG/LDK7z4p1OGocGZ1uELq |
106 |
+0aNNrY3qmDK4Yq0ID5bhp/gppn7PGrJBvm7zrqXUk7lVqs3NJHFSz4NLNIp41le |
107 |
o0qGl3+8Mhbns1mljpmx |
108 |
=sWpj |
109 |
-----END PGP SIGNATURE----- |