1 |
On 05/10/2011 01:13 PM, Jorge Manuel B. S. Vicetto wrote: |
2 |
> Hi. |
3 |
> |
4 |
> Another issue that was raised in the discussion with the arch teams, |
5 |
> even though it predates the arch teams resources thread as we've talked |
6 |
> about it on FOSDEM 2011 and even before, is getting more automatic |
7 |
> testing done on Gentoo. |
8 |
> |
9 |
> I'm bcc'ing a few teams on this thread as it involves them and hopefully |
10 |
> might interest them as well. |
11 |
> |
12 |
> Both Release Engineering and QA teams would like to have more automatic |
13 |
> testing to find breakages and to help track "when" things break and more |
14 |
> importantly *why* they break. |
15 |
> |
16 |
> To avoid misunderstandings, we already have testing and even automated |
17 |
> testing being done on Gentoo. The "first line" of testing is done by |
18 |
> developers using repoman and or the PM's QA tools. We also have |
19 |
> individual developers and the QA team hopefully checking commits and |
20 |
> everyone testing their packages. |
21 |
> |
22 |
> Furtermore, the current weekly automatic stage building has helped |
23 |
> identify some issues with the tree. The tinderbox work done by Patrick |
24 |
> and Diego, as well as others, has also helped finding broken packages |
25 |
> and or identifying packages affected by major changes before they hit |
26 |
> the tree. The use of repoman, pcheck and or paludis quality assurance |
27 |
> tools in the past and present to generate reports about tree issues, |
28 |
> like Michael's (mr_bones) emails have also helped identifying and |
29 |
> addressing issues. |
30 |
> |
31 |
> Recently, we've got a new site to check the results of some tests |
32 |
> http://qa-reports.gentoo.org/ with the possibility to add more scripts |
33 |
> to provide / run even more tests. |
34 |
> |
35 |
> So, why "more testing"? For starters, more *automatic* testing. Then |
36 |
> more testing as reports from testing can help greatly in identifying |
37 |
> when things break and why they break. As someone that looks over the |
38 |
> automatic stage building for amd64 and x86, and that has to talk to |
39 |
> teams / developers when things break, having more, more in depth and |
40 |
> regular automatic testing would help my (releng) job. The work for the |
41 |
> live-dvd would also be easier if the builds were "automated" and the job |
42 |
> wasn't "restarted" every N months. Furthermore, creating a framework for |
43 |
> developers to be able to schedule testing for proposed changes, in |
44 |
> particular for substantial changes, might (should?) help improve the |
45 |
> user's experience. |
46 |
> |
47 |
> I hope you agree with "more testing" by now, but what testing? It's good |
48 |
> to test something, but "what" do we want to test and "how" do we want to |
49 |
> test? |
50 |
> |
51 |
> |
52 |
> I think we should try to have at least the following categories of tests: |
53 |
> |
54 |
> * Portage (overlays?) QA tests |
55 |
> tests with the existing QA tools to check the consistency of |
56 |
> dependencies and the quality of ebuilds / eclasses. |
57 |
> |
58 |
> * (on demand?) package (stable / unstable) revdep rebuild (tinderbox) |
59 |
> framework to schedule testing of proposed changes and check their impact |
60 |
> |
61 |
> * Weekly (?) stable / unstable stage / ISO arch builds |
62 |
> the automatic stage building, including new specs for the testing tree |
63 |
> as we currently only test the stable tree. |
64 |
> |
65 |
> * (schedule?) specific tailored stage4 builds |
66 |
> testing of specific tailored "real world" images (web server, intranet |
67 |
> server, generic desktop, GNOME desktop, KDE desktop, etc). |
68 |
> |
69 |
> * Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds |
70 |
> automatic creation of the live-DVD to test a very broad set of packages |
71 |
> |
72 |
> * automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI / |
73 |
> GUI / log parsing ?) |
74 |
> framework to test the built stages / install media and ensure it works |
75 |
> as expected |
76 |
> |
77 |
> |
78 |
> I don't have a framework for conducting some of these tests, including |
79 |
> the stage/iso validation, but some of them can use the existing tools |
80 |
> like the stage building and the tree QA tests. |
81 |
> |
82 |
> Do you have any suggestions about the automatic testing? Do you know of |
83 |
> other tests or tools that we can and should use to improve QA on Gentoo? |
84 |
|
85 |
You might take a look at autotest from kernel.org. It's a Python based |
86 |
framework for automating testing. It's specific towards kernel testing, |
87 |
but could be modified for your needs. |
88 |
|
89 |
|
90 |
|
91 |
|
92 |
-- |
93 |
Jack Morgan |
94 |
Pub 4096R/761D8E0A 2010-09-13 Jack Morgan <jack@×××××××.com> |
95 |
Fingerprint = DD42 EA48 D701 D520 C2CD 55BE BF53 C69B 761D 8E0A |