1 |
Hi, |
2 |
You are right, of course. |
3 |
The target is to standardize something that would encourage maintainers |
4 |
to actually provide non-upstream scripts to test packages. At the same |
5 |
time, it should be possible to use those scripts for automated testing |
6 |
without |
7 |
human interference. |
8 |
|
9 |
Even if they are provided independently, not as part of ebuilds, we need a |
10 |
to have them associated with a particular ebuild if need be as in case of a |
11 |
bug fix, or many ebuilds in case of basic functionality for all versions of |
12 |
a |
13 |
package. |
14 |
> And under this scheme, individual projects can define their custom |
15 |
> hooks and tricks in the ebuild themselves as metadata ( like we |
16 |
> currently to via eclass variables ), and be handled, not by portage, |
17 |
> or PMS, not by EAPI, but by the QA tool in conjunction with eclass |
18 |
> designers. |
19 |
> |
20 |
I'm not sure how to deal with eclasses but I guess I'll find out. |
21 |
|
22 |
On Tue, May 17, 2016 at 2:16 PM, Tobias Klausmann <klausman@g.o> |
23 |
wrote: |
24 |
|
25 |
> Hi! |
26 |
> |
27 |
> On Tue, 17 May 2016, Kent Fredric wrote: |
28 |
> |
29 |
> > On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@×××××.com> |
30 |
> wrote: |
31 |
> > > For normal users we wouldn't. But currently, arch-testers need to make |
32 |
> a |
33 |
> > > judgement call on what to test when a stable-req bug is filed. Tests |
34 |
> run in |
35 |
> > > src_test are provided by upstream, and does not guarantee that a |
36 |
> package |
37 |
> > > that has been merged will actually run on the system. |
38 |
> > > If the maintainer could add a couple small scripts to check basic |
39 |
> > > functionality |
40 |
> > > of the merged package, it would make testing for arch testers much |
41 |
> easier |
42 |
> > > and reliable. |
43 |
> > > Let me give an example. Let's say |
44 |
> > > app-misc/screenfetch-2.7.7 is the current stable package for |
45 |
> screenfetch in |
46 |
> > > the portage tree. |
47 |
> > > However, on running, it produces an error on the top, along with the |
48 |
> proper |
49 |
> > > output. |
50 |
> > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a |
51 |
> simple |
52 |
> > > script |
53 |
> > > |
54 |
> > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then |
55 |
> > > eerror "Still producing error" |
56 |
> > > fi |
57 |
> > > |
58 |
> > > To make sure the build is properly updating the screenfetch version, |
59 |
> and |
60 |
> > > that |
61 |
> > > the bug has in fact been fixed. This is the only way I can see to |
62 |
> reliabily |
63 |
> > > and automatically test packages that have been merged successfully |
64 |
> > |
65 |
> > |
66 |
> > I don't think this needs to be an EAPI change. And if we can acheive |
67 |
> > the goal without one, the better. |
68 |
> |
69 |
> Agreed. |
70 |
> |
71 |
> > [... lots of interesting stuff ...] |
72 |
> |
73 |
> Or maintainers and teams could do what the Emacs team does: |
74 |
> provide test plans[0]. |
75 |
> |
76 |
> Naturally, I'd prefer something automated over the very hands-on |
77 |
> tests that are a bit of a necessity for an application like |
78 |
> Emacs, but they are still better than nothing. |
79 |
> |
80 |
> Either way, I think it is preferable to have non-upstream tests |
81 |
> of whatever shape we pick to be separate from the ebuilds and the |
82 |
> source files, for the simple reason that most users won't care |
83 |
> about them. Those who do can retrieve them in the same manner as |
84 |
> the ATs. |
85 |
> |
86 |
> And as for my pet peeve, tests that are known to fail, can we |
87 |
> also annotate that somehow so I don't waste hours running a test |
88 |
> suite that gives zero signal on whether I should add the stable |
89 |
> keyword? Even a one-line hin in the stabilization request would |
90 |
> be nice. As it is, I keep a list of known-to-fail packages and my |
91 |
> testing machinery tells me to not bother with FEATURES=test in |
92 |
> those case. |
93 |
> |
94 |
> Not ideal, but less time-wasty. |
95 |
> |
96 |
> Regards, |
97 |
> Tobias |
98 |
> |
99 |
> [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans |
100 |
> |
101 |
> |
102 |
> -- |
103 |
> if (user_specified) |
104 |
> /* Didn't work, but the user is convinced this is the |
105 |
> * place. */ |
106 |
> linux-2.4.0-test2/drivers/parport/parport_pc.c |
107 |
> |
108 |
> |