1 |
On Mon, 7 Jun 2010 12:10:04 +0200 |
2 |
Thilo Bangert <bangert@g.o> wrote: |
3 |
|
4 |
> i do agree, that all packages should build successfully including the |
5 |
> test phase. RESTRICTing the test and an open bug when this is not the |
6 |
> case. |
7 |
|
8 |
I see more and more calls for either 1) "fixing the test suite", as if |
9 |
that is suddenly not an UPSTREAM issue but the ebuilds' maintainers' or |
10 |
2) setting RESTRICT=test, which would have everyone ignore the |
11 |
useful aspects of a package's test suite. |
12 |
|
13 |
The main problem with RESTRICT=test is that it's too restrictive - it |
14 |
prevents a normal emerge(1) or ebuild(1) run from entering the test |
15 |
phase at all, with no way to work around it, except by editing the |
16 |
ebuild to remove the restriction. |
17 |
|
18 |
====== |
19 |
marga /keeps/gentoo/local/app-portage/foobar # ebuild foobar-1.ebuild |
20 |
test Forcing test. |
21 |
* checking ebuild |
22 |
checksums ;-) ... [ ok ] |
23 |
* checking auxfile |
24 |
checksums ;-) ... [ ok ] |
25 |
* checking miscfile |
26 |
checksums ;-) ... [ ok ] |
27 |
* CPV: app-portage/foobar-1 |
28 |
* REPO: JeR |
29 |
* USE: elibc_glibc kernel_linux ppc test userland_GNU |
30 |
>>> Unpacking source... |
31 |
>>> Source unpacked in /var/tmp/portage/app-portage/foobar-1/work |
32 |
>>> Compiling source in /var/tmp/portage/app-portage/foobar-1/work ... |
33 |
>>> Source compiled. |
34 |
* Skipping make test/check due to ebuild restriction. |
35 |
>>> Test phase [explicitly disabled]: app-portage/foobar-1 |
36 |
marga /keeps/gentoo/local/app-portage/foobar # cat foobar-1.ebuild |
37 |
# Copyright 1999-2010 Gentoo Foundation |
38 |
# Distributed under the terms of the GNU General Public License v2 |
39 |
# $Header: $ |
40 |
|
41 |
SLOT="0" |
42 |
RESTRICT="test" |
43 |
====== |
44 |
|
45 |
If you believe that test suites are a Good Thing, then you should view |
46 |
RESTRICT=test as the ultimate solution to fix the problem of a broken |
47 |
test suite that will never be fixed. Now, in case a test suite could |
48 |
actually be destructive, dangerous to run on an unprepared system, then |
49 |
there RESTRICT=test is a fine solution. |
50 |
|
51 |
When instead a test suite should do a SKIP but erroneously does a FAIL, |
52 |
then RESTRICT=test is not the solution. Fixing the test suite is, but |
53 |
then that's not the maintainer's problem, but upstream's. Oddly enough |
54 |
we have QA checks in place (for ICEs, for instance) that direct users |
55 |
directly to upstream (through the HOMEPAGE variable), when it's |
56 |
suddenly the maintainer's problem if a package fails its test suite |
57 |
(because of FEATURES=userpriv or a missing kernel feature, perhaps - |
58 |
nothing the maintainer can prepare the user's system for). There's |
59 |
currently no way to describe the test suite's requirements to the user |
60 |
except by building in many checks or perhaps by simply listing them in |
61 |
an ewarn. |
62 |
|
63 |
Since the variable will be passed on to every next ebuild you're stuck |
64 |
with it, unless you are prepared to edit out RESTRICT=test, see if the |
65 |
new version still fails, then edit it in again when you find nothing |
66 |
has been fixed. This in turn doesn't make for easy ebuild maintenance. |
67 |
|
68 |
There's something wrong with the way we do test phases, the way some |
69 |
of us rely on them to foretell the stability or quality of a package |
70 |
version, and the way we stop test suites from failing (by |
71 |
only having a binary RESTRICT=test). |
72 |
|
73 |
We could easily extend metadata.xml to describe the test phase to the |
74 |
developer and user right now, for one thing, and aim for a more |
75 |
automated approach to fixing the binary problem in the future. |
76 |
|
77 |
Another solution is to change how emerge(1) and ebuild(1) deal with |
78 |
RESTRICT=test. On the one hand FEATURES=test should be enabled |
79 |
by testers and users with caution, as many test suites simply don't do |
80 |
what you might think they do - there is no guarantee that a program |
81 |
will run well in the Real World by merely satisfying its (limited?) test |
82 |
conditions. With that in mind, RESTRICT=test should perhaps not block |
83 |
testing the way that it does now. |
84 |
|
85 |
|
86 |
jer |