Gentoo Archives: gentoo-dev

From: Jeroen Roovers <jer@g.o>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"
Date: Mon, 07 Jun 2010 12:03:09
In Reply to: Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue" by Thilo Bangert
1 On Mon, 7 Jun 2010 12:10:04 +0200
2 Thilo Bangert <bangert@g.o> wrote:
4 > i do agree, that all packages should build successfully including the
5 > test phase. RESTRICTing the test and an open bug when this is not the
6 > case.
8 I see more and more calls for either 1) "fixing the test suite", as if
9 that is suddenly not an UPSTREAM issue but the ebuilds' maintainers' or
10 2) setting RESTRICT=test, which would have everyone ignore the
11 useful aspects of a package's test suite.
13 The main problem with RESTRICT=test is that it's too restrictive - it
14 prevents a normal emerge(1) or ebuild(1) run from entering the test
15 phase at all, with no way to work around it, except by editing the
16 ebuild to remove the restriction.
18 ======
19 marga /keeps/gentoo/local/app-portage/foobar # ebuild foobar-1.ebuild
20 test Forcing test.
21 * checking ebuild
22 checksums ;-) ... [ ok ]
23 * checking auxfile
24 checksums ;-) ... [ ok ]
25 * checking miscfile
26 checksums ;-) ... [ ok ]
27 * CPV: app-portage/foobar-1
28 * REPO: JeR
29 * USE: elibc_glibc kernel_linux ppc test userland_GNU
30 >>> Unpacking source...
31 >>> Source unpacked in /var/tmp/portage/app-portage/foobar-1/work
32 >>> Compiling source in /var/tmp/portage/app-portage/foobar-1/work ...
33 >>> Source compiled.
34 * Skipping make test/check due to ebuild restriction.
35 >>> Test phase [explicitly disabled]: app-portage/foobar-1
36 marga /keeps/gentoo/local/app-portage/foobar # cat foobar-1.ebuild
37 # Copyright 1999-2010 Gentoo Foundation
38 # Distributed under the terms of the GNU General Public License v2
39 # $Header: $
41 SLOT="0"
42 RESTRICT="test"
43 ======
45 If you believe that test suites are a Good Thing, then you should view
46 RESTRICT=test as the ultimate solution to fix the problem of a broken
47 test suite that will never be fixed. Now, in case a test suite could
48 actually be destructive, dangerous to run on an unprepared system, then
49 there RESTRICT=test is a fine solution.
51 When instead a test suite should do a SKIP but erroneously does a FAIL,
52 then RESTRICT=test is not the solution. Fixing the test suite is, but
53 then that's not the maintainer's problem, but upstream's. Oddly enough
54 we have QA checks in place (for ICEs, for instance) that direct users
55 directly to upstream (through the HOMEPAGE variable), when it's
56 suddenly the maintainer's problem if a package fails its test suite
57 (because of FEATURES=userpriv or a missing kernel feature, perhaps -
58 nothing the maintainer can prepare the user's system for). There's
59 currently no way to describe the test suite's requirements to the user
60 except by building in many checks or perhaps by simply listing them in
61 an ewarn.
63 Since the variable will be passed on to every next ebuild you're stuck
64 with it, unless you are prepared to edit out RESTRICT=test, see if the
65 new version still fails, then edit it in again when you find nothing
66 has been fixed. This in turn doesn't make for easy ebuild maintenance.
68 There's something wrong with the way we do test phases, the way some
69 of us rely on them to foretell the stability or quality of a package
70 version, and the way we stop test suites from failing (by
71 only having a binary RESTRICT=test).
73 We could easily extend metadata.xml to describe the test phase to the
74 developer and user right now, for one thing, and aim for a more
75 automated approach to fixing the binary problem in the future.
77 Another solution is to change how emerge(1) and ebuild(1) deal with
78 RESTRICT=test. On the one hand FEATURES=test should be enabled
79 by testers and users with caution, as many test suites simply don't do
80 what you might think they do - there is no guarantee that a program
81 will run well in the Real World by merely satisfying its (limited?) test
82 conditions. With that in mind, RESTRICT=test should perhaps not block
83 testing the way that it does now.
86 jer