1 |
Piotr Jaroszyński wrote: |
2 |
> Hello, |
3 |
> |
4 |
> There was some discussion about forcing/not forcing tests in EAPI-1, but there |
5 |
> was clearly no compromise. Imho, tests are very important and thus I want to |
6 |
> discuss them a little more, but in more sensible fashion. |
7 |
> |
8 |
> Firstly each test can be(not all categories are mutually exclusive): |
9 |
> - not existant |
10 |
> - non-functional |
11 |
> - not runnable from ebuild |
12 |
> - useful but unreasonable resource-wise |
13 |
> - useful and reasonable resource-wise |
14 |
> - necessary |
15 |
> - known to partially fail but with a way of skipping failing tests |
16 |
> - known to partially fail but with no easy way of skipping failing tests |
17 |
> Is that list comprehensive? |
18 |
|
19 |
I've been running with FEATURES=test for a long time now. Here's some |
20 |
of the more interesting cases: |
21 |
|
22 |
- fail only on little/big-endian archs |
23 |
- fail only with/without root privs |
24 |
- fail only if dependencies are / are not compiled with certain optional |
25 |
support |
26 |
- fail only with GCC >=4 |
27 |
- are expected to fail and are only meant as a regression test |
28 |
- take 3 minutes on x86 and 3 hours on mips |
29 |
- fail on hardened |
30 |
- fail with/without new tar versions |
31 |
- fail with/without new flex versions (etc.) |
32 |
- fail if a kernel component is a module instead of built-in |
33 |
- fail if certain environment variables are set |
34 |
- fail if compiled with certain (safe) CFLAGS/CXXFLAGS |
35 |
|
36 |
Can we qualify each of these into one of your categories? (NB: I |
37 |
realize there are solutions for each of these examples. I'm pointing |
38 |
out that not only is the situation not black and white, it often ranges |
39 |
in the ultra-violet.) |
40 |
|
41 |
> Secondly we must answer the question how precisely we want to distinguish |
42 |
> them, so users/dev can choose which categories of tests they want to run. |
43 |
> What comes to mind is: |
44 |
> - run all tests |
45 |
> - run only necessary tests |
46 |
> - run only reasonable tests |
47 |
> - don't run tests at all |
48 |
> Again, is that list comprehensive? |
49 |
|
50 |
- run only tests that don't require extra deps |
51 |
- run only tests that work on hardened |
52 |
- run only tests that work on my arch |
53 |
|
54 |
> Please don't post solutions unless we figure out which options we really want |
55 |
> to deliver. |
56 |
|
57 |
Sorry. (neener neener) ;) Would people accept running src_test() by |
58 |
default only on packages in the system set? There are some that we |
59 |
might want to turn off - glibc, gcc, binutils, autoconf, and automake |
60 |
are on my current short list. coreutils is also a lot of fun. db takes |
61 |
six hours. |
62 |
|
63 |
Anyone, however, who is of the opinion that tests for their packages are |
64 |
so important that they should never be skipped, and who is willing to |
65 |
deal with the bug reports that will undeniably be generated as a result, |
66 |
should IMO be allowed to shoot themselves in the foot, while those |
67 |
uninterested can go about their business without further interruption. |
68 |
|
69 |
In no way should this be tied to EAPI. |
70 |
|
71 |
|
72 |
-- |
73 |
where to now? if i had to guess |
74 |
dirtyepic gentoo org i'm afraid to say antarctica's next |
75 |
9B81 6C9F E791 83BB 3AB3 5B2D E625 A073 8379 37E8 (0x837937E8) |
76 |
|
77 |
-- |
78 |
gentoo-dev@g.o mailing list |