[gentoo-dev] Re: RFC: USE=qa-test

[Peter Hjalmarsson]

ons 2009-10-07 klockan 16:46 +0200 skrev Gilles Dartiguelongue:
> Le mardi 06 octobre 2009 à 20:38 -0600, Ryan Hill a écrit :
> > Some packages, like dbus[1], have testing features that, while useful for
> > developers and arch-testers, aren't something that should be foisted on
> > users.  Dbus' case is extreme, as it builds-in functions that are useful for
> > unit testing, but result in an insecure and unstable package (I just "fixed" a
> > bunch of testsuite failures i've been seeing in dbus-using packages by
> > disabling USE=test). Other packages have testsuites that take an unreasonable
> > amount of time to build/run (db, ppl, boost, that faad/faac one that takes
> > six hours), are pretty much guaranteed to fail (gcc, binutils), have strange
> > dependency quirks (can't run the tests unless the package is already
> > installed, create circular dependencies), or a dozen other situations I can't
> > think of right now.
> >
> > I'd like to propose a new USE flag, qa-test or a better name, to handle these
> > cases in a consistent way.  This would give us a way to differentiate between
> > tests that everyone should run and tests that only devs and arch-testers
> > would be interested in, making enabling FEATURES=test by default in a future
> > EAPI a little more palatable.    Use of this flag would be up to the
> > maintainer, of course.
>
> while it might sound sane, I think this proposal covers too much cases,
> most of which should actually be filled as bugs to the maintainers of
> the packages for not fixing the testsuite (or not filling an upstream
> bug) before commiting to the tree.
>
> For gnome ebuilds as someone commented out, the test failure rate is
> quite stable, and we are slowly trying to get around them, or at least
> not commiting ebuilds with new regressions in the testsuite.
>
> Use of RESTRICT="test" shouldn't be encouraged as it disables tests
> completely while part of them might still work and be relevant.
>


The problem comes with packages like dbus where the testsuit is pretty
useful for ATs, but if dbus is merged with FEATURES="test" then other
testsuits for other packages depending on dbus breaks, which AFAICS
tells us that a dbus built with tests is pretty broken for general
usage.

Example:

FEATURES="test" emerge -1q dbus && FEATURES="test" emerge -1q
dbus-python
and dbus-python will FAIL its testsuit.

However
FEATURES="-test" USE="-test" emerge -1q dbus && FEATURES="test" emerge
-1q dbus-python
and dbus-python will compile, test and install without problems.

Now how do we ensure that the ATs still can test a package even under
these conditions?
Because as I see it RESTRICT is only for packages whose testsuits are
broken by design and never meant to be run on a working system (yeah
*drm*, I am looking at you).

I think we first hand should decide over these packages, after a
decision is made about these problems with MORE then a compile-time
impact, we can start talking about less important problems (i.e.
problems that do not break stuff) like how long time a testsuit is
supposed to run.

And FEATURES="test" IS sometimes a good thing, I learned this the hard
way once when something broke grub for me, and one of the things that
made my system unbootable was one of those things the grub testsuit
checks for.