Re: [gentoo-dev] tests

[Maurice van der Pot]

On Tue, May 01, 2007 at 03:08:56PM +0200, Piotr Jaroszyński wrote:
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?

Isn't it easier to list a set of boolean properties of _individual_
tests?  
- We don't need "non existent". 
- Non-functional and known to partially fail come down to "known to
  fail" for individual tests.  
- I have no idea what "not runnable from ebuild" is.  
- Unreasonable could be "resource hungry" or "needs additional deps"
  (those are two different things).
- If a test is "necessary" I don't see why we should allow it to be
  skipped.
- And about skipping failing tests. There is always a way to skip
  failing tests: not running any of them. It's just the granularity that
  is different, but the user doesn't care about that.

> Secondly we must answer the question how precisely we want to distinguish
> them, so users/dev can choose which categories of tests they want to run.
> What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?

I'd say, let the user decide based on the properties, fex:

run known to fail  : no
run resource hungry: yes
run additional deps: no

if ( (known to fail   == false || run known to fail   == true) &&
     (resource hungry == false || run resource hungry == true) &&
     (additional deps == false || run additional deps == true) ) 
{
  run the test
}

So for each test/set of tests and for each possible reason not to run
it, either the reason does not apply to this test or the user explicitly
said to run it anyway.

You don't see the "way of skipping failing tests" in this last part.
This is because if the decision is made not to run a test, the smallest
set that can be skipped (possibly all tests for an ebuild) are skipped.

> Please don't post solutions unless we figure out which options we really want
> to deliver.

I'm sorry if this counts as a solution, but I wasn't sure the list you
gave was the kind of list we need and I didn't know another way of
explaining the use of the properties I listed.

Regards,
Maurice.

-- 
Maurice van der Pot

Gentoo Linux Developer   griffon26@g.o     http://www.gentoo.org
Creator of BiteMe!       griffon26@××××××××.com   http://www.kfk4ever.com