| 1 |
On Sun, 2005-09-04 at 20:12 -0500, Daniel Goller wrote: |
| 2 |
> sounds like you suggest to trick ~arch users into testing "unripe" |
| 3 |
> ebuilds/bumps/versions by sending it into ~arch to get the testing done while |
| 4 |
> someone in a chroot would be much better equipped for doing the testing with? |
| 5 |
|
| 6 |
No. |
| 7 |
|
| 8 |
You've got to look at it in the context of the environment we work in. |
| 9 |
|
| 10 |
The vast majority of our packages do not come with test strategies, |
| 11 |
comprehensive test scripts, automated test cases that provide a large |
| 12 |
coverage of the code base, or any other QA tools that a career software |
| 13 |
QA person would expect / desire. We don't have activities in place to |
| 14 |
fill this gap - it's probably beyond our very limited resources anyhow. |
| 15 |
We don't require (and don't cover in the quizzes) devs to have any |
| 16 |
training or experience of deterministic software testing, or of |
| 17 |
regression testing. It is not our policy to require devs to write |
| 18 |
regression tests for every valid bug posted in Bugzilla (or for any bug |
| 19 |
at all, for that matter). |
| 20 |
|
| 21 |
A package maintainer does what testing he can, and there are times when |
| 22 |
we'd all wish the maintainer had done more :) But there will always be |
| 23 |
the need for a wider audience to be encouraged to test the package. A |
| 24 |
good rule of thumb is that a QA budget should be 40% of the development |
| 25 |
budget. We don't have the manpower to come anywhere near that. |
| 26 |
|
| 27 |
We have only a fraction of the resources of Debian or RedHat - there |
| 28 |
comes a point where we have to make up the difference *somehow*. For |
| 29 |
better or for worse, historically in Gentoo we've done it by turning to |
| 30 |
our users. |
| 31 |
|
| 32 |
Many *users* look on package.masked packages as being dangerous to |
| 33 |
install, but are much more willing to run ~arch packages. If you mask a |
| 34 |
version of a popular package, you'll get a lot of correspondence asking |
| 35 |
you when you'll unmask it, but you won't get much testing feedback to go |
| 36 |
with it. Once the package moves to ~arch, the amount of feedback |
| 37 |
improves substantially. |
| 38 |
|
| 39 |
If a package maintainer believes that a package WORKSFORME *after due |
| 40 |
diligence*, then he's not only entitled to move it to ~arch, but he's |
| 41 |
got no reason not to. |
| 42 |
|
| 43 |
I've been through this first hand over the last 14 months with PHP 5. |
| 44 |
It's a big package, one that I use all day every working day, and a |
| 45 |
topic I co-wrote the official certification study guide for. It's fair |
| 46 |
to say that it's a package I know a lot about, and that others in the |
| 47 |
community recognise I know well. But, with over 100 separate features |
| 48 |
to enable and disable, even over 14 months there are large parts of the |
| 49 |
package that I won't have tested in depth, and there will always be |
| 50 |
features that I'll never touch. |
| 51 |
|
| 52 |
I kept PHP5 masked for those 14 months, and (as Jakub and others can |
| 53 |
confirm) most of the feedback has been limited to "unmask that |
| 54 |
puppy" (sometimes put in stronger terms ;-) There were some bugs from |
| 55 |
users who had found issues, but not many. |
| 56 |
|
| 57 |
Rather than unmask the packages before they were read, I changed to |
| 58 |
another approach. I moved the work out of Portage into an overlay |
| 59 |
instead. This worked well. It has attracted a bunch of regulars to |
| 60 |
#gentoo-apache who have spent the last few months finding the bugs that |
| 61 |
existed, and making sure that they're fixed and stay fixed. It looks |
| 62 |
likely that we'll get some new devs out of that too :) |
| 63 |
|
| 64 |
Overlays are easy for larger pieces of work like PHP, Java, |
| 65 |
Gnome/Gentopia, but they'll always be small packages where an overlay |
| 66 |
feel like too much effort. They may not be the answer to everything. |
| 67 |
|
| 68 |
If we'd seen through the policy that every package has to belong to a |
| 69 |
herd, then we could organise overlays by herd - and maybe leave it up to |
| 70 |
the arch teams to import "stable" packages from the overlays into the |
| 71 |
Portage tree proper. |
| 72 |
|
| 73 |
Best regards, |
| 74 |
Stu |
| 75 |
-- |
| 76 |
Stuart Herbert stuart@g.o |
| 77 |
Gentoo Developer http://www.gentoo.org/ |
| 78 |
http://stu.gnqs.org/diary/ |
| 79 |
|
| 80 |
GnuGP key id# F9AFC57C available from http://pgp.mit.edu |
| 81 |
Key fingerprint = 31FB 50D4 1F88 E227 F319 C549 0C2F 80BA F9AF C57C |
| 82 |
-- |
Archives