Re: [gentoo-dev] Re: Stabilisation procedure

[Rich Freeman]

On Thu, Nov 17, 2016 at 4:37 AM, Michael Palimaka <kensington@g.o> wrote:
> On 17/11/16 20:16, Rich Freeman wrote:
>> On Thu, Nov 17, 2016 at 2:16 AM, Michael Palimaka <kensington@g.o> wrote:
>>> * A leaf package such as {{package|kde-apps/kcalc}} may not require any
>>> runtime testing at all
>>
>> I'm not really a big fan of this, but if we REALLY didn't want to do
>> any runtime testing on a package then we should have some way to tag
>> the package as such, and then have some way to do automated
>> build-test-only stabilization.  If you aren't doing runtime testing
>> then manual stabilization adds zero value.
>
> How much value do you think we gain from runtime testing a package like
> kcalc as part of the stabilisation process, considering that it already
> sat in ~arch for at least 30 days?

We ensure that it actually runs at all with non-~arch dependencies?

The 30 days spent in ~arch tells you very little about whether the
package works with stable dependencies, since only those running mixed
keywords would be testing that.

>
> Also, based on conversations with various arch team members, my
> understanding is that a lot of stabilisations that happen right now are
> already build-only.
>

Certainly this isn't the documented process and it is the first I've
heard of this.

I think one of two things make sense:

1.  Manual runtime testing followed by stabilization.
2.  Automated build testing followed by stabilization, with no human involved.

What doesn't make sense is manual build testing.  The person is adding
zero value.

>>
>> Overall though the writeup was good and maybe it will trigger some
>> discussion.  I tend to think that if we want to do things like testing
>> permutations and such then automated build-only tools might be the way
>> to address this.  Manual effort should be focused on things like
>> runtime testing where it adds the most value.  This also strikes me as
>> the sort of thing that could probably be assigned out to volunteers
>> who do not have commit access.
>
> There's a few tools for this out there already. I've personally been
> working to update app-portage/tatt for git - see
> https://asciinema.org/a/cqsy983t9jimszvypcjr3zg5m for a demo.
>

Assuming we decide we don't care about runtime testing (which I'm not
sure I'm a fan of), it sounds like the only thing this is missing is:
1.  Running as a service on Gentoo infra without any person at the keyboard.
2.  Automatically monitoring the bug queue for anything that can be
stabilized, taking into account blockers/dependencies/etc.
3.  Posting failures to the bug, and then removing that bug from the
queue until somebody marks it as ready to go back in.
4.  If there is a success updating the bug (including closing if the
last arch) and doing the commit using an infra account.

However, I'd be interested in metrics on failures discovered in
runtime testing and so on, or missed with a lack of runtime testing.
I'll admit that a lot of runtime testing tends to be fairly shallow,
but I do think there is something to be said for doing some kind of
runtime testing.

I think we need to think about why we actually have a stable branch.
Does it offer any value if all we do is build testing, when I'm sure
the maintainers at least build their packages in ~arch before they
commit them?

-- 
Rich