[gentoo-dev] Re: Stabilisation procedure

[Michael Palimaka]

On 17/11/16 22:56, Rich Freeman wrote:
> On Thu, Nov 17, 2016 at 4:37 AM, Michael Palimaka <kensington@g.o> wrote:
>> On 17/11/16 20:16, Rich Freeman wrote:
>>> On Thu, Nov 17, 2016 at 2:16 AM, Michael Palimaka <kensington@g.o> wrote:
>>>> * A leaf package such as {{package|kde-apps/kcalc}} may not require any
>>>> runtime testing at all
>>>
>>> I'm not really a big fan of this, but if we REALLY didn't want to do
>>> any runtime testing on a package then we should have some way to tag
>>> the package as such, and then have some way to do automated
>>> build-test-only stabilization.  If you aren't doing runtime testing
>>> then manual stabilization adds zero value.
>>
>> How much value do you think we gain from runtime testing a package like
>> kcalc as part of the stabilisation process, considering that it already
>> sat in ~arch for at least 30 days?
>
> We ensure that it actually runs at all with non-~arch dependencies?
>
> The 30 days spent in ~arch tells you very little about whether the
> package works with stable dependencies, since only those running mixed
> keywords would be testing that.

What is the *real* risk that kde-apps/kcalc builds against stable
dev-libs/gmp but then starts producing funny numbers at runtime?

Let's put it another way - assume we're stabilising a new version of
dev-libs/gmp instead. Should we test already-stable kde-apps/kcalc
first? What about the other hundred reverse dependencies?

Just to be clear, I'm not advocating banning runtime testing. I just
think that, considering the state of the stable tree, we should consider
very careful in which situations we actually gain value from it. That's
for another thread, however. I deliberately worded the document so that
the final decision on the exact level of testing required (runtime or
otherwise) is between the person filing the stabilisation request and
the person actioning it.

>> Also, based on conversations with various arch team members, my
>> understanding is that a lot of stabilisations that happen right now are
>> already build-only.
>>
>
> Certainly this isn't the documented process and it is the first I've
> heard of this.

Indeed it is not documented, but then again not a lot about
stabilisation is well documented.

Consider bug #584468, a typical bulk stabilisation request from the
GNOME team consisting of 188 packages of varying types. I very much
doubt each arch team member sat down and tested file-roller to make sure
it extracts archives with stable libarchive and looks OK with stable
GTK, then makes sure gnucash-docs still looks pretty with stable
docbook-xsl-stylesheets, and so on 186 more times.

> However, I'd be interested in metrics on failures discovered in
> runtime testing and so on, or missed with a lack of runtime testing.
> I'll admit that a lot of runtime testing tends to be fairly shallow,
> but I do think there is something to be said for doing some kind of
> runtime testing.

I'd be interested in metrics like this too, but I'm not sure how we'd
collect such information.

Runtime testing is important, and in an ideal world every package would
receive detailed runtime testing before being moved to stable. Where we
are now, I think forcing runtime testing in every case costs more than
we gain.

Building and running for as long as possible the next stable glibc has a
comparatively low cost compared to the pain if it broke. Manually
running 300 kde-apps/* packages every three months is a guaranteed waste
of time. Most packages probably sit somewhere in between.

> I think we need to think about why we actually have a stable branch.
> Does it offer any value if all we do is build testing, when I'm sure
> the maintainers at least build their packages in ~arch before they
> commit them?

The whole point of the 30 day waiting period is that issues can and do
appear that the maintainer did not encounter when they bumped.