Re: [gentoo-project] Call for agenda items - Council meeting 2016-08-14

[Andrew Savchenko]

Hi all,

On Thu, 4 Aug 2016 11:24:43 -0500 William Hubbs wrote:
> On Thu, Aug 04, 2016 at 04:15:14PM +0200, Kristian Fiskerstrand wrote:
> > Dear all,
> >
> > the Gentoo Council will meet again on Sunday, August 14 at 19:00 UTC
> > in #gentoo-council on FreeNode.
> >
> > Please reply to this message on the gentoo-project list with any items
> > the council should put on its agenda to discuss or vote on.
>
> I feel that our stable tree is so far behind on all
> architectures that we are doing our stable users a disservice, so I
> would like to open up a discussion here, and maybe some policy changes
> at the next meeting.

Thank you for caring about this issue. I had similar thoughts
myself, but was too slow on writing e-mail :) IMO stable tree has
three problems: 
1) too old packages
2) too few packages
3) stabilization often takes too long, such stable packages are
broken or buggy, while their unstable versions are fixed and work
fine. (It is not possible to fix all bugs without version or
revision bump, thus stabilization is needed to fix many bugs.)

This results in fact that many users and some devs (including
myself) do not use stable at all.

> Ultimately, I think we need some form of automated stabilization, e.g.
> if a package version sits in ~ for 30 days and there are no blockers at
> that point, the new version should go automatically to stable on all
> architectures where there is a previous stable version.

Automation should be possible only after rigorous testing. If this
will be implemented, than this may be done (as Brian pointed out in
another mail in this thread, there is an ongoing effort on this
matter as GSoC project, which is great). But I'd like to emphasize
that automated stabilization without:
a) build testing;
b) run-time testing;
c) fixing all serious open bugs affecting package being stabilized.

Automation tests will definitely help here, but I'm not sure that
packages can be fully tested without human intervention: 
- how to automatically test GUI app run-time?
- how to determine which open bugs from bugzilla are serious enough
to block stabilization, and which bugs shouldn't block the process?

IMO automation can help with build tests, maybe limited run-time
testing, but no more.

> The first issue is maintainers not filing stable requests for new
> versions of packages in a timely manor. I'm not sure how to get around
> this, but I feel that once a version of a package is stable, we are
> doing a disservice to our stable users by not keeping stable as current
> as possible.  I am as bad as anyone; it is easy to forget to file
> stable requests until someone pings me or files the request
> themselves.

We have another problem: arch teams often can't keep up with
stabilization requests, they are hanging for months, I even have
several bugs open for more than half a year. Storming arch teams
with stabilization requests will make things even worse.

> I have heard other maintainers say specifically that they do not file
> stable requests unless a user asks them to, but Again, I do not feel
> comfortable with this arrangement if there is an old version of the
> package in stable. Users shouldn't have to ask for newer versions to be
> stabilized; this should be driven by the maintainers.

I usually stabilize versions when they are too old or I have a user
request. The idea is not to stabilize as often as unstable version
is updated, since arch teams are unable to keep up even with
request rate they have now.

> The second issue is slow arch teams. Again, by not moving packages from
> ~ to stable, we are doing a disservice to our stable users.

And here comes my idea. We need an approved policy of how to remove
packages from stable (I'll name this procedure as "dekeywording"
below). Right now we only have a mention in the devmanual, that arch
teams must be informed via a bug [1] on dekeywording. But we have no
determined time frame, nor policy about reverse dependencies.

I propose to empower maintainers to remove packages from stable if
arch team can't stabilize package within 3 months. Of course, time
counts from a point when there are no blockers for stabilization;
if arch team finds a problem blocking stabilization, timer is reset
and will start again only after maintainer will fix the issue.

So if 3 months have elapsed, maintainer should notify arch team and
all reverse deps that package will be removed from stable tree. For
reverse deps this means that they will be removed from stable as
well if dependency is hard or will have some USE flag(s) stable
masked if dependency is optional.

Now we have few questions:

1) Should all reverse dependencies be just CC'ed on original
dekeywording bug or should separates bugs be created for every
reverse dep package? Apparently reverse dep tree can be quite large.

2) For how long maintainer should wait after dekeywording bugs will
be created before taking actual action? I suppose 1 month should be
same (the same as for last-rites).

3) What to do if dekeywording affects @system or other widely used
packages?

4) Of course 3+1 month waiting period can be discussed and
tuned here.

The very idea of this proposal is to create a mechanism of removing
packages from stable if arch team can't keep up with them, so we'll
have a natural selection of packages in stable.

Now one more question comes: if package was removed from stable,
will be there any policies to keep it from entering stable for a
while? Obviously flipping packages to and from stable will make no
good for both users and arch teams. Probably we should ban such
packages from being requested for stabilization for a while (e.g.
for 3 months). Of course, exceptions can be made, e.g. if such
package becomes a hard dependency of @system package an so on.

> I can think of two ways we can improve our situation.
>
> We can allow maintainers to stabilize new versions of certain types of
>  packages on all arches where there is a previous version of the package stable
>  without filing stable requests. This would take a significant load off
>  of the arch teams.
>
> For packages that do not fit the first group, we could require stable
> requests, but allow maintainers to stabilize the new versions after a
> timeout (I would propose 30 days).
 
If maintainer has host(s) with stable tree where package can be
tested on arch in question, then yes, this should be done.
Otherwise we'll just add undertested packages in stable which
will undermine the very idea of stable.

[1] https://devmanual.gentoo.org/keywording/index.html

Best regards,
Andrew Savchenko