[gentoo-dev] [RFC] Future of gentoo's stable and unstable trees: what are your thoughts?

[Sergei Trofimovich]

TL;DR;TL;DR:
------------

This email seeks for one step towards less toil tied to gentoo's
keywording/stabilization process. I've CCed a few groups who
might be interested in making this area better:

- gentoo-dev@ as it affects most devs (and non-devs!)
- wg-stable@ as it overlaps quite a bit with efforts staged on STABLEREQ
  (should I join? :)
- arch-leads@ as it directly helps (or breaks everything for) arch teams
- all individual arches for wider visibility

TL;DR
-----

I see the problem of lagging stable and unstable trees as:

1. lack of automation
2. lack of manpower

The PROPOSAL to solve the '1. automation' part is to draft
a new GLEP. If there is any interest (I assume there is!) I'll start one.
Let's call that fufure 'life of KEYWORDS'. It will cover:

- Update on GLEP-40 ("x86 stabilization can do only x86@ team")
  to allow package maintainers to do ARCH=x86 stabilization.
  It will be an arch-agnostic way: each arch will have minimal requirements
  to setup environment suitable for stabilization and keywording:
  CFLAGS to have, hardware required, whatever else is practical.
- Formalize list of stable arches as such (will be covered by
  'arches.desc' GLEP)
- Formalize what is a "stable arch". In short:
  - arch is marked as such in 'arches.desc'
  - performs most of STABLEREQ/KEYWORDREQ or gives rationale
    why progress can't be easily done before 90-days timeout
- Formalize and automate process of dropping keywords for timed out
  STABLEREQ/KEYWORDREQ requests.
- Automate process of restoring dropped KEYWORDS due to bumps
  adding new unkeyworded dependencies. repoman already complains
  about those. What is left is to grab them in batches time to time
  and handle those as if those were KEYWORDREQ.
- File more automated STABLEREQs to rely less on lazy maintainers
  (I am example of lazy maintainer not siling STABLEREQs enough).
- Formalize which STABLEREQ/KEYWORDREQ can be done automatically
  by arch teams (or maybe anyone else having the hardware!).
  In short: anything not marked as "Runtime testing required"
  on bugzilla and not having any blocker bugs.

The proposal to solve the '2. manpower' part is:
- Write more docs and make stabilisation process easier for everyone.

Important detail: the list is not in set in stone but rather a guideline
of things to cover.

Please feel free to propose other topics, questions, concerns, likes or
any other actionable feedback!

Story mode
----------

Hi all!

As you might suspect arch testing (an important process part of
gentoo's health). Recently arch testing became a point of contentions
among gentoo developers.

The non-exhaustive list of complaints is:

- Minor (usually understaffed) arch ${A} is slow and does not process
  STABLEREQ/KEYWORDREQ packages for many months. Lag forces maintainers
  to keep more old versions in the tree prohibiting removal of not
  really-maintained packages.

  In theory policy allows dropping stable keywords from such packages
  but in practice people frequently do not do it because it's a large
  and tedious tree-wide change.

- Packages on major arch (amd64 or x86) are not stabilized frequently
  enough. Packages fall through the cracks in bugzilla, STABLEREQs don't
  get filed by maintainers.

- <Add your problem here so we can understand and analyze it better>

What are the actual problems?
-----------------------------

- Arch testing is complicated, repetitive and (somethat) unrewarding.

  People only notice when things don't work.

- Arch testing is complicated: each architecture (or OS) has it's own
  quirks.

  It's hard for a developer to join empty arch team as documentation
  on specifics is frequently lacking.

  Some example questions could arise:

  - When keywording for ARCH=ppc64 should we test on both powerpc64 and
    powerpc64le or only one of them?

  - Does -Wl,--as-needed work on ia64?

- There seem to be a little of coordination between arch teams which
  tools to use to handle stabilization pipeline.

  Some examples:

  - Q: How to grab a list for keywording?

    A: use 'getatoms.py' from https://wiki.gentoo.org/wiki/Package_testing#Tools
       but there is a few caveats.

  - Q: How to keyword a package before stabilization?

    A: $ ekeyword ~ia64 <a long-list-of-packages>

    Caveat: can easily drop a keyword from ia64 down to ~ia64 for a
    package or two causing tree breakage.

  - Q: How to mark a package as explicitly not working or an ${arch}?

    A: Use KEYWORDS="-${arch}" but bots and ekeyword ignore it.

  - Q: How to mark a single package version as broken for a particular
       arch?

  - Q: What is the format of commit message for arch commit? Should
       package version be there? Should it be one commit per package?
       Per bug? Per arch?

    A: Everyone does it their own way.

What can we do about it?
------------------------

The short answer is: we need more automation and more manpower.

Both are related: in an ideal world robots would do all package testing
for us. People would only need to add new packages into system.

More detailed plan (actions and changes)
----------------------------------------

The plan is to get feedback from gentoo community and form a new
GLEP that will supersede GLEP-40.

We will need input from many active members: arch teams, wg-stable team
and gentoo devs.

The questions that will be covered in the new GLEP:

1. Q: What is a supported stable arch?

   A: - Arch is marked 'stable' in 'profiles.desc' (and upcoming
        'arches.desc')
      - Arch that has at least one team member willing to do
        STABLEREQ/KEYWORDREQ processing
      - The STABLEREQ/KEYWORDREQ request timeout is 90 days

        After timeout expires all keywords for requested package
        will be demoted. To avoid tedious manual de-keywording we
        will need tooling for that.

2. Q: How to make arch testing faster and easier?

   A: - KEYWORDREQ/STABLEREQ bugs not marked as "runtime testing
        required" will be automatically tested and keyworded.

        [handwave] automated tinderbox setup would help a lot
        to now upfront what fails to built, fails tests.

3. Q: How to programmatically get "readiness" status for a particular
      package? Say, how many STABLE-blocking bugs the package has?

   A: Suggestions welcome! Maybe add "package list" field for bugs?

4. Q: How to push more packages into STABLE?

   A: File automatic STABLEREQ bugs more aggressively if no known bugs
      exist for a package version. The rough workflow is the following:

      - Grab a list of candidates for stabilization (will need
        additional tooling)
      - File STABLEREQ against maintainer only first.
      - Maintainer will have a 2-week timeout to either proceed with
        request:
        - add "runtime testing required fields", CC relevant arches to
          start stabilization
        - or set blocking bugs against STABLEREQ to stop stabilization
      - After 2-week timeout tooling automatically CCes arches and
        stabilization happens (ideally with minimal manual work)
      - Profit!

5. Q: <you questions you always wanted to ask>

What do you think of all that?
==============================

Can this proposal make a difference and make gentoo better and easier to
work with?

Does it try to attack the right thing?

Does it completely miss the point?

Does it sound fun?

Please do share your thoughts!

Thank you!

--

  Sergei