Re: [gentoo-dev] Proposal: profiles/arches.desc - improve repoman flexibility (with other benefits)

[Brian Dolbec]

On Sun, 26 Mar 2017 20:54:51 +0200
"Andreas K. Huettel" <dilfridge@g.o> wrote:

> Hey all,
>
> here's the updated proposal for a new "arches.desc" file, after
> feedback.
>
> In short:
> We introduce a new file "arches.desc" which essentially describes if
> an arch (not a profile) is stable or not. The meaning of
> profiles.desc is not affected; profiles.desc still describes single
> profiles of each arch. This is introduced in particular to help
> moving arches from stable to testing while keeping the "~arch"
> deptree consistent, or moving arches from testing to stable and
> easily preparing a consistent "arch" deptree.
>
> Details:
>
> 1] File location:
> profiles/arches.desc    or   metadata/arches.desc
>
>
> 2] File format: whitespace-separated, two columns, lines starting
> with # are comments
> first column: arch name ("amd64")
> second column: one of the three values "stable", "testing", "unstable"
> Example:
> ===========================
> # Example arches.desc file
> amd64	stable
> mips	testing
> m68k	unstable
> ===========================
>


While this is a simple format.  This is not a standard data input file
format for language tools to map it into native language variables and
types.

I would much prefer for any new files to be created in a format that
most languages have data input modules for and are easily read/edited
by humans.  While this can be read and split easily in python code.  It
is not future proof for additional data being added and/or removed.

 For the repoman stage3 rewrites. I am moving all configurable data
 from the code into yaml based files in /metadata/repoman.  These files
 will be easily edited by all developers for updates to banned eclasses
 and various other values not needing code changes.   

So with a general file name of arches.desc  Is there any other data
that we want to include in that file?  Possibly migrated from other
file(s).  In that case a dictionary format yaml file might be best.
My example below has additional info over what was proposed.  
It is an example only to show the possible benefit of such a file type.

  ie:
-----------------------------------------------------------------------
---
# the --- in the first line indicates this as a yaml file
# comments are allowed as well as whitespace
# DATA format for arches is dictionary

amd64:
    stability: stable
    bits: 64
    description:  Includes CPU manufaturers such as Intel, AMD, others...
    comments: The most common/popular arch in the tree...
    email: amd64@...

mips:
    stability: testing
    bits: 32
    description: Risc based achitecture ... (not fact checked)
    comments: Primarily used in some laptops and single board development systems...
    email: foo@...

m68k:
    stability: unstable
    bits: 8
    description: Motorola...
    comments: embedded...
    email: embedded@...
EOF
-----------------------------------------------------------------------------
The above example makes it easily modified to include new / remove obsolete data in the future.
The above list could add or remove the email field without affecting the proposed use of the 
stability field used by any applications.  That is not the case with a position dependent 
space separated list as was proposed.

Nearly all language types have native parsers for this type of file (no custom parser code required).
the above file info can easily be manipulated into various lists such as archlist, stable_arches, testing_arches,...
or any other data that is included for them.

>
> 3] Meaning of the three values "stable", "testing", "unstable" for
> repoman
>
> * stable: When a profile of arch is tested, then repoman checks
> consistency for "arch" and for "~arch" separately.
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and -d / -e switches).
> This is the current behaviour and should be the default if nothing is
> specified for an arch.
>
> * testing: When a profile of arch is tested, then repoman treats
> "arch" as "~arch", and tests consistency only for "~arch".
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and -d / -e switches).
> A new switch for repoman may be provided to temporarily upgrade an
> arch from "testing" to "stable" (for arch team work).
>
> * unstable: When a profile of arch is tested, then repoman treats
> "arch" as an error and aborts. Consistency is only tested for "~arch".
> Which profiles of the arch are tested is still controlled by
> profiles.desc (and -d / -e switches).
>
> * optionally, additional value "broken":
> Do not test this arch at all. (In my opinion not necessary, since we
> can always set all profiles of an arch to exp.)
>
>
> 4] Meaning for other tools
> All arches set to "stable" are considered "stable arches", meaning
> * they get listed first in eshowkw
> * they require stabilization requests in bugzilla
> * ...
>
>
> 5] Initial value in the Gentoo repository
> On introduction, the setting will be "stable" for all stable arches,
> "testing" for all arches where "inofficial" stable keywords exist
> (sh, s390, ...), and "unstable" everywhere else.
>
>
> 6] Rationale
> At the moment we have several cases where repoman finds its limits:
>
> a) An arch loses its stable status (imagine s390), but the arch team
> doesnt want to drop all the stable keywords since they are useful for
> stage building. Since stable keywords on s390 are an
> arch-team-internal thing, everyone can drop the latest stable version
> of a package, and it's the arch team's responsibility to keep their
> "unofficial stable tree" straight. Right now this means that we have
> to set all *profiles* of this arch to "exp", otherwise repoman throws
> errors about a broken stable deptree. If we do that, repoman does
> however also not check ~s390 consistency, meaning that the ~s390
> deptree will soon be broken as well. With arches.conf, one could set
> (arch: testing, profile: stable), which results in stable keywords
> being ignored, but the ~s390 deptree still being enforced.
>
> b) An arch prepares for becoming a stable arch (think arm64).
> So, first the ~arm64 deptree should be fine, and then stable keywords
> can be added. However, to avoid repoman errors as long as the stable
> deptree isnt complete yet the profiles need to be set to dev/exp, and
> again this brings the danger of the ~arm64 deptree getting
> inadvertently broken. Again the combination (arch: testing, profile:
> stable) helps.
>
> Finally, at the moment the "semi-official" algorithm to figure out if
> an *arch* is stable (i.e., requires stabilization requests etc bla
> bla) is to check if the arch has any stable profile. This makes
> non-stable arches which want to keep a consistent deptree (think
> mips) impossible. Reading the arch status from arches.desc instead
> solves this problem.
>
>
> 7] Compatibility
>
> a) No arches.desc and new system, or arch not listed in arches.desc
> Arches are treated as "stable" by repoman (current behaviour), with
> profile status according to profiles.desc. So, business as usual.
> Gentoolkit and other tools trying to determine a list of stable
> arches should fall back to current method of scanning profiles.desc
> for stable profiles.
>
> b) arches.desc and old system
> Tools ignore the unknown file (?).
> Repoman and other tools may emit surplus dependency errors when
> profiles are checked on arches that are "testing" (they check the
> consistency of the stable tree alone, which is not OK, since "arch"
> is supposed to be treated like "~arch"). This affects only
> development work and can be fixed by updating repoman.
>
> c) PMS may need to be amended to allow the additional file.
>
>
> 8] Several repositories
>
> If arches.desc is present in several repositories, then the strictest
> setting for an arch wins. [I don't really see many usecases for this
> though.]
>
>
> Congratulations for getting this far. What's your opinion?
>



-- 
Brian Dolbec <dolsen>