Re: [gentoo-dev] [pre-GLEP] Repository configuration file (layout.conf)

[Michał Górny]

On Wed, 2021-05-19 at 11:22 -0400, Wolfgang E. Sanyer wrote:
> On Wed, May 19, 2021 at 11:02 AM Michał Górny <mgorny@g.o> wrote:
> >
> > On Wed, 2021-05-19 at 10:37 -0400, Wolfgang E. Sanyer wrote:
> > > See some comments in-line below.
> > >
> > >
> > > On Wed, May 19, 2021 at 8:33 AM Michał Górny <mgorny@g.o> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Please review the pre-GLEP inlined below.  Its purpose is to formally
> > > > define the format of layout.conf.  It's pretty much inevitable these
> > > > days, so we should specify it.  However, it doesn't really fit into PMS,
> > > > and other formats (Manifests, metadata.xml) are already defined
> > > > in GLEPs, so this just follows suit.
> > > >
> > > > Pre-GLEP follows.
> > > >
> > > > ---
> > > > GLEP: 9999
> > > > Title: Repository configuration file (layout.conf)
> > > > Author: Michał Górny <mgorny@g.o>
> > > > Type: Standards Track
> > > > Status: Draft
> > > > Version: 1.0
> > > > Created: 2021-05-19
> > > > Last-Modified: 2021-05-19
> > > > Post-History: 2021-05-19
> > > > Content-Type: text/x-rst
> > > > ---
> > > >
> > > > Abstract
> > > > ========
> > > >
> > > > The ``metadata/layout.conf`` file format is specified as used by Portage
> > > > and PkgCore.  A standard set of configuration keys is described
> > > > including the keys currently used in the Gentoo repository.
> > > >
> > > >
> > > > Motivation
> > > > ==========
> > > >
> > > > The ``metadata/layout.conf`` file was first added to the Gentoo
> > > > repository in Oct 2011, to facilitate setting of hashes used
> > > > in Manifest2 files.  In Mar 2012, it was used to indicate the transition
> > > > to the new ``md5-dict`` cache format.  In Jul 2013, it started being
> > > > used to indicate the repository's masters and effectively became
> > > > obligatory for all repositories.
> > > >
> > > > Today, ``layout.conf`` is used for various repository configuration
> > > > knobs that can be expressed as simple values and therefore
> > > > do not justify adding new files to the repository.  This primarily
> > > > involves the configuration of development tools but also includes a few
> > > > keys relevant to the behavior of the package manager.
> > > >
> > > > However, ``layout.conf`` is currently not covered by any formal
> > > > specification.  The PMS neglects its existence entirely, and the keys
> > > > used are roughly defined by their first use of Portage or PkgCore.
> > > > This GLEP aims to overcome this by providing a formal specification
> > > > for the file, as well as an up-to-date list of permitted configuration
> > > > keys.
> > > >
> > > >
> > > > Specification
> > > > =============
> > > >
> > > > layout.conf file format
> > > > -----------------------
> > > >
> > > > Every ebuild repository must contain a ``metadata/layout.conf`` file.
> > > > The file uses a line-oriented text format.  Lines starting with ``#``
> > > > represent comments and are ignored, as are lines consisting entirely
> > > > of whitespace.  The remaining lines must contain a key followed
> > > > by equals sign (``=``), followed by a (possibly empty) value.  Each of
> > > > these elements can be surrounded by additional whitespace that
> > > > is stripped.
> > >
> > > Is a space around the "=" optional? I see it used below, but here it sounds
> > > like there should be no space.
> >
> > The last sentence says you can add extra space and it will be stripped.
> > If you can think of a better way of wording that (I really wanted to
> > avoid 'optional whitespace, followed by key, followed by optional
> > whitespace... ' ;-).
>
> 🤦‍♂️ ofc I would miss that.
>
> I always thought that python did a pretty good job
> [explaining their format mini-language][1]. That's obviously overkill for what
> you're describing here, but maybe something like this would be helpful?
>
> # comments are allowed
> <key> = [<value1> [<value2>...]]
>
> Any lines starting with a ``#`` represents a comment and are ignored, as are
> lines consisting entirely of whitespace.
>
> A valid line consists of key and an equal sign. (A valid key is a
> string containing
> any character except for a space)[^1]. The list of space-separated
> values is optional.
>
> Note: any additional spaces are ultimately stripped
>
> [1]: https://docs.python.org/3/library/string.html#format-specification-mini-language
> [^1]: Should we include this or nah?

Thanks, this looks like a nice idea.  I'm going to send a v2 shortly
and we'll see how it worked out.

>
> > >
> > > > Configuration keys
> > > > ------------------
> > > >
> > > > The ``layout.conf`` file must contain the ``masters`` key.  Other keys
> > > > listed in this specification are entirely optional.  The package
> > > > managers may choose to implement a subset of listed keys.  Unknown keys
> > > > must be ignored.
> > > >
> > > > The following keys are currently defined:
> > >
> > > Is the intent for the GLEP to define and specify every possible key-value
> > > combination? If not, then perhaps this section should be truncated a bit? i.e.
> > > "here are some examples of currently defined keys" and then list a handful.
> > >
> > > Additionally, maybe the spec should include some sort of requirement for
> > > documentation of variables, similar to what you've done below.
> > >
> > > Edit: ok, nevermind I see your recommendation at the bottom. But still,
> > > I would probably either (1) insist that new keys be added to this GLEP, or
> > > (2) insist that new keys are documented properly.
> >
> > Let's put it like this: the goal is to document all the useful keys.
> > I can live with some ancient-not-really-useful keys not being documented
> > (e.g. Portage has some stale/redundant Manifest-related keys, I think).
>
> Perhaps then some sort of use of the word "reserved" should be in here?
> i.e. "The following keys are reserved for use by portage and pkgcheck.
> any other keys may be defined and used at will"
>
> > >
> > > > masters = <space-separated values>
> > > >   Specifies the master repositories of this repository.  For stand-alone
> > > >   repositories, this must be set to an empty value.  Otherwise, it can
> > > >   list one or more repositories, separated by spaces.  This key must
> > > >   be specified.
> > >
> > > I realize this is the only compulsory key, but perhaps it should still go in
> > > a separate section in order to ensure that it stands out? i.e. "The following
> > > keys are mandory..." and then later "the following optional keys are
> > > currently defined"
> >
> > I'm not sure, to be honest.  I didn't really want to add a section for
> > one key ;-).
>
> np, just a thought.
>
> > >
> > > > manifest-hashes = <space-separated values>
> > > >   Specifies the list of hashes that should be used for new distfiles
> > > >   in the Manifest files.  The development tools may create a subset
> > > >   of the specified hashes if it is not updating the checksums for
> > > >   the specified distfile, or does not support the hash in question.
> > > >   The hash names are specified in GLEP 74.  [#GLEP74]_  The default
> > > >   set of hashes is implementation-defined.
> > > >
> > > > manifest-required-hashes = <space-separated values>
> > > >   Specifies the list of hashes that must be used in Manifest files.
> > > >   The development tools must support all the hashes listed there,
> > > >   and update distfile checksums to use these hashes (refetching
> > > >   if necessary).  This must be a subset of manifest-hashes.  If not
> > > >   specified, all hashes from manifest-hashes (or the default set)
> > > >   are considered required.
> > > >
> > > > use-manifests = ``strict``, ``true`` or ``false``
> > > >   Indicates the policy for creating and using Manifest files.  If set
> > > >   to ``strict``, Manifest files are created and files are required to
> > > >   match digests found in Manifests.  If set to ``true``, Manifests
> > > >   are created but digest mismatches are ignored.  If set to ``false``,
> > > >   Manifests are not used at all.  The default is ``strict``.
> > > >
> > > > update-changelog = ``true`` or ``false``
> > > >   Indicates whether the development tools should write ChangeLog files.
> > > >   The default is ``false``.
> > > >
> > > > cache-formats = <space-separated values>
> > > >   Specifies one or more cache formats used by the repository.
> > > >   The currently defined values are ``pms`` for the original format
> > > >   specified in the PMS and ``md5-dict`` for the md5-dict format
> > > >   introduced in Portage 2.2.0_alpha68.  The default is
> > > >   implementation-defined.
> > > >
> > > > eapis-deprecated = <space-separated values>
> > > >   Specifies one or more EAPIs that are to be considered deprecated
> > > >   by the development tools for use in ebuilds, i.e. their use should
> > > >   trigger a warning.  If not specified, no EAPIs are deprecated.
> > > >
> > > > eapis-banned = <space-separated values>
> > > >   Specifies one or more EAPIs that are to be considered banned
> > > >   by the development tools for use in ebuilds, i.e. their use should
> > > >   be blocked.  If not specified, no EAPIs are banned.
> > > >
> > > > repo-name = <string>
> > > >   Specifies the repository name.  If specified, it must be equal
> > > >   to the contents of ``profiles/repo_name``.  If not specified,
> > > >   it defaults to the same value.  Discouraged.
> > > >
> > > > aliases = <space-separated values>
> > > >   Specified one or more additional names that can be used to reference
> > > >   the repository (e.g. in repository dependencies).  If not specified,
> > > >   no aliases are defined.
> > > >
> > > > thin-manifests = ``true`` or ``false``
> > > >   If enabled, Manifest files in the package directory must contain only
> > > >   ``DIST`` entries.  If disabled, Manifest files in the package
> > > >   directory must list digests for all files found in the package
> > > >   directory and the files directory.  The default is ``false``.
> > > >
> > > > sign-commits = ``true`` or ``false``
> > > >   Indicates whether git commits are to be signed (using ``git commit
> > > >   --gpg-sign``.  The default is ``false``.
> > > >
> > > > sign-manifests = ``true`` or ``false``
> > > >   Indicates whether individual package Manifests should be PGP-signed.
> > > >   Note that this refers to the historical behavior of signing individual
> > > >   Manifests, not the GLEP 74 behavior of signing the top-level Manifest.
> > > >   [#GLEP74]_  The default is ``true`` if PGP signing is configured.
> > > >
> > > > properties-allowed = <space-separated values>
> > > >   Specifies the list of ``PROPERTIES`` tokens that are permitted
> > > >   to be used in ebuilds.  If present, the development tools should issue
> > > >   a warning if ``PROPERTIES`` contains any tokens that are not listed
> > > >   here.  If not specified, all tokens are permitted.
> > > >
> > > > restrict-allowed = <space-separated values>
> > > >   Same as properties-allowed, except for ``RESTRICT``.
> > > >
> > > > profile-formats = <space-separated values>
> > > >   Specifies the format used by profiles and/or extensions to it.
> > > >   The default is ``pms`` indicating the format specified in the PMS.
> > > >   Other values are implementation-defined.
> > > >
> > > >
> > > > It is recommended that any future keys are added to this GLEP before
> > > > being implemented.
> > > >
> > > >
> > > > Example
> > > > -------
> > > >
> > > > The following is an example configuration for a git repository with
> > > > Gentoo set as a master::
> > > >
> > > >     masters = gentoo
> > > >
> > > >     # git: do not use ChangeLog, use thin, unsigned Manifests
> > > >     update-changelog = false
> > > >     thin-manifests = true
> > > >     sign-manifests = false
> > > >
> > > >     # force the new md5-dict cache format
> > > >     cache-formats = md5-dict
> > > >
> > > >
> > > > Rationale
> > > > =========
> > > >
> > > > This GLEP is written almost 10 years after ``layout.conf`` was
> > > > originally introduced.  This made it necessary to write it in such a way
> > > > that both the modern and historical implementations in Portage
> > > > and PkgCore, as well as the use in the Gentoo repository
> > > > and a reasonably large subset of the other repositories would remain
> > > > compliant.
> > > >
> > > > The historical default of assuming ``masters = gentoo`` when unspecified
> > > > is omitted as it is not portable and verbosely deprecated for many
> > > > years in Portage.  All repositories are required to explicitly specify
> > > > their masters, or an empty value if they are stand-alone.
> > > >
> > > > The default for Manifest hashes and cache formats are left to be
> > > > implementation-defined, as the defaults changed over time and do not
> > > > match between package managers.  In particular, Portage attempts to
> > > > autodetect the cache format currently used in a given repository.
> > > >
> > > > The repo-name key has been originally added as an alternative to
> > > > ``profiles/repo_name``.  However, the latter file is still required
> > > > for PMS compliance.  Furthermore, given that it is much easier to parse,
> > > > there seems to be no appealing reason to work towards replacing that
> > > > file.  This means that for all practical reasons, the repo-name key
> > > > is redundant and is listed here for completeness only.
> > > >
> > > > The profile-formats key has been introduced to permit Portage-specific
> > > > extensions to the profile directory without having to introduce custom
> > > > EAPIs.  The exact extensions are considered outside the scope of this
> > > > specification.
> > > >
> > > >
> > > > Backwards Compatibility
> > > > =======================
> > > >
> > > > The existing implementations found in Portage and PkgCore conform
> > > > to this specification, so does the ``metadata/layout.conf`` file
> > > > found in the Gentoo repository.
> > > >
> > > >
> > > > Reference Implementation
> > > > ========================
> > > >
> > > > The support for ``metadata/layout.conf`` is already a part of Portage
> > > > and PkgCore.
> > > >
> > > >
> > > > References
> > > > ==========
> > > >
> > > > .. [#GLEP74] GLEP 74: Full-tree verification using Manifest files
> > > >    (https://www.gentoo.org/glep/glep-0074.html)
> > > >
> > > >
> > > > Copyright
> > > > =========
> > > >
> > > > This work is licensed under the Creative Commons Attribution-ShareAlike
> > > > 4.0
> > > > International License. To view a copy of this license, visit
> > > > https://creativecommons.org/licenses/by-sa/4.0/.
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Michał Górny
> > > >
> > > >
> > > >
> > >
> >
> > --
> > Best regards,
> > Michał Górny
> >
> >
> >
>

-- 
Best regards,
Michał Górny