Re: [gentoo-dev] [pre-GLEP] Repository configuration file (layout.conf)

[Wolfgang E. Sanyer]

On Wed, May 19, 2021 at 11:02 AM Michał Górny <mgorny@g.o> wrote:
>
> On Wed, 2021-05-19 at 10:37 -0400, Wolfgang E. Sanyer wrote:
> > See some comments in-line below.
> >
> >
> > On Wed, May 19, 2021 at 8:33 AM Michał Górny <mgorny@g.o> wrote:
> > >
> > > Hi,
> > >
> > > Please review the pre-GLEP inlined below.  Its purpose is to formally
> > > define the format of layout.conf.  It's pretty much inevitable these
> > > days, so we should specify it.  However, it doesn't really fit into PMS,
> > > and other formats (Manifests, metadata.xml) are already defined
> > > in GLEPs, so this just follows suit.
> > >
> > > Pre-GLEP follows.
> > >
> > > ---
> > > GLEP: 9999
> > > Title: Repository configuration file (layout.conf)
> > > Author: Michał Górny <mgorny@g.o>
> > > Type: Standards Track
> > > Status: Draft
> > > Version: 1.0
> > > Created: 2021-05-19
> > > Last-Modified: 2021-05-19
> > > Post-History: 2021-05-19
> > > Content-Type: text/x-rst
> > > ---
> > >
> > > Abstract
> > > ========
> > >
> > > The ``metadata/layout.conf`` file format is specified as used by Portage
> > > and PkgCore.  A standard set of configuration keys is described
> > > including the keys currently used in the Gentoo repository.
> > >
> > >
> > > Motivation
> > > ==========
> > >
> > > The ``metadata/layout.conf`` file was first added to the Gentoo
> > > repository in Oct 2011, to facilitate setting of hashes used
> > > in Manifest2 files.  In Mar 2012, it was used to indicate the transition
> > > to the new ``md5-dict`` cache format.  In Jul 2013, it started being
> > > used to indicate the repository's masters and effectively became
> > > obligatory for all repositories.
> > >
> > > Today, ``layout.conf`` is used for various repository configuration
> > > knobs that can be expressed as simple values and therefore
> > > do not justify adding new files to the repository.  This primarily
> > > involves the configuration of development tools but also includes a few
> > > keys relevant to the behavior of the package manager.
> > >
> > > However, ``layout.conf`` is currently not covered by any formal
> > > specification.  The PMS neglects its existence entirely, and the keys
> > > used are roughly defined by their first use of Portage or PkgCore.
> > > This GLEP aims to overcome this by providing a formal specification
> > > for the file, as well as an up-to-date list of permitted configuration
> > > keys.
> > >
> > >
> > > Specification
> > > =============
> > >
> > > layout.conf file format
> > > -----------------------
> > >
> > > Every ebuild repository must contain a ``metadata/layout.conf`` file.
> > > The file uses a line-oriented text format.  Lines starting with ``#``
> > > represent comments and are ignored, as are lines consisting entirely
> > > of whitespace.  The remaining lines must contain a key followed
> > > by equals sign (``=``), followed by a (possibly empty) value.  Each of
> > > these elements can be surrounded by additional whitespace that
> > > is stripped.
> >
> > Is a space around the "=" optional? I see it used below, but here it sounds
> > like there should be no space.
>
> The last sentence says you can add extra space and it will be stripped.
> If you can think of a better way of wording that (I really wanted to
> avoid 'optional whitespace, followed by key, followed by optional
> whitespace... ' ;-).

🤦‍♂️ ofc I would miss that.

I always thought that python did a pretty good job
[explaining their format mini-language][1]. That's obviously overkill for what
you're describing here, but maybe something like this would be helpful?

# comments are allowed
<key> = [<value1> [<value2>...]]

Any lines starting with a ``#`` represents a comment and are ignored, as are
lines consisting entirely of whitespace.

A valid line consists of key and an equal sign. (A valid key is a
string containing
any character except for a space)[^1]. The list of space-separated
values is optional.

Note: any additional spaces are ultimately stripped

[1]: https://docs.python.org/3/library/string.html#format-specification-mini-language
[^1]: Should we include this or nah?

> >
> > > Configuration keys
> > > ------------------
> > >
> > > The ``layout.conf`` file must contain the ``masters`` key.  Other keys
> > > listed in this specification are entirely optional.  The package
> > > managers may choose to implement a subset of listed keys.  Unknown keys
> > > must be ignored.
> > >
> > > The following keys are currently defined:
> >
> > Is the intent for the GLEP to define and specify every possible key-value
> > combination? If not, then perhaps this section should be truncated a bit? i.e.
> > "here are some examples of currently defined keys" and then list a handful.
> >
> > Additionally, maybe the spec should include some sort of requirement for
> > documentation of variables, similar to what you've done below.
> >
> > Edit: ok, nevermind I see your recommendation at the bottom. But still,
> > I would probably either (1) insist that new keys be added to this GLEP, or
> > (2) insist that new keys are documented properly.
>
> Let's put it like this: the goal is to document all the useful keys.
> I can live with some ancient-not-really-useful keys not being documented
> (e.g. Portage has some stale/redundant Manifest-related keys, I think).

Perhaps then some sort of use of the word "reserved" should be in here?
i.e. "The following keys are reserved for use by portage and pkgcheck.
any other keys may be defined and used at will"

> >
> > > masters = <space-separated values>
> > >   Specifies the master repositories of this repository.  For stand-alone
> > >   repositories, this must be set to an empty value.  Otherwise, it can
> > >   list one or more repositories, separated by spaces.  This key must
> > >   be specified.
> >
> > I realize this is the only compulsory key, but perhaps it should still go in
> > a separate section in order to ensure that it stands out? i.e. "The following
> > keys are mandory..." and then later "the following optional keys are
> > currently defined"
>
> I'm not sure, to be honest.  I didn't really want to add a section for
> one key ;-).

np, just a thought.

> >
> > > manifest-hashes = <space-separated values>
> > >   Specifies the list of hashes that should be used for new distfiles
> > >   in the Manifest files.  The development tools may create a subset
> > >   of the specified hashes if it is not updating the checksums for
> > >   the specified distfile, or does not support the hash in question.
> > >   The hash names are specified in GLEP 74.  [#GLEP74]_  The default
> > >   set of hashes is implementation-defined.
> > >
> > > manifest-required-hashes = <space-separated values>
> > >   Specifies the list of hashes that must be used in Manifest files.
> > >   The development tools must support all the hashes listed there,
> > >   and update distfile checksums to use these hashes (refetching
> > >   if necessary).  This must be a subset of manifest-hashes.  If not
> > >   specified, all hashes from manifest-hashes (or the default set)
> > >   are considered required.
> > >
> > > use-manifests = ``strict``, ``true`` or ``false``
> > >   Indicates the policy for creating and using Manifest files.  If set
> > >   to ``strict``, Manifest files are created and files are required to
> > >   match digests found in Manifests.  If set to ``true``, Manifests
> > >   are created but digest mismatches are ignored.  If set to ``false``,
> > >   Manifests are not used at all.  The default is ``strict``.
> > >
> > > update-changelog = ``true`` or ``false``
> > >   Indicates whether the development tools should write ChangeLog files.
> > >   The default is ``false``.
> > >
> > > cache-formats = <space-separated values>
> > >   Specifies one or more cache formats used by the repository.
> > >   The currently defined values are ``pms`` for the original format
> > >   specified in the PMS and ``md5-dict`` for the md5-dict format
> > >   introduced in Portage 2.2.0_alpha68.  The default is
> > >   implementation-defined.
> > >
> > > eapis-deprecated = <space-separated values>
> > >   Specifies one or more EAPIs that are to be considered deprecated
> > >   by the development tools for use in ebuilds, i.e. their use should
> > >   trigger a warning.  If not specified, no EAPIs are deprecated.
> > >
> > > eapis-banned = <space-separated values>
> > >   Specifies one or more EAPIs that are to be considered banned
> > >   by the development tools for use in ebuilds, i.e. their use should
> > >   be blocked.  If not specified, no EAPIs are banned.
> > >
> > > repo-name = <string>
> > >   Specifies the repository name.  If specified, it must be equal
> > >   to the contents of ``profiles/repo_name``.  If not specified,
> > >   it defaults to the same value.  Discouraged.
> > >
> > > aliases = <space-separated values>
> > >   Specified one or more additional names that can be used to reference
> > >   the repository (e.g. in repository dependencies).  If not specified,
> > >   no aliases are defined.
> > >
> > > thin-manifests = ``true`` or ``false``
> > >   If enabled, Manifest files in the package directory must contain only
> > >   ``DIST`` entries.  If disabled, Manifest files in the package
> > >   directory must list digests for all files found in the package
> > >   directory and the files directory.  The default is ``false``.
> > >
> > > sign-commits = ``true`` or ``false``
> > >   Indicates whether git commits are to be signed (using ``git commit
> > >   --gpg-sign``.  The default is ``false``.
> > >
> > > sign-manifests = ``true`` or ``false``
> > >   Indicates whether individual package Manifests should be PGP-signed.
> > >   Note that this refers to the historical behavior of signing individual
> > >   Manifests, not the GLEP 74 behavior of signing the top-level Manifest.
> > >   [#GLEP74]_  The default is ``true`` if PGP signing is configured.
> > >
> > > properties-allowed = <space-separated values>
> > >   Specifies the list of ``PROPERTIES`` tokens that are permitted
> > >   to be used in ebuilds.  If present, the development tools should issue
> > >   a warning if ``PROPERTIES`` contains any tokens that are not listed
> > >   here.  If not specified, all tokens are permitted.
> > >
> > > restrict-allowed = <space-separated values>
> > >   Same as properties-allowed, except for ``RESTRICT``.
> > >
> > > profile-formats = <space-separated values>
> > >   Specifies the format used by profiles and/or extensions to it.
> > >   The default is ``pms`` indicating the format specified in the PMS.
> > >   Other values are implementation-defined.
> > >
> > >
> > > It is recommended that any future keys are added to this GLEP before
> > > being implemented.
> > >
> > >
> > > Example
> > > -------
> > >
> > > The following is an example configuration for a git repository with
> > > Gentoo set as a master::
> > >
> > >     masters = gentoo
> > >
> > >     # git: do not use ChangeLog, use thin, unsigned Manifests
> > >     update-changelog = false
> > >     thin-manifests = true
> > >     sign-manifests = false
> > >
> > >     # force the new md5-dict cache format
> > >     cache-formats = md5-dict
> > >
> > >
> > > Rationale
> > > =========
> > >
> > > This GLEP is written almost 10 years after ``layout.conf`` was
> > > originally introduced.  This made it necessary to write it in such a way
> > > that both the modern and historical implementations in Portage
> > > and PkgCore, as well as the use in the Gentoo repository
> > > and a reasonably large subset of the other repositories would remain
> > > compliant.
> > >
> > > The historical default of assuming ``masters = gentoo`` when unspecified
> > > is omitted as it is not portable and verbosely deprecated for many
> > > years in Portage.  All repositories are required to explicitly specify
> > > their masters, or an empty value if they are stand-alone.
> > >
> > > The default for Manifest hashes and cache formats are left to be
> > > implementation-defined, as the defaults changed over time and do not
> > > match between package managers.  In particular, Portage attempts to
> > > autodetect the cache format currently used in a given repository.
> > >
> > > The repo-name key has been originally added as an alternative to
> > > ``profiles/repo_name``.  However, the latter file is still required
> > > for PMS compliance.  Furthermore, given that it is much easier to parse,
> > > there seems to be no appealing reason to work towards replacing that
> > > file.  This means that for all practical reasons, the repo-name key
> > > is redundant and is listed here for completeness only.
> > >
> > > The profile-formats key has been introduced to permit Portage-specific
> > > extensions to the profile directory without having to introduce custom
> > > EAPIs.  The exact extensions are considered outside the scope of this
> > > specification.
> > >
> > >
> > > Backwards Compatibility
> > > =======================
> > >
> > > The existing implementations found in Portage and PkgCore conform
> > > to this specification, so does the ``metadata/layout.conf`` file
> > > found in the Gentoo repository.
> > >
> > >
> > > Reference Implementation
> > > ========================
> > >
> > > The support for ``metadata/layout.conf`` is already a part of Portage
> > > and PkgCore.
> > >
> > >
> > > References
> > > ==========
> > >
> > > .. [#GLEP74] GLEP 74: Full-tree verification using Manifest files
> > >    (https://www.gentoo.org/glep/glep-0074.html)
> > >
> > >
> > > Copyright
> > > =========
> > >
> > > This work is licensed under the Creative Commons Attribution-ShareAlike
> > > 4.0
> > > International License. To view a copy of this license, visit
> > > https://creativecommons.org/licenses/by-sa/4.0/.
> > >
> > >
> > > --
> > > Best regards,
> > > Michał Górny
> > >
> > >
> > >
> >
>
> --
> Best regards,
> Michał Górny
>
>
>