Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Robin H. Johnson]

On Thu, Oct 26, 2017 at 10:12:25PM +0200, Michał Górny wrote:
> Hi, everyone.
>
> After a week of hard work, I'd like to request your comments
> on the draft of GLEP 74. This GLEP aims to replace the old tree-signing
> GLEPs 58 and 60 with a superior implementation and more complete
> specification.
Edits inline, with trimming content.

Very strong proposal, I approve of this replacing my earlier work, as it learns
from the prototypes, failed implementation, and the intervening years of Gentoo
experience.

> 2. Alike the original Manifest2, the files should be split into two
>    groups — files whose authenticity is critical, and those whose
>    mismatch may be accepted in non-strict mode. The same classification
>    should apply both to files listed in Manifests, and to stray files
>    present only in the repository.
nit: s/Alike/Like/, or rewrite the sentence.

> Manifest file locations and nesting
> -----------------------------------
> The ``Manifest`` file located in the root directory of the repository
> is called top-level Manifest, and it is used to perform the full-tree
> verification. In order to verify the authenticity, it must be signed
> using OpenPGP, using the armored cleartext format.
Are detached signatures also permitted (for all levels of Manifest)?
>
> The Manifest files can also specify ``IGNORE`` entries to skip Manifest
> verification of subdirectories and/or files. Files and directories
> starting with a dot are always implicitly ignored. All files that
> are not ignored must be covered by at least one of the Manifests.
Do we need to keep that implicit ignore rule? Rather, convert it to being
always explicit.

There is only one such file in the rsync checkout presently:
metadata/.checksum-test-marker (see bug #572168, it is used to detect
mis-configured mirrors).

A SVN or Git repo might also have dot-named directories.

> All the files covered by a Manifest tree must reside on the same
> filesystem. It is an error to specify entries applying to files
> on another filesystem. If subdirectories of the Manifest tree reside
> on a different filesystem, they must be explicitly excluded
> via ``IGNORE``.
Distfiles aren't required to be in the same filesystem.

> New Manifest tags
> -----------------
...
> ``IGNORE <path>``
>   Ignores a subdirectory or file from Manifest checks. If the specified
>   path is present, it and its contents are omitted from the Manifest
>   verification (always pass).
Should this be accepted even by strict-mode? Alternatively, should strict mode
require that other content is kept outside of the tree?

> Algorithm for full-tree verification
> ------------------------------------
...
> 2. Start at the top-level Manifest file. Verify its OpenPGP signature.
>    Optionally verify the ``TIMESTAMP`` entry if present. Remove
>    the top-level Manifest from the *present* set.
This spec does not state how the timestamp should be verified. 
Borrow from the original GLEP?

> 4. Process all ``IGNORE`` entries. Remove any paths matching them
>    from the *present* set.
>
> 5. Collect all files covered by ``DATA``, ``MISC``, ``OPTIONAL``,
>    ``EBUILD`` and ``AUX`` entries into the *covered* set.
Clarification request: point out again in this section, that IGNORE entries are
prohibited from also matching any other entry. It is mentioned further up, but
a reminder is good.

> Checksum algorithms
> -------------------
> This section is informational only. Specifying the exact set
> of supported algorithms is outside the scope of this specification.
...
> The method of introducing new hashes is defined by GLEP 59 [#GLEP59]_.
> It is recommended that any new hashes are named after the Python
> ``hashlib`` module algorithm names, transformed into uppercase.
Would we ever consider algorithm parameters? Yes, outside of this spec, but checking anyway.

> Manifest compression
> --------------------
...
> The specification permits uncompressed Manifests to exist alongside
> their compressed counterparts, and multiple compressed formats
> to coexist. If that is the case, the files must have the same
> uncompressed content and the specification is free to choose either
> of the files using the same base name.
GLEP61, for the transition period, required compressed & uncompressed Manifests
in the same directory to have identical content. Include mention of that here.

Saying that either can be used is a potential issue.

> Tree design
> -----------
...

Add a minor header here, to say this is the end of the 'Tree design' section?
> In the independent model, each sub-Manifest file is independent
> of the parent Manifests. As a result, each of them needs to be signed
> and verified independently. However, the parent Manifests still need
> to list sub-Manifests (albeit without verification data) in order
> to detect removal or replacement of subdirectories. This has
> the following implications:
...

> File verification model
> -----------------------
>
> The verification model aims to provide full coverage against different
> forms of attack. In particular, three different kinds of manipulation
> are considered:
> ...
Selective denial of syncing was also one of the attacks in the original GLEPs
that was considered. See details re timestamp below.

> Timestamp field
> ---------------
>
> The top-level Manifests optionally allows using a ``TIMESTAMP`` tag
> to include a generation timestamp in the Manifest. A similar feature
> was originally proposed in GLEP 58 [#GLEP58]_.
>
> The timestamp can be used to detect delay or replay attacks against
> Gentoo mirrors.
>
> Strictly speaking, this is already provided by the various
> ``metadata/timestamp.*`` files provided already by Gentoo which are also
> covered by the Manifest. However, including the value in the Manifest
> itself has a little cost and provides the ability to perform
> the verification stand-alone.
There's a critical part of the GLEP58 spec that got missed here:
https://www.gentoo.org/glep/glep-0058.html#timestamps-additional-distribution-of-metamanifest
The timestamp needs to be usable to verify if the mirror is update to date vs
known masters.

The attack being defended against is that local community mirror (or MITM)
isn't deliberately handing them an unmodified but stale copy of the tree.

I do approve of changing the format of the tag; but it still needs to be
linkable to a more verifiable source of truth, 

> Backwards Compatibility
> =======================
>
> This GLEP provides optional means of preserving backwards compatibility.
> To preserve the backwards compatibility, the following needs to be
> ensured:
>
> - all files within the package directory must be covered by ``Manifest``
>   file inside that package directory,
This implies that IGNORE entries are NOT permitted to cover any file in
a package directory during the transition period.
>
> - all distfiles used by the package must be covered by ``Manifest``
>   file inside the package directory,
This implies that non-package-dir DIST entries may be a duplicate of a
package-level DIST during the transition.

> - all files inside the ``files/`` subdirectory of a package directory
>   need to be use the deprecated ``AUX`` tag (rather than ``DATA``),
>
> - all ``.ebuild`` files inside the package directory need to use
>   the deprecated ``EBUILD`` tag (rather than ``DATA``),
Could we please note here, for the transitional period, that the
file equivalence rule is applicable? 
During the transitional, the package Manifests may contain two entries for a
given file: (DATA, EBUILD) or (DATA, AUX). The MISC type remains the
same.

> - the Manifest files inside the package directory can be signed
>   to provide authenticity verification.
Why do we need to specify this in backwards compat, it's still permitted above.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@g.o
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136