Re: [gentoo-dev] [RFC] GLEP 74: Full-tree verification using Manifest files

[Michał Górny]

W dniu pią, 27.10.2017 o godzinie 21∶05 +0000, użytkownik Robin H.
Johnson napisał:
> On Thu, Oct 26, 2017 at 10:12:25PM +0200, Michał Górny wrote:
> > 2. Alike the original Manifest2, the files should be split into two
> >    groups — files whose authenticity is critical, and those whose
> >    mismatch may be accepted in non-strict mode. The same classification
> >    should apply both to files listed in Manifests, and to stray files
> >    present only in the repository.
>
> nit: s/Alike/Like/, or rewrite the sentence.

Done.

> > Manifest file locations and nesting
> > -----------------------------------
> > The ``Manifest`` file located in the root directory of the repository
> > is called top-level Manifest, and it is used to perform the full-tree
> > verification. In order to verify the authenticity, it must be signed
> > using OpenPGP, using the armored cleartext format.
>
> Are detached signatures also permitted (for all levels of Manifest)?

I'd say no. Keeping it always contained in a single file is simpler.

> > The Manifest files can also specify ``IGNORE`` entries to skip Manifest
> > verification of subdirectories and/or files. Files and directories
> > starting with a dot are always implicitly ignored. All files that
> > are not ignored must be covered by at least one of the Manifests.
>
> Do we need to keep that implicit ignore rule? Rather, convert it to being
> always explicit.
>
> There is only one such file in the rsync checkout presently:
> metadata/.checksum-test-marker (see bug #572168, it is used to detect
> mis-configured mirrors).
>
> A SVN or Git repo might also have dot-named directories.

I like the implicit idea better as it is more consistent with normal
tool behavior, like 'ls' not listing the files. Dotfiles can be created
by many random tools or even the filesystem (especially in some cases
of overlay filesystems).

That said, the case of 'lost+found' just occurred to me. I suppose this
one we will want to always IGNORE.

> > All the files covered by a Manifest tree must reside on the same
> > filesystem. It is an error to specify entries applying to files
> > on another filesystem. If subdirectories of the Manifest tree reside
> > on a different filesystem, they must be explicitly excluded
> > via ``IGNORE``.
>
> Distfiles aren't required to be in the same filesystem.

I've updated the sentence to clearly indicate it's about «local (non-
``DIST``) files».

>
> > New Manifest tags
> > -----------------
>
> ...
> > ``IGNORE <path>``
> >   Ignores a subdirectory or file from Manifest checks. If the specified
> >   path is present, it and its contents are omitted from the Manifest
> >   verification (always pass).
>
> Should this be accepted even by strict-mode? Alternatively, should strict mode
> require that other content is kept outside of the tree?

Yes, it should. I'd really prefer if strict mode still worked out-of-
the-box for most of our users without requiring them to do major
reshuffling of their systems.

Plus, see 'lost+found' above.

> > Algorithm for full-tree verification
> > ------------------------------------
>
> ...
> > 2. Start at the top-level Manifest file. Verify its OpenPGP signature.
> >    Optionally verify the ``TIMESTAMP`` entry if present. Remove
> >    the top-level Manifest from the *present* set.
>
> This spec does not state how the timestamp should be verified.
> Borrow from the original GLEP?

Let's try:

| 2. Start at the top-level Manifest file. Verify its OpenPGP signature.
|    Optionally verify the ``TIMESTAMP`` entry if present.
|    If the timestamp is significantly out of date compared to the local
|    clock or a trusted source, halt or require manual intervention
|    from the user. Remove the top-level Manifest from the *present* set.

Maybe it would look better if I split it into sub-points.

>
> > 4. Process all ``IGNORE`` entries. Remove any paths matching them
> >    from the *present* set.
> >
> > 5. Collect all files covered by ``DATA``, ``MISC``, ``OPTIONAL``,
> >    ``EBUILD`` and ``AUX`` entries into the *covered* set.
>
> Clarification request: point out again in this section, that IGNORE entries are
> prohibited from also matching any other entry. It is mentioned further up, but
> a reminder is good.

I've added an extra step:

| 6. Verify the entries in *covered* set for incompatible duplicates
|    and collisions with ignored files as explained in `Manifest file
|    locations and nesting`_.

>
> > Checksum algorithms
> > -------------------
> > This section is informational only. Specifying the exact set
> > of supported algorithms is outside the scope of this specification.
>
> ...
> > The method of introducing new hashes is defined by GLEP 59 [#GLEP59]_.
> > It is recommended that any new hashes are named after the Python
> > ``hashlib`` module algorithm names, transformed into uppercase.
>
> Would we ever consider algorithm parameters? Yes, outside of this spec, but checking anyway.

I can't say for sure but so far I've went for 'no'. That's why gemato
does not support e.g. SHAKE* algorithms. If we ever decide to do that,
I suppose we can do it inside hash name, e.g. FOO-<param1>-<param2>...

>
> > Manifest compression
> > --------------------
>
> ...
> > The specification permits uncompressed Manifests to exist alongside
> > their compressed counterparts, and multiple compressed formats
> > to coexist. If that is the case, the files must have the same
> > uncompressed content and the specification is free to choose either
> > of the files using the same base name.
>
> GLEP61, for the transition period, required compressed & uncompressed Manifests
> in the same directory to have identical content. Include mention of that here.

Can do. But I'll do it in 'Backwards compatibility' section:

| - if the Manifest files inside the package directory are compressed,
|   a uncompressed file of identical content must coexist.

> Saying that either can be used is a potential issue.

Why? It also says that they must be identical, so it's of no difference
to the implementation which one is used.

> > Tree design
> > -----------
>
> ...
>
> Add a minor header here, to say this is the end of the 'Tree design' section?

It's not the end, it's description of the alternative. Both belong
in one section. I could add additional section level but I'd rather
not do that for a single use.

> > In the independent model, each sub-Manifest file is independent
> > of the parent Manifests. As a result, each of them needs to be signed
> > and verified independently. However, the parent Manifests still need
> > to list sub-Manifests (albeit without verification data) in order
> > to detect removal or replacement of subdirectories. This has
> > the following implications:
>
> ...
>
> > File verification model
> > -----------------------
> >
> > The verification model aims to provide full coverage against different
> > forms of attack. In particular, three different kinds of manipulation
> > are considered:
> > ...
>
> Selective denial of syncing was also one of the attacks in the original GLEPs
> that was considered. See details re timestamp below.

But that's not covered by 'file verification model', is it? So I suppose
it's better to detail it below.

>
> > Timestamp field
> > ---------------
> >
> > The top-level Manifests optionally allows using a ``TIMESTAMP`` tag
> > to include a generation timestamp in the Manifest. A similar feature
> > was originally proposed in GLEP 58 [#GLEP58]_.
> >
> > The timestamp can be used to detect delay or replay attacks against
> > Gentoo mirrors.
> >
> > Strictly speaking, this is already provided by the various
> > ``metadata/timestamp.*`` files provided already by Gentoo which are also
> > covered by the Manifest. However, including the value in the Manifest
> > itself has a little cost and provides the ability to perform
> > the verification stand-alone.
>
> There's a critical part of the GLEP58 spec that got missed here:
> https://www.gentoo.org/glep/glep-0058.html#timestamps-additional-distribution-of-metamanifest
> The timestamp needs to be usable to verify if the mirror is update to date vs
> known masters.
>
> The attack being defended against is that local community mirror (or MITM)
> isn't deliberately handing them an unmodified but stale copy of the tree.
>
> I do approve of changing the format of the tag; but it still needs to be
> linkable to a more verifiable source of truth,

I've tried to expand it a bit without getting too specific. New content
for paragraphs 2+:

| A malicious third-party may use the principles of exclusion and replay
| to deny an update to clients, while at the same time recording
| the identity of clients to attack. The timestamp field can be used
| to detect that.
|
| In order to provide a more complete protection, the Gentoo
| Infrastructure should provide an ability to obtain the timestamps
| of all Manifests from a recent timeframe over a secure channel
| for comparison.
|
| Strictly speaking, this is already provided by the various
| ``metadata/timestamp.*`` files provided already by Gentoo which are also
| covered by the Manifest. However, including the value in the Manifest
| itself has a little cost and provides the ability to perform
| the verification stand-alone.

> > Backwards Compatibility
> > =======================
> >
> > This GLEP provides optional means of preserving backwards compatibility.
> > To preserve the backwards compatibility, the following needs to be
> > ensured:
> >
> > - all files within the package directory must be covered by ``Manifest``
> >   file inside that package directory,
>
> This implies that IGNORE entries are NOT permitted to cover any file in
> a package directory during the transition period.

Well, obviously you can't use new tags in those files and rely on they
working correctly.

> >
> > - all distfiles used by the package must be covered by ``Manifest``
> >   file inside the package directory,
>
> This implies that non-package-dir DIST entries may be a duplicate of a
> package-level DIST during the transition.

Yes, that's permitted if they're compatible.

> > - all files inside the ``files/`` subdirectory of a package directory
> >   need to be use the deprecated ``AUX`` tag (rather than ``DATA``),
> >
> > - all ``.ebuild`` files inside the package directory need to use
> >   the deprecated ``EBUILD`` tag (rather than ``DATA``),
>
> Could we please note here, for the transitional period, that the
> file equivalence rule is applicable?
> During the transitional, the package Manifests may contain two entries for a
> given file: (DATA, EBUILD) or (DATA, AUX). The MISC type remains the
> same.

Equivalence rule is applicable always. However, there's no point
in duplicating the entry for the same file as that's only going
to increase space use.

> > - the Manifest files inside the package directory can be signed
> >   to provide authenticity verification.
>
> Why do we need to specify this in backwards compat, it's still permitted above.

But it makes no sense when top-level Manifest is signed. This points out
that for tools not supporting full-tree verification smaller signatures
need to be used (skipping the fact that Portage did not ever implement
it).

Updated the two linked files.

-- 
Best regards,
Michał Górny