Gentoo Archives: gentoo-dev

From: "Michał Górny" <mgorny@g.o>
To: gentoo-dev@l.g.o
Cc: "Michał Górny" <mgorny@g.o>
Subject: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation
Date: Thu, 08 Feb 2018 17:09:35
Message-Id: 20180208170922.5756-1-mgorny@gentoo.org
1 Remove the limitation that all files covered by the Manifest must reside
2 on a single filesystem. This breaks valid uses of overlayfs without
3 providing any real advantage.
4
5 The removal is justified further in the updated rationale section.
6 ---
7 glep-0074.rst | 66 +++++++++++++++++++++++++++++++++++------------------------
8 1 file changed, 39 insertions(+), 27 deletions(-)
9
10 RST: https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst
11 HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html
12
13 diff --git a/glep-0074.rst b/glep-0074.rst
14 index 3835247..2f8deb2 100644
15 --- a/glep-0074.rst
16 +++ b/glep-0074.rst
17 @@ -6,10 +6,10 @@ Author: Michał Górny <mgorny@g.o>,
18 Ulrich Müller <ulm@g.o>
19 Type: Standards Track
20 Status: Final
21 -Version: 1
22 +Version: 1.1
23 Created: 2017-10-21
24 -Last-Modified: 2017-12-16
25 -Post-History: 2017-10-26, 2017-11-16
26 +Last-Modified: 2018-02-08
27 +Post-History: 2017-10-26, 2017-11-16, 2018-02-08
28 Content-Type: text/x-rst
29 Requires: 59, 61
30 Replaces: 44, 58, 60
31 @@ -126,13 +126,6 @@ a different file type. If the tree contain files of other types
32 that are not otherwise ignored, they need to be covered by an explicit
33 ``IGNORE``.
34
35 -All the local (non-``DIST``) files covered by a Manifest tree must
36 -reside on the same filesystem. It is an error to specify entries
37 -applying to files on another filesystem. If files or directories that
38 -are not otherwise ignored reside on a different filesystem, or symbolic
39 -links point to targets on a different filesystem, they must
40 -be explicitly excluded via ``IGNORE``.
41 -
42
43 Path and filename encoding
44 --------------------------
45 @@ -325,22 +318,18 @@ Algorithm for finding parent Manifests
46 In order to find the top-level Manifest from the current directory
47 the following algorithm can be used:
48
49 -1. Store the current directory as *original* and the device ID
50 - of the containing filesystem (``st_dev``) as *startdev*,
51 -
52 -2. If the device ID of the containing filesystem (``st_dev``)
53 - of the current directory is different than *startdev*, stop.
54 +1. Store the current directory as *original*,
55
56 -3. If the current directory contains a ``Manifest`` file:
57 +2. If the current directory contains a ``Manifest`` file:
58
59 a. If an ``IGNORE`` entry in the ``Manifest`` file covers
60 the *original* directory (or one of the parent directories), stop.
61
62 b. Otherwise, store the current directory as *last_found*.
63
64 -4. If the current directory is the root system directory (``/``), stop.
65 +3. If the current directory is the root system directory (``/``), stop.
66
67 -5. Otherwise, enter the parent directory and jump to step 2.
68 +4. Otherwise, enter the parent directory and jump to step 2.
69
70 Once the algorithm stops, *last_found* will contain the relevant
71 top-level Manifest. If *last_found* is null, then the directory tree
72 @@ -594,16 +583,39 @@ additional files in it, skipping some of the categories from syncing.
73 This configuration can extend beyond the limits of this GLEP,
74 e.g. by allowing wildcards or regular expressions.
75
76 -The algorithm is restricted to work on a single filesystem. This is
77 -mostly relevant when scanning for top-level Manifest -- we do not want
78 -to cross filesystem boundaries then. However, to ensure consistent
79 -bidirectional behavior we need to also ban them when operating downwards
80 -the tree.
81
82 -The directories and files on different filesystems need to be ignored
83 -explicitly as implicitly skipping them would cause confusion.
84 -In particular, tools might then claim that a file does not exist when
85 -it clearly does because it was skipped due to filesystem boundaries.
86 +Cross-filesystem Manifests
87 +--------------------------
88 +
89 +The first version of this specification had an additional requirement
90 +that all files covered by the Manifest tree must reside on a single
91 +filesystem. This requirement has been removed in version 1.1 for
92 +the reasons outlined in this section.
93 +
94 +The original rationale stated that this restriction aims to prevent
95 +crossing filesystem boundaries in the top-level Manifest lookup
96 +algorithm. While that seemed a good idea at the time, there is no real
97 +reason to prevent that and this particular method worked correctly only
98 +if the files were placed in a dedicated filesystem.
99 +
100 +Worse than that, the original rationale did not anticipate the use
101 +of overlayfs which combines multiple filesystems while preserving their
102 +original metadata, including device and inode numbers. As a result,
103 +if the repository was checked out to an overlayfs, it was quite possible
104 +that different files had different device numbers, and the Manifest
105 +checks failed due to crossing filesystem boundaries.
106 +
107 +Given no clear solution to that and no good reason to reject use
108 +of overlayfs, the restriction was lifted.
109 +
110 +The only potential drawback of this is that the implementation may now
111 +follow maliciously placed symbolic links pointing outside the tree.
112 +If a regular file was replaced by such a symlink, the user could
113 +be tricked into reporting the verification failure with the report
114 +containing the checksums of the target file. However, for this to happen
115 +the client would have to use rsync with ``--links`` option but without
116 +``--safe-links`` which is neither the default behavior of rsync nor
117 +the default configuration used by Portage.
118
119
120 Filename character set restriction
121 --
122 2.16.1

Replies