[gentoo-commits] gentoo commit in xml/htdocs/proj/en/glep: glep-0058.html

[Robin H. Johnson (robbat2)]

robbat2     10/01/31 07:53:41

  Modified:             glep-0058.html
  Log:
  Revise GLEP58 per Calchan questions: Additional levels of Manifests are no longer optional; Clarifications added to creation process;

Revision  Changes    Path
1.4                  xml/htdocs/proj/en/glep/glep-0058.html

file : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/proj/en/glep/glep-0058.html?rev=1.4&view=markup
plain: http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/proj/en/glep/glep-0058.html?rev=1.4&content-type=text/plain
diff : http://sources.gentoo.org/viewcvs.py/gentoo/xml/htdocs/proj/en/glep/glep-0058.html?r1=1.3&r2=1.4

Index: glep-0058.html
===================================================================
RCS file: /var/cvsroot/gentoo/xml/htdocs/proj/en/glep/glep-0058.html,v
retrieving revision 1.3
retrieving revision 1.4
diff -p -w -b -B -u -u -r1.3 -r1.4
--- glep-0058.html	13 Jan 2010 03:28:33 -0000	1.3
+++ glep-0058.html	31 Jan 2010 07:53:41 -0000	1.4
@@ -27,9 +27,9 @@
 </tr>
 <tr class="field"><th class="field-name">Title:</th><td class="field-body">Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest</td>
 </tr>
-<tr class="field"><th class="field-name">Version:</th><td class="field-body">1.6</td>
+<tr class="field"><th class="field-name">Version:</th><td class="field-body">1.7</td>
 </tr>
-<tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0058.txt?cvsroot=gentoo">2010/01/13 03:26:53</a></td>
+<tr class="field"><th class="field-name">Last-Modified:</th><td class="field-body"><a class="reference external" href="http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/proj/en/glep/glep-0058.txt?cvsroot=gentoo">2010/01/31 07:53:30</a></td>
 </tr>
 <tr class="field"><th class="field-name">Author:</th><td class="field-body">Robin Hugh Johnson &lt;robbat2&#32;&#97;t&#32;gentoo.org&gt;,</td>
 </tr>
@@ -45,7 +45,7 @@
 </tr>
 <tr class="field"><th class="field-name">Updated:</th><td class="field-body">November 2007, June 2008, July 2008, October 2008, January 2010</td>
 </tr>
-<tr class="field"><th class="field-name">Post-History:</th><td class="field-body">December 2009</td>
+<tr class="field"><th class="field-name">Post-History:</th><td class="field-body">December 2009, January 2010</td>
 </tr>
 </tbody>
 </table>
@@ -53,31 +53,36 @@
 <div class="contents topic" id="contents">
 <p class="topic-title first">Contents</p>
 <ul class="simple">
-<li><a class="reference internal" href="#abstract" id="id1">Abstract</a></li>
-<li><a class="reference internal" href="#motivation" id="id2">Motivation</a></li>
-<li><a class="reference internal" href="#specification" id="id3">Specification</a><ul>
-<li><a class="reference internal" href="#procedure-for-creating-the-metamanifest-file" id="id4">Procedure for creating the MetaManifest file:</a></li>
-<li><a class="reference internal" href="#verification-of-one-or-more-items-from-the-metamanifest" id="id5">Verification of one or more items from the MetaManifest:</a></li>
-<li><a class="reference internal" href="#procedure-for-verifying-an-item-in-the-metamanifest" id="id6">Procedure for verifying an item in the MetaManifest:</a><ul>
-<li><a class="reference internal" href="#notes" id="id7">Notes:</a></li>
+<li><a class="reference internal" href="#abstract" id="id2">Abstract</a></li>
+<li><a class="reference internal" href="#motivation" id="id3">Motivation</a></li>
+<li><a class="reference internal" href="#specification" id="id4">Specification</a><ul>
+<li><a class="reference internal" href="#procedure-for-creating-the-metamanifest-file" id="id5">Procedure for creating the MetaManifest file:</a><ul>
+<li><a class="reference internal" href="#summary" id="id6">Summary:</a></li>
+<li><a class="reference internal" href="#process" id="id7">Process:</a></li>
+<li><a class="reference internal" href="#notes" id="id8">Notes:</a></li>
 </ul>
 </li>
+<li><a class="reference internal" href="#verification-of-one-or-more-items-from-the-metamanifest" id="id9">Verification of one or more items from the MetaManifest:</a></li>
+<li><a class="reference internal" href="#procedure-for-verifying-an-item-in-the-metamanifest" id="id10">Procedure for verifying an item in the MetaManifest:</a><ul>
+<li><a class="reference internal" href="#id1" id="id11">Notes:</a></li>
 </ul>
 </li>
-<li><a class="reference internal" href="#implementation-notes" id="id8">Implementation Notes</a><ul>
-<li><a class="reference internal" href="#metamanifest-and-the-new-manifest2-filetypes" id="id9">MetaManifest and the new Manifest2 filetypes</a></li>
-<li><a class="reference internal" href="#timestamps-additional-distribution-of-metamanifest" id="id10">Timestamps &amp; Additional distribution of MetaManifest</a></li>
-<li><a class="reference internal" href="#metamanifest-size-considerations" id="id11">MetaManifest size considerations</a></li>
 </ul>
 </li>
-<li><a class="reference internal" href="#backwards-compatibility" id="id12">Backwards Compatibility</a></li>
-<li><a class="reference internal" href="#thanks" id="id13">Thanks</a></li>
-<li><a class="reference internal" href="#references" id="id14">References</a></li>
-<li><a class="reference internal" href="#copyright" id="id15">Copyright</a></li>
+<li><a class="reference internal" href="#implementation-notes" id="id12">Implementation Notes</a><ul>
+<li><a class="reference internal" href="#metamanifest-and-the-new-manifest2-filetypes" id="id13">MetaManifest and the new Manifest2 filetypes</a></li>
+<li><a class="reference internal" href="#timestamps-additional-distribution-of-metamanifest" id="id14">Timestamps &amp; Additional distribution of MetaManifest</a></li>
+<li><a class="reference internal" href="#metamanifest-size-considerations" id="id15">MetaManifest size considerations</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#backwards-compatibility" id="id16">Backwards Compatibility</a></li>
+<li><a class="reference internal" href="#thanks" id="id17">Thanks</a></li>
+<li><a class="reference internal" href="#references" id="id18">References</a></li>
+<li><a class="reference internal" href="#copyright" id="id19">Copyright</a></li>
 </ul>
 </div>
 <div class="section" id="abstract">
-<h1><a class="toc-backref" href="#id1">Abstract</a></h1>
+<h1><a class="toc-backref" href="#id2">Abstract</a></h1>
 <p>MetaManifest provides a means of verifiable distribution from Gentoo
 Infrastructure to a user system, while data is conveyed over completely
 untrusted networks and system, by extending the Manifest2 specification,
@@ -85,7 +90,7 @@ and adding a top-level Manifest file, wi
 Manifests.</p>
 </div>
 <div class="section" id="motivation">
-<h1><a class="toc-backref" href="#id2">Motivation</a></h1>
+<h1><a class="toc-backref" href="#id3">Motivation</a></h1>
 <p>As part of a comprehensive security plan, we need a way to prove that
 something originating from Gentoo as an organization (read Gentoo-owned
 hardware, run by infrastructure), has not been tampered with. This
@@ -114,7 +119,7 @@ mirrors, and allows detection of all cas
 by deliberate delay, replay [C08a, C08b] or alteration).</p>
 </div>
 <div class="section" id="specification">
-<h1><a class="toc-backref" href="#id3">Specification</a></h1>
+<h1><a class="toc-backref" href="#id4">Specification</a></h1>
 <p>For lack of a better name, the following solution should be known as the
 MetaManifest. Those responsible for the name have already been sacked.</p>
 <p>MetaManifest basically contains hashes of every file in the tree, either
@@ -127,19 +132,27 @@ are protected.</p>
 <p>In the following, the MetaManifest file is a file named 'Manifest',
 located at the root of a repository.</p>
 <div class="section" id="procedure-for-creating-the-metamanifest-file">
-<h2><a class="toc-backref" href="#id4">Procedure for creating the MetaManifest file:</a></h2>
+<h2><a class="toc-backref" href="#id5">Procedure for creating the MetaManifest file:</a></h2>
+<div class="section" id="summary">
+<h3><a class="toc-backref" href="#id6">Summary:</a></h3>
+<p>The objective of creating the MetaManifest file(s) is to ensure that
+every single file in the tree occurs in at least one Manifest.</p>
+</div>
+<div class="section" id="process">
+<h3><a class="toc-backref" href="#id7">Process:</a></h3>
 <ol class="arabic simple">
 <li>Start at the root of the Gentoo Portage tree (gentoo-x86, although
 this procedure applies to overlays as well).</li>
 <li>Initialize two unordered sets: COVERED, ALL.<ol class="arabic">
-<li>'ALL' will contain every file in the tree.</li>
-<li>'COVERED' will contain every file that is mentioned in an existing
-Manifest2.</li>
+<li>'ALL' shall contain every file that exists in the present tree.</li>
+<li>'COVERED' shall contain EVERY file that is mentioned in an existing
+Manifest2. If a file is mentioned in a Manifest2, but does not
+exist, it must still be included. No files should be excluded.</li>
 </ol>
 </li>
 <li>Traverse the tree, depth-first.<ol class="arabic">
 <li>At the top level only, ignore the following directories: distfiles,
-packages, local</li>
+packages, local.</li>
 <li>If a directory contains a Manifest file, extract all relevant local
 files from it (presently: AUX, MISC, EBUILD; but should follow the
 evolution of Manifest2 entry types per [#GLEP60]), and place them
@@ -171,22 +184,28 @@ for further notes].</li>
 </ol>
 </li>
 </ol>
+</div>
+<div class="section" id="notes">
+<h3><a class="toc-backref" href="#id8">Notes:</a></h3>
 <p>The above does not conflict the proposal contained in GLEP33, which
 restructure eclasses to include subdirectories and Manifest files, as
 the Manifest rules above still provide indirect verification for all
 files after the GLEP33 restructuring if it comes to pass.</p>
-<p>If other Manifests are added (such as per-category, per first-level
-directory, or protecting versioned eclasses), the size of the
-MetaManifest will be greatly reduced, and this specification was written
-with such a possible future addition in mind.</p>
+<p>Additional levels of Manifests are required, such as per-category, and
+in the eclasses, profiles and metadata directories. This ensures that a
+change to a singular file causes the smallest possible overall change in
+the Manifests as propagated. Creation of the additional levels of
+Manifests uses the same process as described above, simply starting at a
+different root point.</p>
 <p>MetaManifest generation will take place as part of the existing process
 by infrastructure that takes the contents of CVS and prepares it for
 distribution via rsync, which includes generating metadata. In-tree
-Manifest files are not checked at this point, as they are assumed to be
-correct.</p>
+Manifest files are not validated at this point, as they are assumed to
+be correct.</p>
+</div>
 </div>
 <div class="section" id="verification-of-one-or-more-items-from-the-metamanifest">
-<h2><a class="toc-backref" href="#id5">Verification of one or more items from the MetaManifest:</a></h2>
+<h2><a class="toc-backref" href="#id9">Verification of one or more items from the MetaManifest:</a></h2>
 <p>There are two times that this may happen: firstly, immediately after the
 rsync has completed - this has the advantage that the kernel file cache
 is hot, and checking the entire tree can be accomplished quickly.
@@ -194,7 +213,7 @@ Secondly, the MetaManifest should be che
 package.</p>
 </div>
 <div class="section" id="procedure-for-verifying-an-item-in-the-metamanifest">
-<h2><a class="toc-backref" href="#id6">Procedure for verifying an item in the MetaManifest:</a></h2>
+<h2><a class="toc-backref" href="#id10">Procedure for verifying an item in the MetaManifest:</a></h2>
 <p>In the following, I've used term 'M2-verify' to note following the hash
 verification procedures as defined by the Manifest2 format - which
 compromise checking the file length, and that the hashes match. Which
@@ -231,8 +250,8 @@ directly and indirectly) by the ebuild.<
 </ol>
 </li>
 </ol>
-<div class="section" id="notes">
-<h3><a class="toc-backref" href="#id7">Notes:</a></h3>
+<div class="section" id="id1">
+<h3><a class="toc-backref" href="#id11">Notes:</a></h3>
 <ol class="arabic simple">
 <li>For initial implementations, it is acceptable to check EVERY item in
 the eclass and profiles directory, rather than tracking the exact
@@ -249,20 +268,27 @@ explicitly declares what files from the 
 </div>
 </div>
 <div class="section" id="implementation-notes">
-<h1><a class="toc-backref" href="#id8">Implementation Notes</a></h1>
+<h1><a class="toc-backref" href="#id12">Implementation Notes</a></h1>
 <p>For this portion of the tree-signing work, no actions are required of
 the individual Gentoo developers. They will continue to develop and
 commit as they do presently, and the MetaManifest is added by
 Infrastructure during the tree generation process, and distributed to
 users.</p>
+<p>Any scripts generating Manifests and the MetaManifest may find it useful
+to generate multiple levels of Manifests in parallel, and this is
+explicitly permitted, provided that every file in the tree is covered by
+at least one Manifest or the MetaManifest file. The uppermost
+Manifest (MetaManifest) is the only item that does not occur in any
+other Manifest file, but is instead GPG-signed to enable it's
+validation.</p>
 <div class="section" id="metamanifest-and-the-new-manifest2-filetypes">
-<h2><a class="toc-backref" href="#id9">MetaManifest and the new Manifest2 filetypes</a></h2>
+<h2><a class="toc-backref" href="#id13">MetaManifest and the new Manifest2 filetypes</a></h2>
 <p>While [#GLEP60] describes the addition of new filetypes, these are NOT
 needed for implementation of the MetaManifest proposal. Without the new
 filetypes, all entries in the MetaManifest would be of type 'MISC'.</p>
 </div>
 <div class="section" id="timestamps-additional-distribution-of-metamanifest">
-<h2><a class="toc-backref" href="#id10">Timestamps &amp; Additional distribution of MetaManifest</a></h2>
+<h2><a class="toc-backref" href="#id14">Timestamps &amp; Additional distribution of MetaManifest</a></h2>
 <p>As discussed by [C08a,C08b], malicious third-party mirrors may use the
 principles of exclusion and replay to deny an update to clients, while
 at the same time recording the identity of clients to attack.</p>
@@ -284,19 +310,19 @@ verification process. The decision about
 user-configuration setting, with the ability to override.</p>
 </div>
 <div class="section" id="metamanifest-size-considerations">
-<h2><a class="toc-backref" href="#id11">MetaManifest size considerations</a></h2>
+<h2><a class="toc-backref" href="#id15">MetaManifest size considerations</a></h2>
 <p>With only two levels of Manifests (per-package and top-level), every
 rsync will cause a lot of traffic transferring the modified top-level
 MetaManifest. To reduce this, first-level directory Manifests are
-strongly recommended. Alternatively, if the distribution method
-efficiently handles small patch-like changes in an existing file,
-using an uncompressed MetaManifest may be acceptable (this would
-primarily be distributed version control systems). Other suggestions
-in reducing this traffic are welcomed.</p>
+required. Alternatively, if the distribution method efficiently handles
+small patch-like changes in an existing file, using an uncompressed
+MetaManifest may be acceptable (this would primarily be distributed
+version control systems). Other suggestions in reducing this traffic are
+welcomed.</p>
 </div>
 </div>
 <div class="section" id="backwards-compatibility">
-<h1><a class="toc-backref" href="#id12">Backwards Compatibility</a></h1>
+<h1><a class="toc-backref" href="#id16">Backwards Compatibility</a></h1>
 <ul class="simple">
 <li>There are no backwards compatibility issues, as old versions of
 Portage do not look for a Manifest file at the top level of the tree.</li>
@@ -306,7 +332,7 @@ conducted easily.</li>
 </ul>
 </div>
 <div class="section" id="thanks">
-<h1><a class="toc-backref" href="#id13">Thanks</a></h1>
+<h1><a class="toc-backref" href="#id17">Thanks</a></h1>
 <p>I'd like to thank the following people for input on this GLEP.</p>
 <ul class="simple">
 <li>Patrick Lauer (patrick): Prodding me to get all of the tree-signing
@@ -318,7 +344,7 @@ work finished, and helping to edit.</li>
 </ul>
 </div>
 <div class="section" id="references">
-<h1><a class="toc-backref" href="#id14">References</a></h1>
+<h1><a class="toc-backref" href="#id18">References</a></h1>
 <dl class="docutils">
 <dt>[C08a] Cappos, J et al. (2008). &quot;Package Management Security&quot;.</dt>
 <dd>University of Arizona Technical Report TR08-02. Available online
@@ -329,7 +355,7 @@ from: <a class="reference external" href
 </dl>
 </div>
 <div class="section" id="copyright">
-<h1><a class="toc-backref" href="#id15">Copyright</a></h1>
+<h1><a class="toc-backref" href="#id19">Copyright</a></h1>
 <p>Copyright (c) 2006-2010 by Robin Hugh Johnson. This material may be
 distributed only subject to the terms and conditions set forth in the
 Open Publication License, v1.0.</p>
@@ -340,7 +366,7 @@ Open Publication License, v1.0.</p>
 <div class="footer">
 <hr class="footer" />
 <a class="reference external" href="glep-0058.txt">View document source</a>.
-Generated on: 2010-01-13 03:27 UTC.
+Generated on: 2010-01-31 07:53 UTC.
 Generated by <a class="reference external" href="http://docutils.sourceforge.net/">Docutils</a> from <a class="reference external" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a> source.
 
 </div>