[gentoo-scm] Thin Manifests for git

[Nirbheek Chauhan]

(re-sending since I sent it using my gentoo account the first time;
but I'm subscribed with my gmail acount)

Since we're closer to completing the git migration now, I feel this
should be brought up again to discuss the best way of going about with
it.

Below is what I'm proposing; feel free to pick holes in it :)

=======
Concept:
=======
* Store only DIST checksums in Manifest for git
  - Due to the distributed nature of git, to do mischief, you need to
change every clone in the world to be successful
  - This has nothing to do with strength of the hash used by git
* Needs to be done after the move to git (see "Migration:" below)
* User's rsync-ed portage still needs full Manifests (see "How:" below)

-> Anything else that should be a part of this?

==========
Advantages:
==========
* Saves an extra step for the developer while testing
 - *edit* *manifest* *test* => *edit* *test*
* Reduces conflicts
 - Especially helpful in very active overlays where merging takes place
* Doesn't store redundant data
 - git's workflow already ensures the integrity of content due to the
distributed nature
 - No need to store checksums for files stored in the tree

-> Any other advantages?

====
How:
====
* EBUILD/MISC/AUX checksums are not stored in the git tree
 - Generate them on the rsync side before distribution
* Portage support for thin manifests
 - If Manifest contains *only* DIST checksums *and* the tree is a
dvcs tree; use only those
   + This is for local testing by devs, and overlays
   + Make emerge check if the $dvcs tree has conflicts and warn/error
out (optional)
      ~ Since we don't have file manifests, portage leaves the work
of verification to the dvcs
      ~ If the dvcs is in an uncertain state, portage should detect that
         > Might want to skip if there are performance problems
 - If Manifest contains any other checksums; old behaviour
   + Users of rsync portage will see no change

-> Anything else that needs to be done?

========
Migration:
========
* We tell all devs to move to a newer portage immediately after the
git conversion
 - Release a new version of portage much before the migration to git
with support for all this
 - All Manifests in-tree are pruned in one massive commit after the
conversion to git, and before opening the tree for dev push use.
 - Push hook to reject commits with old manifests
* Overlay users
 - Overlay users will have to move to a newer portage which supports
thin manifests
 - Need to find a way to inform users of this

-> Anything else in migration?

===========
Compatibility:
===========

* Compatible with Manifest2 (per-category manifests)
* Per-dev signed commits become obsolete since we most of the
checksums are generated during rsync conversion
 - Aim of signed commits is to stop MITM attacks on rsync mirrors
 - With Manifest2, we can sign the grouped Manifest with a "Gentoo" gpg key
   + This alleviates MITM attacks

-> Any other Manifest-related proposals this might conflict with?

======
Coding:
======
* Portage support for thin manifests
 - Most important; a release must be done atleast 1 month before the migration
 - Testing can be done with overlays
* Repoman support for thin manifests
 - Testing can be done with overlays
* Push hooks to reject old-style manifests
* Rsync-side scripts for generation of "Fat" Manifests
* Rsync-side scripts for "Gentoo" signing of Manifests (conditional)

-> Anything else that needs coding?

--
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team