On Fri, Nov 18, 2005 at 10:01:27PM -0800, Robin H. Johnson wrote:
> Hi,
> 
<snip logic args for why alt chfs are needed>
Nobody is debating the need for it, don't really need to go back over 
that terrain- alt chf support in manifest/digest is needed so that we 
can transition cleanly from chf to cfh long term as algos are broken.

Hopefully that statement will shut up another pointless discussion 
about preimaging md5 (we don't care, alt chf is needed regardless of 
md5's current state).
:)

Back to signing...

> Solving the problem isn't hard. All we need is to consider each change
> to a Manifest separately.
> 
> Ergo, instead of a Manifest being re-generated each time, it needs to
> act like a FIFO queue.
> Each queue element consists of:
> - checksum/existing Manifest element of items changed in that action ONLY (however it should be possible to forcibly include files).
> - Signature around the above checksum.
> 
> So now the new Manifest structure looks roughly like this (abbreviated):
> -- PGP
> MD5 ...
> MD5 ...
> -- SIG
> -- SIG
> -- PGP
> MD5 ...
> -- SIG
> -- SIG
> etc.
>
> This has one important implication for backwards compatibility in
> checking of Manifests.
> In the case that a filename appears more than once in the file, only
> the last instance of it should be used, as that is the one that relates
> to the current version of the file. It's 4 lines of code in the current
> portage that need to be removed for this to work (see my -core post for
> where exactly).
Last is the fun part.  You're example above is (effectively) unordered 
signed chunks; if you're doing 'use the last', you need to enforce 
ordering of the signed chunk.

Am I missing something, or is the proposal above unordered in the 
file?

> Generation of the above is reasonably simple, just make the checksums in
> a string, clear-sign via gpg separately/tmpfile, append to Manifest.
> 
> There is one last part that does need to be taken care of.
> At what point is it safe to remove a checksum/signature from the
> Manifest?
> You cannot remove a single checksum from inside a signature block, as
> that invalidates the signature.
> So instead, now we need to wait for the existence more recent
> checksums/signatures to exist for every item in a block before the old
> block can be removed 

Glep19 people, if they ever get moving need a way to tag on a signed 
list of removals from the manifest also- so effectively a del of some 
sort is needed (again, complicated by unordered).

> Notes:
> There were some concerns about the speed of Manifest checking.
> I did some simple benchmarking, using all existing signed Manifests
> (40% of the tree). Using a Pentium-D 3Ghz, running on only one core, it
> took 43 seconds to check all 4000 Manifests. 90% of the time was in the
> setup/teardown for gpg.
> 
> Getting around the setup/teardown time problem basically means we need
> something of our own to interface with the gpg API (gpgme). This is not
> too hard, GPG ships with an example that is most of the way there
> already - 'gpgv'.
> 
> I'm comfortable with the GPG/GPGME codebase, and writing crypto-related
> code, so I'm going to tackle that, starting after I've eaten dinner
> tonight.
Other part that's going to be fun is the fifo pipelining of commits- 
aside from verification at commit time, with the proposed process, 
will need to push that file to osprey so that it can act on the list 
of commands (doing cvs up and re-verification, then updating of rsync 
tree)...

> Even still, re-checking every digest in the tree should not happen on
> every CVS->rsync window. It's computationally pointless. Just check the
> changes.
...and the list of deltas (essentially) will need to be pushed to 
users.
This one's a bit interesting; thoughts on how?  Appending to an ever 
growing file in the tree won't really cut it, imo...

Unless you're thinking about trying to track what rsync updates, which 
is going to be fun since librsync is kind of dead (and incompatible).
~harring