Re: [gentoo-dev] digest reorganization and enhancements

[Nicholas Jones]

This entire issue changes direction based on how hard and from
where the pushes come. There are 3 general directions, as I see
it.

1. Unify Manifest & digests.
2. Create a new digest file.
3. Break backwards compatibility.

I'd prefer 3. I'm pretty sure it's the first time I've
ever suggested that or had it happen since I started
maintaining portage. :-/

> SRC_URI portage-2.0.51_rc7.tar.bz2 274572 MD5 1234 SHA1 abcd RMD160 9876
> EBUILD portage-2.0.51_rc7.ebuild 11806 MD5 xyz SHA1 fifteen
>
> (using fake checksums for readability).

My suggestion involves a break in compatibility after a expedited
transition period. Kind of a bummer really.

A primary issue with unified digest/Manifest is that you force
Maintainers to verify tarballs and be liable for them. With the
seperated digest/Manifest we can begin signing both the Manifest
and the digests allowing the introducer of the package to be
liable for the tarballs and leave the Manifest, Ebuilds, and
supporting files to be verified by the {arch,package} Maintainer.

With signatures moving along slowly, we have the capability
to revoke a signature and all potentially infected packages.
If Arch maintainers are constantly removing/changing the
signatures on a digest, you will have no obvious knowledge
of which files they have touched a particular file.

> Maybe the system can also be extended to incorporate
> GLEP 25 without adding a ton of new files, I'd need some
> input from Brian on that issue.

I'd rather this be an external sync tool than have portage
need to deal with it. Supporting it via an uncompressed MD5
is easily done though. We're breaking compat with my suggestion
anyway.

> And finally a summarizing list of reasons for the format:
> - keep all checksums of a package in one place
> - removes one level of indirection for signing

These are good and bad depending on how you look at it.

> - digest generation currently recreates the Manifest anyway
> - removing files from the tree

You lose direct info on who originally verified the tarballs.

> - allows for easy addition of new digest algorithms
> - any syntax modification to the current digest files brings
> compability problems with all currently existing portage
> versions while Manifest changes do not

Sadly, I missed fixing digests when I did the Manifest code.

The fix is simply a matter of time. But for all realistic,
time-constrained, viewpoints, it is accurate. It's a one-time
adjustment of the digest format.

SHA1 and many others can be added to the digests after 2.0.51
comes out. It supports digests of varying forms but there
should be a reasonable delay before implementation as the
impact on the slow-to-update user will be manual. A script
could be provided, I imagine. The rescue documentation still
applies.

> - potential to discover file collisions easier (currently
> you can have the same file in two digests with different
> checksums, not a real problem yet though)

Tools exist for detecting this. md5_check in bin/.
They should probably be integrated into repoman.


So...
The overall scheme:

  1. Transition quickly into 2.0.51.
  2. Announce the strong-recommendation/need to update.
  3. Post new rescue tarballs.
  4. Wait 2-4 weeks.
  5. Announce again... ?
  6. Enable portage's SHA1 creation.
  7. Break compat in the tree.
  8. Hopefully never do that again.

--NJ