Gentoo Archives: gentoo-dev

From: Marius Mauch <genone@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] Multi hash support in portage - status
Date: Thu, 24 Nov 2005 00:07:27
1 So, along with the gpg signing stuff came along again the question to
2 have multiple hash formats in digests and manifests.
4 Current status is that portage only generates MD5 checksums and can
5 verify both MD5 and SHA1 checksums. Creation of SHA1 is also possible
6 but has so far been disabled as older portage versions would break if
7 they found a non-MD5 line in digest files (this was fixed somewhere
8 last year in the .51 series).
10 Ok I have three modifications that are pending to go into portage:
11 - The first simply enables creation of SHA1 checksums (and others if
12 implemented like with the second mod), if you want to try it yourself
13 see the attached patch.
14 - Another thing that has been requested often is to offer even more
15 hashing functions. Earlier today I sent a patch to the
16 gentoo-portage-dev list that adds optional support for SHA256 and
17 RMD160 if dev-python/pycrypto is installed on the system.
18 - The last and most intrusive change is support for a new enhanced
19 Manifest format (called Manifest2 for now). Don't worry, there will be
20 GLEP and more info before this gets added, I just list here for
21 reference below
23 The first two changes are ready to be added and deployed (in .54), so
24 without looking at the third one it would be a no brainer. But of
25 course there is a drawback: The current Manifest/digest format is quite
26 inefficient wrt storing multiple checksums as it repeats the filename
27 and filesize for every checksum added, it looks like this:
29 MD5 82e806ed62f0596fb7bef493d225712f metadata.xml 269
30 RMD160 39d775de55f9963f8946feaf088aa0324770bacb metadata.xml 269
31 SHA1 4fd7b285049d0e587f89e86becf06c0fd77bae6d metadata.xml 269
32 SHA256 3787959f4a775b1e787b35ff8380949d8f68bd67b81c2cf5a748733c9740cb94
33 metadata.xml 269
35 The Manifest2 format solves this problem (and has some other benefits)
36 by listing all checksums on one line:
38 MISCFILE metadata.xml 269 RMD160
39 39d775de55f9963f8946feaf088aa0324770bacb SHA1
40 4fd7b285049d0e587f89e86becf06c0fd77bae6d SHA256
41 3787959f4a775b1e787b35ff8380949d8f68bd67b81c2cf5a748733c9740cb94 MD5
42 82e806ed62f0596fb7bef493d225712f
44 Not much of a difference you might say, but this is just looking at a
45 pure Manifest2 entry. To keep compability with existing portage
46 versions we have to list both the old format and the new format in the
47 Manifest (digests are handled differently with Manifest2, but the
48 concept also applies to them), which can potentially increase the tree
49 size by ~10% (at a guess). I'm talking about actual data size here, not
50 required discspace.
51 And before you ask "why manifest2 if it adds this overhead?",
52 the main point isn't the new format but a long term reduction of the
53 tree size by removing the digest files (but wait for the GLEP to discuss
54 this).
56 So much for background information, now to the actual question:
57 Would you rather have now the ability to create multi-hash digests and
58 Manifests with the result of a short and mid-term larger portage tree
59 (in the long term the format will be phased out hopefully) or rather
60 wait for Manifest2 support (which will definitely include multi hash
61 support)?
63 Basically just getting some feedback before adding it and later getting
64 the complaints about bloating the tree ;)
66 Note that this is (technically) completely unrelated to gpg signing of
67 Manifests, so any gpg related bitching doesn't belong here.
68 Generally only reply here if you're replying to the question I posted
69 (implementation discussions belong on gentoo-portage-dev, and the
70 quota for whining/trolling/flaming for this month was already exceeded).
72 Marius
74 --
75 Public Key at
77 In the beginning, there was nothing. And God said, 'Let there be
78 Light.' And there was still nothing, but you could see a bit better.


File name MIME type
signature.asc application/pgp-signature