Re: [gentoo-dev] proposed md5sum change

[Brian Harring]

Replies are below...

On Thu, 2003-06-12 at 03:15, Paul de Vrieze wrote:
> On Thursday 12 June 2003 09:53, Evan Powers wrote:
> > On Thursday 12 June 2003 02:56 am, Seemant Kulleen wrote:
> > > Anyway, the current approach keeps it simple in that the md5sum is off
> > > the *item(s) that is/are downloaded*.  The first reason I can see is what
> > > I stated above.  There are other reasons I can see as well.  You know,
> > > immediately upon fetching the set of source items that they are bad.  So,
> > > no disk i/o or cpu cycles are spent in the unpacking; and no potentially
> > > nasty code is even untarred on the system, yet.
> >
> > Well, I can think of a way to address this part of the issue, anyway.
> >
> > * The portage tree still has MD5 digests for the item(s) which is/are
> > downloaded.
> >
> This would make things easy for most people, and would save also lots of cpu
> cycles.
Agreed.  Ignoring the cpu cost of decompression, gzip decompressing and
md5ing is fairly close to i/o at least for my machine, which is an
xp1700 w/ about 40mb/s throughput on my hd.  Bzip2 (as jjw pointed out)
is an entirely different beast, decompressing it is not close to i/o
speeds.  At this stage of the game, recompressed/patched tarballs would
be the minority- down the line assuming diffing/patching takes off, this
might be something to think about.
 
> > * After downloading, emerge executes a (user specified?) program on the
> > newly downloaded file. This program applies some transform to the file;
> > maybe it decompresses whatever format the file is in and re-compresses it
> > with bzip2, or maybe it only format-shifts files which are over a certain
> > size threshold, whatever.
> >
> Allways fun, maybe we would also have a list of allowed/blacklisted extensions
> like RPM files which are allready compressed. (Or openoffice files/jar-balls
> which loose validity if compressed in a different format, but could be
> recompressed in zip format)
>
> > * Next, emerge adds a new record to a database (text, one record per line,
> > for example) somewhere in /var. This database has the original name of the
> > downloaded file, the original MD5 digest, the new name, and the new MD5
> > digest.
> >
> Maybe extending the serverside digest to also include an unpacked digest
> (where applicable) would be smarter in validation of patch based and oddball
> cases. (This doesn't mean that client-side digesting isn't useful, it is for
> not having to unpack first).
How about this, and mind you this is just for dealing w/ md5sum's-
instead of doing any db-style stuff, just create a file along side (w/in
the distfile dir most likely) that contains the uncompressed data's
md5sum.  If you go about creating a db type setup, you're going to run
into major issues in an environment where the distfiles dir is shared
out to other systems since you're not going to be sharing the db.
Basically I could see this- a simple script that a user can use to
convert non-gz distfiles to bzip2 tarballs which creates the file (think
linux-2.4.19.tar.bz2 and linux-2.4.19.tar.md5)... for distfile
diffing/patching, it uses the same method.  If the reconstructed and
recompressed verion's md5 matches what portage has, hoozah, no need to
create the file- if not (say upgrading openoffice), we create the file.
Also, it dawned on me that md5summing the data has an added bonus of
being indifferent to the patching/differencing method.  In other words,
we could use the standard unified diff's that are provided for the
kernel versions for instance.

As for transforming, would it really be needed?  I've spent a bit of
time rooting through the distfile dir and I don't recall seeing
non-versioned names, although as always, I could be wrong.


--
gentoo-dev@g.o mailing list