Gentoo Archives: gentoo-dev

From: "Robin H. Johnson" <robbat2@g.o>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] GLEP61 - Manifest2 compression
Date: Mon, 08 Feb 2010 05:23:47
In Reply to: Re: [gentoo-dev] GLEP61 - Manifest2 compression by Brian Harring
On Sun, Feb 07, 2010 at 05:02:22PM -0800, Brian Harring wrote:
> On Sun, Jan 31, 2010 at 10:04:40AM +0000, Robin H. Johnson wrote: > > Changes: > > - This GLEP can stand independently of GLEP58. > > - Add XZ to compression types list. > > - Move cutoff to 32KiB. Provide size example w/ 32KiB+gzip. > > - Split specification into generation and validation. > One concern w/ this glep- the intention seems to be to reduce on disk > space requirements but the addition of compression raises questions > for rsync transferance of the manifests. > > Have you done any testing to quantify how much of an increase in rsync > bandwidth this will add? Specifically thinking about the metamanifest > on this one.
The top-level MetaManifest, in the case of fully split (eg a Manifest in every 1st-level directory $CAT/Manifest and the other dirs), is only 33KiB. 21 existing packages have Manifests larger than 32KiB, texlive stuff come in the worst here. I do agree, that depending on the block alignment, there is an increase in transfer size in some cases, but I have not conducted rigorous tests to work out long-term statistics on changes. The more aggressively that Manifests are added to each subdirectory tree, the less Manifest2 compression is actually required, as the individual Manifests are more likely to fit within the size limit. With the 1st-level case again, here's a size count breakdown for the new (Meta)Manifests:
>=32KiB: 27 >=64KiB: 10 >=128KiB: 3 (dev-perl @ 179KiB, metadata/ @ 5.2MiB, profiles/ @ 300KiB)
I think the best course of action is to end up generating the compressed MetaManifests when we start generated the MetaManifests themselves, but not placing them into the tree yet. Instead simply use them to measure rsync transfer size impact on the generation server and produce statistics to see if the cutoff could benefit from being altered, or if the disk space should be wasted in favour of smaller transfer size. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee & Infrastructure Lead E-Mail : robbat2@g.o GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85


Subject Author
Re: [gentoo-dev] GLEP61 - Manifest2 compression Brian Harring <ferringb@×××××.com>