| 1 |
On Sun, Feb 07, 2010 at 05:02:22PM -0800, Brian Harring wrote: |
| 2 |
> On Sun, Jan 31, 2010 at 10:04:40AM +0000, Robin H. Johnson wrote: |
| 3 |
> > Changes: |
| 4 |
> > - This GLEP can stand independently of GLEP58. |
| 5 |
> > - Add XZ to compression types list. |
| 6 |
> > - Move cutoff to 32KiB. Provide size example w/ 32KiB+gzip. |
| 7 |
> > - Split specification into generation and validation. |
| 8 |
> One concern w/ this glep- the intention seems to be to reduce on disk |
| 9 |
> space requirements but the addition of compression raises questions |
| 10 |
> for rsync transferance of the manifests. |
| 11 |
> |
| 12 |
> Have you done any testing to quantify how much of an increase in rsync |
| 13 |
> bandwidth this will add? Specifically thinking about the metamanifest |
| 14 |
> on this one. |
| 15 |
The top-level MetaManifest, in the case of fully split (eg a Manifest in |
| 16 |
every 1st-level directory $CAT/Manifest and the other dirs), is only |
| 17 |
33KiB. |
| 18 |
|
| 19 |
21 existing packages have Manifests larger than 32KiB, texlive stuff |
| 20 |
come in the worst here. |
| 21 |
|
| 22 |
I do agree, that depending on the block alignment, there is an increase |
| 23 |
in transfer size in some cases, but I have not conducted rigorous tests |
| 24 |
to work out long-term statistics on changes. |
| 25 |
|
| 26 |
The more aggressively that Manifests are added to each subdirectory |
| 27 |
tree, the less Manifest2 compression is actually required, as the |
| 28 |
individual Manifests are more likely to fit within the size limit. |
| 29 |
|
| 30 |
With the 1st-level case again, here's a size count breakdown for the new |
| 31 |
(Meta)Manifests: |
| 32 |
>=32KiB: 27 |
| 33 |
>=64KiB: 10 |
| 34 |
>=128KiB: 3 (dev-perl @ 179KiB, metadata/ @ 5.2MiB, profiles/ @ 300KiB) |
| 35 |
|
| 36 |
I think the best course of action is to end up generating the compressed |
| 37 |
MetaManifests when we start generated the MetaManifests themselves, but |
| 38 |
not placing them into the tree yet. Instead simply use them to measure |
| 39 |
rsync transfer size impact on the generation server and produce |
| 40 |
statistics to see if the cutoff could benefit from being altered, or if |
| 41 |
the disk space should be wasted in favour of smaller transfer size. |
| 42 |
|
| 43 |
-- |
| 44 |
Robin Hugh Johnson |
| 45 |
Gentoo Linux: Developer, Trustee & Infrastructure Lead |
| 46 |
E-Mail : robbat2@g.o |
| 47 |
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 |
Archives