Gentoo Logo
Gentoo Spaceship




Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Headers:
To: gentoo-dev@g.o
From: "Robin H. Johnson" <robbat2@g.o>
Subject: Re: GLEP61 - Manifest2 compression
Date: Mon, 8 Feb 2010 05:23:03 +0000
On Sun, Feb 07, 2010 at 05:02:22PM -0800, Brian Harring wrote:
> On Sun, Jan 31, 2010 at 10:04:40AM +0000, Robin H. Johnson wrote:
> > Changes:
> > - This GLEP can stand independently of GLEP58.
> > - Add XZ to compression types list.
> > - Move cutoff to 32KiB. Provide size example w/ 32KiB+gzip.
> > - Split specification into generation and validation.
> One concern w/ this glep- the intention seems to be to reduce on disk 
> space requirements but the addition of compression raises questions 
> for rsync transferance of the manifests.
> 
> Have you done any testing to quantify how much of an increase in rsync 
> bandwidth this will add?  Specifically thinking about the metamanifest 
> on this one.
The top-level MetaManifest, in the case of fully split (eg a Manifest in
every 1st-level directory $CAT/Manifest and the other dirs), is only
33KiB.

21 existing packages have Manifests larger than 32KiB, texlive stuff
come in the worst here.

I do agree, that depending on the block alignment, there is an increase
in transfer size in some cases, but I have not conducted rigorous tests
to work out long-term statistics on changes.

The more aggressively that Manifests are added to each subdirectory
tree, the less Manifest2 compression is actually required, as the
individual Manifests are more likely to fit within the size limit.

With the 1st-level case again, here's a size count breakdown for the new
(Meta)Manifests:
>=32KiB: 27
>=64KiB: 10
>=128KiB: 3 (dev-perl @ 179KiB, metadata/ @ 5.2MiB, profiles/ @ 300KiB)

I think the best course of action is to end up generating the compressed
MetaManifests when we start generated the MetaManifests themselves, but
not placing them into the tree yet. Instead simply use them to measure
rsync transfer size impact on the generation server and produce
statistics to see if the cutoff could benefit from being altered, or if
the disk space should be wasted in favour of smaller transfer size.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@g.o
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85


Replies:
Re: GLEP61 - Manifest2 compression
-- Brian Harring
References:
Tree-signing GLEPs update
-- Robin H. Johnson
GLEP61 - Manifest2 compression
-- Robin H. Johnson
Re: GLEP61 - Manifest2 compression
-- Brian Harring
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: GLEP61 - Manifest2 compression
Next by thread:
Re: GLEP61 - Manifest2 compression
Previous by date:
Re: Calling unknown commands in an ebuild
Next by date:
Re: GLEP61 - Manifest2 compression


Updated Jun 29, 2012

Summary: Archive of the gentoo-dev mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.