Title: Manifest2 compression
Version: $Revision: 1.6 $
Last-Modified: $Date: 2010/01/31 09:55:43 $
Author: Robin Hugh Johnson <email@example.com>
Type: Standards Track
Created: July 2008
Updated: October 2008, January 2010
Post-History: December 2009, January 2010
Deals with compression of large Manifest2 files.
With the introduction of MetaManifest, and full-tree Manifest coverage,
we are faced with the possibility of having very large Manifests.
Preliminary experiments with MetaManifest, show that with just the
existing per-package Manifests, the full MetaManifest (top-level only,
no first-level sub directories), for a tree including metadata/, exceeds
8MiB in size. Applying common compression can achieve a 50-60% reduction
in this size.
Additionally, some of the larger already-existing Manifests in the tree
can also be reduced.
This GLEP is not mandatory for the tree-signing specification, but
instead helps to cut down the size impact of large Manifest2 files, some
of which are already present in the tree. As such, it is also able to
stand on it's own.
Creation of compressed Manifests:
32KiB is suggested as a arbitrary cut-off point to start generating
compressed Manifest2 files.
The compression must only applied during the creation of a tree intended
for end users. No Manifests stored in a VCS should be compressed in the
VCS. For the main gentoo-portage tree, this means that the compressed
Manifests should be generated using the CVS to Rsync process.
The Manifest compression process is required to ensure that inconsistent
compressed versions do not exist.
Validation of Manifests:
When searching for a Manifest2 file, if the basename form does not
exist, the package manager should search in the same location using
common compressed suffixes, and use the compressed file in place of the
gzip, bzip2, lzma, xz should all be supported if available on the given
platform. In the case that multiple versions exist, the package manager
should simply pick one - they should be identical, differing only in
Example Results with a 32KiB cut-off, gzip algorithm
As of 2010/01/30, the suggested cut-off would impact the following 21 existing
Manifests, for a saving of nearly 900KiB::
Additionally, with the MetaManifest proposal, the following new manifests would
also be compressed, for a saving of nearly 4MiB::
The package Manifests should also be maintained as ONLY uncompressed in
For processing of all existing per-package Manifests, if compression is
used, it should be done in parallel to the existing Manifests, to
provide for a changeover period. Newer versions of Portage may later
choose to exclude all non-compressed Manifests during emerge --sync if
compressed versions are guaranteed to exist on the servers.
MetaManifests may come into existence as compressed from the start, as
do not have an backwards compatibility issues.
As a side note, this breaks all manual interaction with Manifests
such as grep, and so should only be applied to large Manifest2 files,
such as the MetaManifest.
.. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format.
Copyright (c) 2008-2010 by Robin Hugh Johnson. This material may be
distributed only subject to the terms and conditions set forth in the
Open Publication License, v1.0.
vim: tw=72 ts=2 expandtab: