Re: [gentoo-dev] GLEP58 - MetaManifest

[Robin H. Johnson]

On Mon, Feb 01, 2010 at 11:27:01PM -0700, Denis Dupeyron wrote:
> You'll find below an email from solar to Robin about MetaManifest. I'm
> adding it to this thread (with solar's authorization) as it seems
> pertinent.
>
> Denis.
>
> On Thu, Jan 21, 2010 at 6:51 PM, Ned Ludd <solar@g.o> wrote:
> > Robin,
> >
> > I recall you wanted me to mail you what we talked about last nite in
> > #gentoo-portage and I'll CC: the council so they have an idea what to
> > maybe expect.
> >
> > So in our talking last night we discussed the fact that if the Manifest
> > format has to change why not just get rid of it all together, and save
> > some serious in tree space with the new MetaManifest's taking over all
> > together. This would include MetaManifest's at the 2-level.
> > You said the MetaManifest would need about 4 fields in them to describe
> > the distfiles etc. Devs would still push normal Manifest's to the cvs
> > tree so DIST can be obtained by the backend infra scripts. But those
> > Manifest's could be dropped from the mirroring. if [ -e CVS ] then
> > portage would need to use the existing Manifest's
First, I'd like to clarify one things for all other readers, as it isn't
clear for anybody else just reading this email.
================
Solar's proposal does the following:
1. Tree in CVS/VCS:
- drop ALL Manifest2 lines _EXCEPT_ DIST.
2. Tree available via rsync:
- Manifests at the following locations ONLY:
	- /MetaManifest
	- /${CAT}/Manifest
	- /profiles/Manifest
	- /eclasses/Manifest
	- /metadata/cache/${CAT}/Manifest
	- /metadata/glsa/Manifest
- Data from ALL Manifests get moved to one of the above.
- MISC/EBUILD etc (non-DIST) lines generated at the same time that the
  rsync tree is prepared.
3. Net savings of approximately 13000 inodes, as the per package
   Manifest data is now one level up, saving the inode from the package.
================

Now, I believe that this above should be possible WITHIN the framework
of my proposed MetaManifest changes.

I specifically stated in GLEP58:
===
The objective of creating the MetaManifest file(s) is to ensure that
every single file in the tree occurs in at least one Manifest.
===

My proposals did not cover removing other Manifest files per solar's
suggestion, as I perceived that to be a much larger objective than my
goal of actually securing the existing tree distribution.

I am entirely open to solar's suggestions, in an additional GLEP, as
they will require that Portage support IS fully in place, because old
versions WILL fail on a tree without per-package Manifest.

> > This method would hands down win my vote. As you know I'm not a fan of
> > format changes in general as they can make the Gentoo experience suck,
> > but if we are going to change formats. Lets do it right.
A potential plan for GLEP58 and solar's changes would be:
1. Council approves GLEP58.
2. Portage support is added, we add MetaManifests everywhere needed
   (top-level, categories, metadata, eclass etc) in the tree.
3. Old Portage versions still work at this point, because they ignore
   the other Manifest files.
4. Wait 6-12 months for Portage upgrade cycle.
5. Change the content of the MetaManifests to be per solar's proposal.
6. Drop per-package Manifests from the tree.

Thus there is ZERO breakage. 

A similar timeline is required for ALL of the other GLEPs I have proposed.
GLEP59 - Hashes:
- Can add new hashes right now.
- Some of the old hashes we can remove right now.
- Have to keep just one old hash for old Portage to still work.
GLEP60 - Filetypes:
- Can add new types right now.
- Cannot remove ANY types for a full upgrade cycle.
GLEP61 - Compression:
- (uncofirmed) Cannot add the compressed files in per-package locations until
   the upgrade cycle is done, as old Portage will complain about their existence.

> The only downside I can see in this method is for people like drobbins
> who mirror our tree but overlay right on top of it then provide it back
> out.  In such cases we should provide our backend scripts to the public
> so they can re MetaManifest.
My MetaManifest generation script is already public. I do agree that we could
do better in documenting and publishing our older rsync generation scripts.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@g.o
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85