Re: [gentoo-dev] [RFC] DIGESTS metadata variable for cache validation

[Brian Harring]

On Tue, Feb 10, 2009 at 12:55:51PM -0800, Zac Medico wrote:
> Brian Harring wrote:
> > On Mon, Feb 09, 2009 at 11:55:41AM -0800, Zac Medico wrote:
> >> All that I can say right now is that I recall questions about it in
> >> the past from overlay maintainers (I don't have a list) and the
> >> funtoo project is the only one which I can name offhand.
> >>
> >> However, the ability to distribute cache via a vcs is only an
> >> ancillary feature which is made possible by the DIGESTS data. The
> >> DIGESTS data is useful regardless of the protocol that is used to
> >> distribute the cache, since it allows the cache to be properly
> >> validated for integrity. So, the real primary reason for introducing
> >> the DIGESTS data is to provide a proper solution for cases like bug
> >> #139134 [1] in which invalid metadata cache goes undetected.
> >
> > I'm sorry, but this proposal smells something awful.  Because of the
> > mtime requirement on cache entries you're proposing jamming another
> > 1.4MB into the cache for validation purposes (which should be 4x that
> > since a full checksum really should be in there) while trying to
> > maintain compatibility.
>
> As I've said before [1], 10 hex digits gives 1.1e12 possible
> combinations and that's probably sufficient for the given application.

And as I said before, I don't agree with you on it (repeating it over 
and over isn't going to convince the other side either).

The 1.4MB is more the concern then arguments over avalanche I might 
add.


> > Frankly, forget compatibility- the current format could stand to die.
> > The repository format is an ever growing mess- leave it as is and
> > work on cutting over to something sane.
>
> Changing the repository layout is a pretty radical thing to do.
> You're welcome to start a new subject for that if you'd like but I'd
> prefer to keep the scope of this thread focussed on the cache format
> for the existing repository layout.

Vacuous arguement via focusing on the 'layout' part rather then the 
repository whole I implied; you're stating that one should not 
discuss changing the repository standard/spec while arguing that 
repealing the requirement that cache mtime entries match ebuild 
mtime (part of the repository spec) should be the point of discussion.

The daft thing about this is that w/ effectively atomic sync (if the 
sync fails then mark the repo as screwed up till a sync completes), 
the current cache format can *still* do validation- no clue if 
paludis has it, but at least pkgcore and portage can handle this via 
awareness of the eclass stacking.

So for git vcses bundling metadata (a bad idea anyways to be storing 
generated content in the mainline vcs), your proposal allows them to 
use a cache.  For every other distribution mechanism that works fine, 
they wind up paying the cost for that corner case.  The 80 pays for 
the 20 isn't the normal form of the 80/20 rule ;)

Note that proper PM implementations *still* have to set the cache 
entries mtime for backwards compatibility w/ older PMs that don't 
support this new unversioned change thus muddying the implementation 
even further.

I reiterate, this belongs in a seperate repository format, along w/ 
the rest of the unversioned repository changes you've been pushing in 
(profile package.mask breaking all non portage PMs is a perfect 
example).


> > Overlay maintainers who want the latest/greatest obviously can convert
> > over also; one would hope their would be enough cleanup to make it
> > worth their time.
> >
> > As for the nasty gentoo-x86 compatibility, basically, do the
> > following:
> >
> > 1) maintain the existing cvs repo as is
> > 2) iron out what cleanup/restructuring is desired.  glep55 being
> > jammed in here is a potential for example.  Nail down the new repo
> > format basically (with an eye for translating the cvs repo to it on
> > the fly).
> > 3) use an eclass index holding the checksums, w/ the cache entries
> > referencing the index numbers rather (sorting the index by
> > consumption, meaning the more ebuilds using it the lower the index):
> > this brings the cache addition down to around 285KB (acceptable imo)
> > while giving full flexibility in the checksums available for eclasses.
> > This is assuming the current flat_list format is still in use in the
> > new repo...
>
> As previously discussed [2], having shared integrity data (as you
> suggest) has implications in terms of reduced simplicity and robustness.

The complexity arguement is a white elephant.   Rsync is the sole 
transport that has atomicity issues; the rest don't (when you check 
out from vcs, you get an exact rev effectively).  Rsync generation 
ought to be preparing the new snapshot then swapping it in, and if I 
recall correctly that's exactly what osprey does now (or whatever node 
y'all are using for generating gentoo-x86 these days).

The point there is that there are specific steps taken preparing the 
repo- those steps already ensure the snapshot/rev is complete prior to 
being available so there isn't real potential of catching it mid 
update.  Via that existing machinery, a shared index is *no issue*- 
the one spot it rears it's head is during a failed/partial sync (the 
repo should not be using in such a state since stale cache is the 
least concern at that point).


From where I'm sitting, the changes you've either slipped in, or want 
to slip in to the repository format are completely ignoring the past 
bad history of breakage doing such things, and ignoring why EAPI 
exists these days.

The reason a new format is realistically needed here is that existing 
PMS compliant implementations will wind up accessing the repo and 
behaving as if everything is fine and dandy without knowing the rules 
used to read no longer match the rules used to generate the repo.  
This is a no go, same reason EAPI awareness had to sit for a long ass 
time to ensure EAPI=1 would be properly masked by eapi aware PMs.

Either way for changes like this (or package.mask as a directory since 
I'm still annoyed by that) the repo needs to be marked in some way, a 
versioned format specifically, so that when stuff like this is added 
the PM can handle it gracefully instead of doing the wrong thing.

~brian