Re: [gentoo-dev] [RFC] DIGESTS metadata variable for cache validation

[Brian Harring]

On Wed, Feb 11, 2009 at 02:01:24AM -0800, Zac Medico wrote:
> Brian Harring wrote:
> > On Tue, Feb 10, 2009 at 12:55:51PM -0800, Zac Medico wrote:
> >> Brian Harring wrote:
> >>> Frankly, forget compatibility- the current format could stand to die.
> >>> The repository format is an ever growing mess- leave it as is and
> >>> work on cutting over to something sane.
> >> Changing the repository layout is a pretty radical thing to do.
> >> You're welcome to start a new subject for that if you'd like but I'd
> >> prefer to keep the scope of this thread focussed on the cache format
> >> for the existing repository layout.
>
> I don't intend to repeal the cache mtime requirement, at least
> (especially) not on gentoo's rsync tree. However, I wouldn't say
> that it's something that necessarily needs to be a requirement for
> other repositories or overlays, moving forward (assuming that an
> alternative validation framework is in place).

So... you want a subset of repositories to have cache algo x, while 
the rest have the old algo.  And since the repo w/ algo x isn't 
marked in some fashion, all managers will have to use new algo x for 
compatibility reasons.  Right...


> > I reiterate, this belongs in a seperate repository format, along w/
> > the rest of the unversioned repository changes you've been pushing in
> > (profile package.mask breaking all non portage PMs is a perfect
> > example).
>
> The package.mask thing is a separate discussion. Let's do that in a
> separate thread.

Package.mask is relevant purely as a demonstration of why unversioned 
changes to the repository formats *needs* to stop.  Generally speaking 
it's pretty shitty behaviour to embrace/extend a format when others 
rely on it for interop.

The annoying thing about this thread is that *no where* am I saying 
you shouldn't be free to experiment.  All I'm stating is that the end 
result isn't a compatible repo- it *is* a new format (version even) 
thus mark it in some way so that the rest of us can start properly 
handling it rather then having to cut last minute releases since we're 
PMS compliant but portage treats PMS as a subset of it's format rules.

Pretty simple request, and not something that shouuld require argument 
as far as I'm concerned.


> > The daft thing about this is that w/ effectively atomic sync (if the
> > sync fails then mark the repo as screwed up till a sync completes),
> > the current cache format can *still* do validation- no clue if
> > paludis has it, but at least pkgcore and portage can handle this via
> > awareness of the eclass stacking.
>
> I want to have a more fault-tolerant solution than that.

I understand your reasoning, and frankly I used to view the rsync 
issue in the same way- it's a naive view however since it implicitly 
is assuming that the resultant repo is *usable*, iow that the actual 
ebuild/eclass/profile data is valid, just that the updating bailed 
during metadata transfer.  There is zero gurantee as to where the 
rsync bailed- meaning you can be missing patches, have trashed 
manifests, etc.

Well aware it's not friendly to require people to force a completed 
sync before being able to use the repo, but it really is the only 
*safe* option- as such the fault tolerant counterarg is a non 
arguement.


> > Note that proper PM implementations *still* have to set the cache
> > entries mtime for backwards compatibility w/ older PMs that don't
> > support this new unversioned change thus muddying the implementation
> > even further.
>
> As said above, I wasn't intending that, at least (especially) not
> for gentoo's rsync tree. I guess you got that idea from the mention
> of bug 139134, but you don't need to worry about it.

Implicitly it's required; if pkgcore is to generate cache entries for 
repo x, it has to do exactly as I said so that any any pre 
cache-modified-managers are still able to use the cache.  That's 
assuming the $PM cares about compatibility...

~harring