Re: [gentoo-dev] [RFC] DIGESTS metadata variable for cache validation

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian Harring wrote:
> On Wed, Feb 11, 2009 at 02:01:24AM -0800, Zac Medico wrote:
>> Brian Harring wrote:
>>> On Tue, Feb 10, 2009 at 12:55:51PM -0800, Zac Medico wrote:
>>>> Brian Harring wrote:
>>>>> Frankly, forget compatibility- the current format could stand to die.
>>>>> The repository format is an ever growing mess- leave it as is and
>>>>> work on cutting over to something sane.
>>>> Changing the repository layout is a pretty radical thing to do.
>>>> You're welcome to start a new subject for that if you'd like but I'd
>>>> prefer to keep the scope of this thread focussed on the cache format
>>>> for the existing repository layout.
>> I don't intend to repeal the cache mtime requirement, at least
>> (especially) not on gentoo's rsync tree. However, I wouldn't say
>> that it's something that necessarily needs to be a requirement for
>> other repositories or overlays, moving forward (assuming that an
>> alternative validation framework is in place).
>
> So... you want a subset of repositories to have cache algo x, while
> the rest have the old algo.  And since the repo w/ algo x isn't
> marked in some fashion, all managers will have to use new algo x for
> compatibility reasons.  Right...

Clients using either validation mechanism can consume the same
cache. If the client recognizes DIGESTS data and it's available in a
given cache entry, naturally the client should prefer the DIGESTS
validation mechanism because it's more reliable.

>>> I reiterate, this belongs in a seperate repository format, along w/
>>> the rest of the unversioned repository changes you've been pushing in
>>> (profile package.mask breaking all non portage PMs is a perfect
>>> example).
>> The package.mask thing is a separate discussion. Let's do that in a
>> separate thread.
>
> Package.mask is relevant purely as a demonstration of why unversioned
> changes to the repository formats *needs* to stop.  Generally speaking
> it's pretty shitty behaviour to embrace/extend a format when others
> rely on it for interop.

I agree that it's a poor practice to change the format in ways that
are not inter-operable. However, as said above, introduction of the
DIGESTS data is inter-operable.

> The annoying thing about this thread is that *no where* am I saying
> you shouldn't be free to experiment.  All I'm stating is that the end
> result isn't a compatible repo- it *is* a new format (version even)
> thus mark it in some way so that the rest of us can start properly
> handling it rather then having to cut last minute releases since we're
> PMS compliant but portage treats PMS as a subset of it's format rules.

As said, the end result of introducing the DIGESTS data _is_ a
compatible repo.

> Pretty simple request, and not something that shouuld require argument
> as far as I'm concerned.

>>> The daft thing about this is that w/ effectively atomic sync (if the
>>> sync fails then mark the repo as screwed up till a sync completes),
>>> the current cache format can *still* do validation- no clue if
>>> paludis has it, but at least pkgcore and portage can handle this via
>>> awareness of the eclass stacking.
>> I want to have a more fault-tolerant solution than that.
>
> I understand your reasoning, and frankly I used to view the rsync
> issue in the same way- it's a naive view however since it implicitly
> is assuming that the resultant repo is *usable*, iow that the actual
> ebuild/eclass/profile data is valid, just that the updating bailed
> during metadata transfer.  There is zero gurantee as to where the
> rsync bailed- meaning you can be missing patches, have trashed
> manifests, etc.
>
> Well aware it's not friendly to require people to force a completed
> sync before being able to use the repo, but it really is the only
> *safe* option- as such the fault tolerant counterarg is a non
> arguement.

Problems aren't only triggered by sync issues. For example, suppose
that the user has locally modified an eclass in a way that results
in a metadata change. The DIGESTS data will provide enough
information to detect cases such as this. Without this data, the
user may be left scratching their head, wondering why their eclass
change hasn't been accounted for.

>>> Note that proper PM implementations *still* have to set the cache
>>> entries mtime for backwards compatibility w/ older PMs that don't
>>> support this new unversioned change thus muddying the implementation
>>> even further.
>> As said above, I wasn't intending that, at least (especially) not
>> for gentoo's rsync tree. I guess you got that idea from the mention
>> of bug 139134, but you don't need to worry about it.
>
> Implicitly it's required; if pkgcore is to generate cache entries for
> repo x, it has to do exactly as I said so that any any pre
> cache-modified-managers are still able to use the cache.  That's
> assuming the $PM cares about compatibility...

As said, clients using either validation mechanism can consume the
same cache, so introduction of the DIGESTS data will be fully
inter-operable.
- --
Thanks,
Zac

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkmXJo4ACgkQ/ejvha5XGaNG7wCgwnOtEKD8VuKLAjFyahAJpQIJ
HWAAn2woN2CxmvAXu5ir0/N7ZvJLrAbc
=NVzc
-----END PGP SIGNATURE-----