Re: [gentoo-dev] [RFC] DIGESTS metadata variable for cache validation

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian Harring wrote:
> On Tue, Feb 10, 2009 at 12:55:51PM -0800, Zac Medico wrote:
>> Brian Harring wrote:
>>> Frankly, forget compatibility- the current format could stand to die.
>>> The repository format is an ever growing mess- leave it as is and
>>> work on cutting over to something sane.
>> Changing the repository layout is a pretty radical thing to do.
>> You're welcome to start a new subject for that if you'd like but I'd
>> prefer to keep the scope of this thread focussed on the cache format
>> for the existing repository layout.

I don't intend to repeal the cache mtime requirement, at least
(especially) not on gentoo's rsync tree. However, I wouldn't say
that it's something that necessarily needs to be a requirement for
other repositories or overlays, moving forward (assuming that an
alternative validation framework is in place).

> Vacuous arguement via focusing on the 'layout' part rather then the
> repository whole I implied; you're stating that one should not
> discuss changing the repository standard/spec while arguing that
> repealing the requirement that cache mtime entries match ebuild
> mtime (part of the repository spec) should be the point of discussion.
>
> The daft thing about this is that w/ effectively atomic sync (if the
> sync fails then mark the repo as screwed up till a sync completes),
> the current cache format can *still* do validation- no clue if
> paludis has it, but at least pkgcore and portage can handle this via
> awareness of the eclass stacking.

I want to have a more fault-tolerant solution than that.

> So for git vcses bundling metadata (a bad idea anyways to be storing
> generated content in the mainline vcs), your proposal allows them to
> use a cache.  For every other distribution mechanism that works fine,
> they wind up paying the cost for that corner case.  The 80 pays for
> the 20 isn't the normal form of the 80/20 rule ;)

What I'm concerned about is costs in terms of support and usability.
When something goes wrong and there's not enough data to detect it,
it triggers problems that confuse and annoy users. I want to have
the DIGESTS data available so that these sorts of problems are easy
to detect and handle appropriately. I think you're being too stingy
about disk space.

> Note that proper PM implementations *still* have to set the cache
> entries mtime for backwards compatibility w/ older PMs that don't
> support this new unversioned change thus muddying the implementation
> even further.

As said above, I wasn't intending that, at least (especially) not
for gentoo's rsync tree. I guess you got that idea from the mention
of bug 139134, but you don't need to worry about it.

> I reiterate, this belongs in a seperate repository format, along w/
> the rest of the unversioned repository changes you've been pushing in
> (profile package.mask breaking all non portage PMs is a perfect
> example).

The package.mask thing is a separate discussion. Let's do that in a
separate thread.

>>> Overlay maintainers who want the latest/greatest obviously can convert
>>> over also; one would hope their would be enough cleanup to make it
>>> worth their time.
>>>
>>> As for the nasty gentoo-x86 compatibility, basically, do the
>>> following:
>>>
>>> 1) maintain the existing cvs repo as is
>>> 2) iron out what cleanup/restructuring is desired.  glep55 being
>>> jammed in here is a potential for example.  Nail down the new repo
>>> format basically (with an eye for translating the cvs repo to it on
>>> the fly).
>>> 3) use an eclass index holding the checksums, w/ the cache entries
>>> referencing the index numbers rather (sorting the index by
>>> consumption, meaning the more ebuilds using it the lower the index):
>>> this brings the cache addition down to around 285KB (acceptable imo)
>>> while giving full flexibility in the checksums available for eclasses.
>>> This is assuming the current flat_list format is still in use in the
>>> new repo...
>> As previously discussed [2], having shared integrity data (as you
>> suggest) has implications in terms of reduced simplicity and robustness.
>
> The complexity arguement is a white elephant.   Rsync is the sole
> transport that has atomicity issues; the rest don't (when you check
> out from vcs, you get an exact rev effectively).  Rsync generation
> ought to be preparing the new snapshot then swapping it in, and if I
> recall correctly that's exactly what osprey does now (or whatever node
> y'all are using for generating gentoo-x86 these days).
>
> The point there is that there are specific steps taken preparing the
> repo- those steps already ensure the snapshot/rev is complete prior to
> being available so there isn't real potential of catching it mid
> update.  Via that existing machinery, a shared index is *no issue*-
> the one spot it rears it's head is during a failed/partial sync (the
> repo should not be using in such a state since stale cache is the
> least concern at that point).

It's too fragile and the potential usability and support issues are
a liability. As said, I want a more fault-tolerant solution than that.

> From where I'm sitting, the changes you've either slipped in, or want
> to slip in to the repository format are completely ignoring the past
> bad history of breakage doing such things, and ignoring why EAPI
> exists these days.

> The reason a new format is realistically needed here is that existing
> PMS compliant implementations will wind up accessing the repo and
> behaving as if everything is fine and dandy without knowing the rules
> used to read no longer match the rules used to generate the repo.
> This is a no go, same reason EAPI awareness had to sit for a long ass
> time to ensure EAPI=1 would be properly masked by eapi aware PMs.
>
> Either way for changes like this (or package.mask as a directory since
> I'm still annoyed by that) the repo needs to be marked in some way, a
> versioned format specifically, so that when stuff like this is added
> the PM can handle it gracefully instead of doing the wrong thing.

Again, that's a separate discussion for a different thread.

- --
Thanks,
Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkmSoe4ACgkQ/ejvha5XGaMf4wCg8ldvDu/w/FDarHXCVv/HOkvy
qQUAn1WGYrqZVdM4nCUjE8bZglGWD9yU
=XN/T
-----END PGP SIGNATURE-----