Re: [gentoo-soc] GSoC - cache sync/self-contained ebuilds

[Zac Medico]

On 03/29/2011 07:31 AM, Michael Seifert wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Am 27.03.2011 21:39, schrieb Zac Medico:
>> On 03/27/2011 07:28 AM, Michael Seifert wrote:
>> I like the BINHOST metadata better, even if it is split up into several
>> files, because it would already contain the ebuild version it was
>> generated for. Probably it would be a good idea to merge the information
>> of both metadata into a single unified format?
>> This would also solve the problem with the missing version control
>> (described below) as well as simplifying the way portage handles
>> metadata. On the other hand it would be an even more substantial change,
>> which is not necessarily a bad thing. Portage is supposed to work the
>> same way as before  just faster.
>>
>>> Well, a new cache format is only part of the solution. In order to
>>> provide revision control that's necessary for practical dependency
>>> calculations when the cache is fetched earlier that the
>>> ebuilds/eclasses, you're also going to need to create individually
>>> fetchable revisioned ebuild/eclass bundles that the cache will refer to
>>> (without any race conditions).
>
> If I understood your point correctly, there can be problems, if there
> are changes to eclasses after a user ran emerge --sync and before the
> ebuilds are fetched (race condition between server and client). In such
> a case, the correct ebuild would be installed using a wrong environment
> (i.e. the wrong eclasses), because the metadata is outdated and contains
> wrong dependencies. Did this cover your outlined case?

That's mostly correct, however, it's not just the eclasses that
introduce the race condition. Just like the eclasses, the ebuilds
themselves can be modified. In addition, so the can the files that may
be included with the ebuilds in the "files" subdirectory ($FILESDIR).

In order to be exhaustively complete, you'd also have to include any
licenses referenced by LICENSE variable if the ebuild (these are located
in the licenses/ subdirectory of the repository).

You'd probably also want to include any matching package.mask entries
from the global profiles/package.mask file, since this is can be very
relevant to dependency calculations.

Finally, there's the user's architecture-specific profile. This can also
affect dependency calculations via things like package.mask,
package.unmask, use.mask, and use.force. If you want to be entirely
exhaustive, then you'll need your ebuild metadata to reference a
snapshot of this profile.

> I can think of three solutions for this:
> 1. Self-contained ebuilds
> Your proposal of packaging the ebuild together with its eclasses and the
> source code would make the need of eclass versioning unnecessary, since
> the ebuild is always packaged within the correct environment.
> (Creates a new race condition, see your link :)

Really, the eclass versioning would not be unnecessary, but it would be
tied directly to the ebuild versioning. If we're going to be completely
exhaustive, then it might make sense to have the ebuild metadata
reference separate eclass, license, package.mask, and profile bundles.

> 2. Using the VCS
> Store the versions of eclasses in the metadata and fetch it from a
> repository before installing the ebuilds. This would pull a dependency
> on a VCS (CVS in this case), though.

Not necessarily. If the ebuild metadata contains UUIDs that the client
can translate to URIs using a predefined protocol, then the client
simply needs to be aware of the protocol so that it can fetch the
appropriate URIs.

> 3. Fetching the eclasses together with the metadata
> This way you would have a local snapshot of the build environment, which
> would eliminate the race condition and the need to identify eclass
> versions from the metadata, because it stays consitent.

Right. But if you decide to split out license, package.mask, and profile
bundles as discussed above, you might also decide to do that for
eclasses as well.
-- 
Thanks,
Zac