Re: [gentoo-soc] GSoC - cache sync/self-contained ebuilds

[Zac Medico]

On 03/27/2011 07:28 AM, Michael Seifert wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Am 24.03.2011 17:58, schrieb Zac Medico:
>> Well, it would be inefficient to open separate TCP connections for
>> individual metadata files since there are so many of them and they are
>> so small. This is why package managers typically download the metadata
>> for all packages as a single bundle. For example, see the type of
>> metadata bundle that is used to implement PORTAGE_BINHOST support:
>>
>>   http://tinderbox.dev.gentoo.org/default-linux/x86/Packages
>>
>
> Is there a specific reason why the PORTAGE_BINHOST metadata is different
> from the metadata/cache format?

They have minor differences because they were designed for slightly
different use cases:

The metadata/cache format was designed to be distributed together with
the ebuilds that it was generated from. Its main drawback is that it can
be slow to read many small files since it may require lots of disk
seeks. We could pack them all into a single file, similar to one that
PORTAGE_BINHOST uses. That would help for tools like eix since it's
faster to read one big file than many small files. However, if it was
fetched earlier and separate from the ebuilds, it wouldn't be very
practical for dependency calculations unless you provided a way to fetch
exactly the same revisions of ebuilds (and inherited eclasses which can
modify dependencies) that the earlier fetched metadata corresponds to.
For example, the cache could be made to refer to a UUID would be used to
generate a URI in order to fetch a particular revision of ebuild/eclass
bundle that exactly corresponds to the cache entry.

The PORTAGE_BINHOST cache format is better than the metadata/cache
format for the use case that it's designed for, however the current
design has a race condition which has been experienced by chromium-os
developers:

  http://code.google.com/p/chromium-os/issues/detail?id=3225

> I like the BINHOST metadata better, even if it is split up into several
> files, because it would already contain the ebuild version it was
> generated for. Probably it would be a good idea to merge the information
> of both metadata into a single unified format?
> This would also solve the problem with the missing version control
> (described below) as well as simplifying the way portage handles
> metadata. On the other hand it would be an even more substantial change,
> which is not necessarily a bad thing. Portage is supposed to work the
> same way as before – just faster.

Well, a new cache format is only part of the solution. In order to
provide revision control that's necessary for practical dependency
calculations when the cache is fetched earlier that the
ebuilds/eclasses, you're also going to need to create individually
fetchable revisioned ebuild/eclass bundles that the cache will refer to
(without any race conditions).

>> It's conceivable that you could simply use rsync to sync the
>> metadata/cache/ subdirectory from
>> rsync://rsync.gentoo.org/gentoo-portage/. However, since the rsync tree
>> constantly mutates and doesn't provide any kind version control, it
>> would not be very practical to use it in this way. If you fetch the
>> metadata and the ebuilds separately, you need a way to guarantee that
>> you can fetch exactly the same revisions of ebuilds that the earlier
>> fetched metadata corresponds to.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk2PSYAACgkQnzX+Jf4GTUyz7ACcCV44bXSEwoyCg/6uMz8E9/2g
> c+EAn1m/BpF7rKkSSmpouousupVCbUHL
> =G4GS
> -----END PGP SIGNATURE-----
>


-- 
Thanks,
Zac