Re: [gentoo-portage-dev] Google SoC and "cache sync"

[Emma Strubell]

Zac Medico wrote:
> The way that I imagine the "cache sync" idea should be implemented
> is like paludis's "unavailable repository" which uses of tarball to
> distribute package metadata[1]. The tarball approach that they use
> seems pretty reasonable. However, it would probably also be nice to
> be able to use a protocol such as rsync to download the
> metadata/cache/ directory from the same URI which is used to fetch
> the ebuilds themselves (maybe paludis supports this already, I don't
> know).

You're offering two different ideas here, right? The "unavailable
repository" method, and the method using the metadata/cache/
directory?

If so, it makes sense to me to take the metadata/cache/ directory
route, since, as you said, multiple repositories aren't yet supported
in portage. At first I was thinking I could contact the guy who might
be working on multiple repository support this summer and work with
him to some extent... but the "unavaliable repository" solution would
basically be dependent on/building off of multiple repository support,
and it seems like building off of something that isn't fully built
would be a bad idea.

And to clarify: the goal of the project is to modify portage so that
instead of fetching all of the ebuilds in the portage tree (or in an
overlay) upon a sync, portage only fetches the metadata and cache info
(via the metadata/cache/ directory) of the tree, and the ebuilds of
packages that are already installed (packages found in the world
file?) And then, additional ebuilds would be fetched only when they
are needed? Or will only metadata/cache/ be fetched upon sync, and
then all ebuilds will be fetched only when they are needed? Am I
completely oversimplifying the project?

Thanks so much for your help,
Emma


On Tue, Mar 31, 2009 at 6:41 PM, Zac Medico <zmedico@g.o> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Emma Strubell wrote:
> > Hi all.
> >
> > So, I'd love to do Google's Summer of Code with you guys. I was perusing
> > the list of ideas on the Gentoo wiki, and the "cache sync" idea seems
> > pretty interesting, especially since it concerns the overall speed of
> > portage, including search, which of course I've already started some
> > work on. However, there is no contact person associated with that
> > project! I figured I'd come here before going to #gentoo-soc to see if
> > anyone is interested in mentoring me on this project, since it seemed
> > like a few of you might be interested.
>
> The way that I imagine the "cache sync" idea should be implemented
> is like paludis's "unavailable repository" which uses of tarball to
> distribute package metadata[1]. The tarball approach that they use
> seems pretty reasonable. However, it would probably also be nice to
> be able to use a protocol such as rsync to download the
> metadata/cache/ directory from the same URI which is used to fetch
> the ebuilds themselves (maybe paludis supports this already, I don't
> know).
>
> In order for the clients to be able to download the metadata/cache/
> directory, first that directory has to be populated (as is done on
> gentoo's master rsync server). I'm currently working on a tool
> called 'egencache' that overlay maintainers will be able to use in
> order to populate the metadata/cache/ directory [2]. It will be
> included in the next portage release.
>
> Before we implement something like "unavailable repository" for
> portage, first we'll have to add multiple repository support, and
> that's a decent sized project of it's own. Somebody has mentioned
> interest in "multiple repository support" on the gentoo-soc list
> [3], but they haven't submitted a proposal to
> http://socghop.appspot.com yet.
>
> [1] http://paludis.pioto.org/configuration/repositories/unavailable.html
> [2] http://bugs.gentoo.org/show_bug.cgi?id=261377
> [3]
> http://archives.gentoo.org/gentoo-soc/msg_e383863a6748e367e13fe53b092f3908.xml
> - --
> Thanks,
> Zac
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.11 (GNU/Linux)
>
> iEYEARECAAYFAknSnA4ACgkQ/ejvha5XGaO6tACgjzAsoXP0cJd0Vr1vJxU2CvLQ
> JtwAn2Sj+GxLyyRpOIdbejPirCljmF2c
> =k5u1
> -----END PGP SIGNATURE-----
>