Re: [gentoo-dev] [RFC] gentoo-x86 migration to repo-per-package

[Fabian Groffen]

On 07-08-2011 00:07:41 +0530, Nirbheek Chauhan wrote:
> On Sat, Aug 6, 2011 at 7:43 PM, Fabian Groffen <grobian@g.o> wrote:
> > In short, the repo-per-package model means that each package
> > (my-cat/package) is a separate repository in some VCS.
> > Instead of having a huge tree that will only grow forever (gx86),
> > packages are just in their own repository.
>
> I had mixed feelings while reading your email. The idea is certainly
> very intriguing, but there's a few things that make it a no-go for me:
>
> 1. One of the big things I've been looking forward to with git is the
> ability to do atomic commits across the tree. Addition of GNOME
> releases, pkgmove changes across the tree, changing ebuild/eclass
> behaviour, etc. without inconsistency or praying that my connection
> doesn't get dropped in the middle of a hundred interrelated commits.
> Without this feature, I think some arch teams and GNOME/KDE teams will
> become sad.

I see this being possible by making a single commit to the rsync tree
generation script.

I also consider alternatives possible, as touched upon by James Cloos in
this thread where large projects like GNOME and KDE have a single
repository for all/most of their ebuilds, and perhaps even eclasses.
Repo-per-package may be too finegrained for projects like these, and
being flexible here is not going to be any problem AFAICT.

> 2. The ability to do "feature" commits across the whole tree instead
> of hundreds of tiny commits everywhere. This combined with the
> ChangeLog generation will save a lot of time and space. This will
> especially benefit arch teams, but I've felt the need for this
> numerous times myself. Example: we moved to using .xz tarballs for
> GNOME, and that touched a lot of ebuilds, and it was extremely
> time-consuming to repeat echangelog && repoman commit per-package.

Consensus is that echangelog is eventually going to disappear, IIRC, and
repoman commit probably can be done on the entire tree/repo, with the
help of sub-repos, or when you have a repo for full GNOME.

Whether you script a loop, or make a single call to repoman, you always
have to pay for running repoman, since it's your QA tool, that you're
not supposed to skip/bypass.

> 3. Adding packages from overlays via `cherry-pick` or `git am` will
> become extremely tedious. If thin manifests are implemented, a series
> of patches + a simple merge hook will be all you need to move
> KDE/GNOME releases from the overlay to the tree. Without a single
> tree, you need to go back to the current way of doing things.

With my proposal you wouldn't do this.  You would simply add a line in
the rsync tree script for including that package.  Most probably the
package would already live on g.g.o or something Gentooish, so it
wouldn't move at all, it would just be included.

In case you would have a repo with multiple packages, you would just
tell the script to now also include the directory where your package
lives.

> 4. We'll need to write extra tools to keep the user's cat/pkg list
> up-to-date; adding and removing repositories as needed, etc. This is
> added complexity for which we'll need volunteers (we've been facing a
> manpower shortage already...)

I don't understand this.  Users don't see anything of this change.
Developers could use subtrees, forests, or just only what they care
about.

> 5. The total size of the tree will increase a *lot* since all these
> repositories will no longer share data. The current gentoo-x86 tree
> stored in git without history takes only ~25MB because ebuilds are
> extremely redundant. The space requirements will balloon once we need
> to store 15,000 repositories. And arch teams will have to store *all*
> of them, often on devices with very low space.

I'm not too concerned about disk space.  Cloning a repo as-needed should
be fairly fast, and even arch teams won't need all 15,000 repositories.
It's easy to throw away repos for packages no longer necessary too.
For the limited disk-space arches, the specialised rsync trees do come
in handy though.


-- 
Fabian Groffen
Gentoo on a different level