[gentoo-dev] git-r3: initial draft for review

[Michał Górny]

Hello, all.

After a few days of thinking, discovering and working, here it is.
The first working draft of new git eclass codenamed 'git-r3'.

First of all, the name is not final. I'm open to ideas. I'm open to
naming it 'git-r1' to put it in line with my other -r1 eclasses :).
I'd definitely like to avoid 'git-3' though, since that version-like
naming was a mistake as almost-'python-2' eclass shown.

Secondly, it's not even final that there will be a new eclass. Most
likely I will commit it as a new eclass since that way is easier for us
but if you prefer I may try to get it and git-2 more API-friendly
and work on making it a almost-drop-in replacement. Since, after all,
internals have actually changed much more than the API.


And now for the major changes:

1. The code has been split into clean 'fetch' and 'checkout' pieces.

That is, it is suited for distinct src_fetch() and src_unpack() phases
that we'll hopefully have in EAPI 6. What's important, the checkout
code does not rely on passing *any* environment variables from fetching
code. It is also made with concurrency in mind, so multiple ebuilds
using the same repository at the same time shouldn't be a problem.

2. Public fetch/checkout API.

git-2 has a lot of private functions and just src_unpack(). git-r3 has
git-r3_fetch() and git-r3_checkout() which are public API intended to
used in ebuilds that need more than plain fetch+unpack. While this
isn't exactly what multi-repo support pursuers wanted, it should make
supporting multiple repos in one ebuild much cleaner.

3. Clean submodule support with bare clones.

Since the submodules are very straightforward in design, I have decided
to move their support into the eclass directly. As a result, the new
eclass cleanly supports submodules, treating them as additional
repositories and doing submodule fetch/checkout recursively. There is
no need for non-bare clones anymore (and therefore their support has
been removed to make code simpler), and submodules work fine with
EVCS_OFFLINE=1.

4. 'Best-effort' shallow clones support.

I did my best to support shallow clones in the eclass. The code is
specifically designed to handle them whenever possible. However, since
shallow clones have a few limitations:

a) only branch/tag-based fetches support shallow clones. Fetching by
commit id forces complete clone (this is what submodules do BTW).

b) there's EGIT_NONSHALLOW option for users who prefer to have full
clones, and possibly for ebuilds that fail with shallow clones.

c) if shallow clones cause even more trouble than that, I will simply
remove their support from the eclass :).

[see notes about testing at the end]

5. Safer default EGIT_DIR choice. EGIT_PROJECT removed.

Since submodules are cloned as separate repositories as well, we can't
afford having EGIT_PROJECT to change the clone dir. Instead, the eclass
uses full path from first repo URI (with some preprocessing) to
determine the clone location. This should ensure non-colliding clones
with most likeliness that two ebuilds using the same repo will use
the same clone without any special effort from the maintainer.

6. Safer default checkout dir. EGIT_SOURCEDIR removed.

git-2 used to default EGIT_SOURCEDIR=${S}. This kinda sucked since if
one wanted to use subdirectory of the git repo, he needed to both set
EGIT_SOURCEDIR and S. Now, the checkout is done to ${WORKDIR}/${P}
by default and ebuilds can safely do S=${WORKDIR}/${P}/foo. I may
provide EGIT_SOURCEDIR if someone still finds it useful.


API/variables removed:

1. EGIT_SOURCEDIR:

a) if you need it for multiple repos, use the fetch/checkout functions
instead,

b) otherwise, play with S instead,

c) if you really need it, lemme know and I'll put it back.

2. EGIT_HAS_SUBMODULES -> no longer necessary, we autodetect them
(and we don't need that much special magic like we used to).

3. EGIT_OPTIONS -> interfered too much with eclass internals.

4. EGIT_MASTER -> people misused it all the time, and it caused issues
for projects that used different default branch. Now we just respect
upstream's default branch.

5. EGIT_PROJECT -> should be no longer necessary.

6. EGIT_DIR -> still exported, but no longer respects user setting it.

7. EGIT_REPACK, EGIT_PRUNE -> I will probably reintroduce it, or just
provide the ability to set git auto-cleanup options.

8. EGIT_NONBARE -> only bare clones are supported now.

9. EGIT_NOUNPACK -> git-2 is only eclass calling the default. Does
anyone actually need this? Is adding custom src_unpack() that hard?

10. EGIT_BOOTSTRAP -> this really belongs in *your* src_prepare().


I've tested the eclass on 113 live packages I'm using. Most of them
work just fine (I've replaced git-2 with the new eclass). Some
of them only require removing the old variables, some need having S
changed. However, I noticed the following issues as well:

1. code.google fails with 500 when trying to do a shallow clone
(probably they implemented their own dumb git server),

2. sys-apps/portage wants to play with 'git log' for ChangeLogs. That's
something that definitely is not going to work with shallow clones ;).
Not that I understand why someone would like a few megs of detailed git
backlog.

3. sys-fs/bedup's btrfs-progs submodule says the given commit id is
'not a valid branch point'. Need to investigate what this means.

4. 'git fetch --depth 1' seems to be refetching stuff even when nothing
changed. Need to investigate it. It may be enough to do an additional
'did anything change?' check.


I will try to look into those issues tomorrow. In the meantime, please
review this eclass and give me your thoughts. Especially if someone has
some more insight on shallow clones. Thanks.

And a fun fact: LLVM subversion checkout (in svn-src) has around ~2.4k
files which consume around 220M on btrfs. LLVM git shallow clone takes
17M.

-- 
Best regards,
Michał Górny