Re: [gentoo-dev] git-r3: initial draft for review

[Rick \\\"Zero_Chaos\\\" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/30/2013 07:37 PM, Michał Górny wrote:
> Hello, all.
>
> After a few days of thinking, discovering and working, here it is.
> The first working draft of new git eclass codenamed 'git-r3'.
>
> First of all, the name is not final. I'm open to ideas. I'm open to
> naming it 'git-r1' to put it in line with my other -r1 eclasses :).
> I'd definitely like to avoid 'git-3' though, since that version-like
> naming was a mistake as almost-'python-2' eclass shown.
>
Seems to be a lot of schizophrenia in versioning here.  I can honestly
tell you that git-2 is greater than git-r3, so I really caution against
this.

> Secondly, it's not even final that there will be a new eclass. Most
> likely I will commit it as a new eclass since that way is easier for us
> but if you prefer I may try to get it and git-2 more API-friendly
> and work on making it a almost-drop-in replacement. Since, after all,
> internals have actually changed much more than the API.
>
>
> And now for the major changes:
>
> 1. The code has been split into clean 'fetch' and 'checkout' pieces.
>
> That is, it is suited for distinct src_fetch() and src_unpack() phases
> that we'll hopefully have in EAPI 6. What's important, the checkout
> code does not rely on passing *any* environment variables from fetching
> code. It is also made with concurrency in mind, so multiple ebuilds
> using the same repository at the same time shouldn't be a problem.
>
> 2. Public fetch/checkout API.
>
> git-2 has a lot of private functions and just src_unpack(). git-r3 has
> git-r3_fetch() and git-r3_checkout() which are public API intended to
> used in ebuilds that need more than plain fetch+unpack. While this
> isn't exactly what multi-repo support pursuers wanted, it should make
> supporting multiple repos in one ebuild much cleaner.
>
> 3. Clean submodule support with bare clones.
>
> Since the submodules are very straightforward in design, I have decided
> to move their support into the eclass directly. As a result, the new
> eclass cleanly supports submodules, treating them as additional
> repositories and doing submodule fetch/checkout recursively. There is
> no need for non-bare clones anymore (and therefore their support has
> been removed to make code simpler), and submodules work fine with
> EVCS_OFFLINE=1.
>
> 4. 'Best-effort' shallow clones support.
>
> I did my best to support shallow clones in the eclass. The code is
> specifically designed to handle them whenever possible. However, since
> shallow clones have a few limitations:
>
> a) only branch/tag-based fetches support shallow clones. Fetching by
> commit id forces complete clone (this is what submodules do BTW).
>
> b) there's EGIT_NONSHALLOW option for users who prefer to have full
> clones, and possibly for ebuilds that fail with shallow clones.
>
> c) if shallow clones cause even more trouble than that, I will simply
> remove their support from the eclass :).
>
> [see notes about testing at the end]
>
> 5. Safer default EGIT_DIR choice. EGIT_PROJECT removed.
>
> Since submodules are cloned as separate repositories as well, we can't
> afford having EGIT_PROJECT to change the clone dir. Instead, the eclass
> uses full path from first repo URI (with some preprocessing) to
> determine the clone location. This should ensure non-colliding clones
> with most likeliness that two ebuilds using the same repo will use
> the same clone without any special effort from the maintainer.
>
> 6. Safer default checkout dir. EGIT_SOURCEDIR removed.
>
> git-2 used to default EGIT_SOURCEDIR=${S}. This kinda sucked since if
> one wanted to use subdirectory of the git repo, he needed to both set
> EGIT_SOURCEDIR and S. Now, the checkout is done to ${WORKDIR}/${P}
> by default and ebuilds can safely do S=${WORKDIR}/${P}/foo. I may
> provide EGIT_SOURCEDIR if someone still finds it useful.

Thank you so much. I've wanted to do this forever.
>
>
> API/variables removed:
>
> 1. EGIT_SOURCEDIR:
>
> a) if you need it for multiple repos, use the fetch/checkout functions
> instead,
>
> b) otherwise, play with S instead,
>
> c) if you really need it, lemme know and I'll put it back.
>
> 2. EGIT_HAS_SUBMODULES -> no longer necessary, we autodetect them
> (and we don't need that much special magic like we used to).
>
> 3. EGIT_OPTIONS -> interfered too much with eclass internals.
>
> 4. EGIT_MASTER -> people misused it all the time, and it caused issues
> for projects that used different default branch. Now we just respect
> upstream's default branch.
>
> 5. EGIT_PROJECT -> should be no longer necessary.
>
How so?

Thanks,
Zero

> 6. EGIT_DIR -> still exported, but no longer respects user setting it.
>
> 7. EGIT_REPACK, EGIT_PRUNE -> I will probably reintroduce it, or just
> provide the ability to set git auto-cleanup options.
>
> 8. EGIT_NONBARE -> only bare clones are supported now.
>
> 9. EGIT_NOUNPACK -> git-2 is only eclass calling the default. Does
> anyone actually need this? Is adding custom src_unpack() that hard?
>
> 10. EGIT_BOOTSTRAP -> this really belongs in *your* src_prepare().
>
>
> I've tested the eclass on 113 live packages I'm using. Most of them
> work just fine (I've replaced git-2 with the new eclass). Some
> of them only require removing the old variables, some need having S
> changed. However, I noticed the following issues as well:
>
> 1. code.google fails with 500 when trying to do a shallow clone
> (probably they implemented their own dumb git server),
>
> 2. sys-apps/portage wants to play with 'git log' for ChangeLogs. That's
> something that definitely is not going to work with shallow clones ;).
> Not that I understand why someone would like a few megs of detailed git
> backlog.
>
> 3. sys-fs/bedup's btrfs-progs submodule says the given commit id is
> 'not a valid branch point'. Need to investigate what this means.
>
> 4. 'git fetch --depth 1' seems to be refetching stuff even when nothing
> changed. Need to investigate it. It may be enough to do an additional
> 'did anything change?' check.
>
>
> I will try to look into those issues tomorrow. In the meantime, please
> review this eclass and give me your thoughts. Especially if someone has
> some more insight on shallow clones. Thanks.
>
> And a fun fact: LLVM subversion checkout (in svn-src) has around ~2.4k
> files which consume around 220M on btrfs. LLVM git shallow clone takes
> 17M.
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSIYNAAAoJEKXdFCfdEflKBdoP/Ru+/12Ypn6lg0h55tI14uiX
xzoev+DN1KK19oMiYZxZL4zkiCp9/YSvTSKEcVpokzYIxQyWU+stJ6uLQYFsxovh
lSP+9pOlMikS0xdjDudFjRcS0GMhZjGytrk/4IYmALxRnuKtAVM7XzEZe6vba4Vm
EJ4CSOCRoA4X0Kh49lSn/LhONUfm9PBrTQVkmOYSNZzPOue9WUP5ppIc7212w0x2
+67lq5hxR3LJItpDgX++sOErlDCF9O+eZe+j2JHklsHMYsKJPdScY3Xad9QBRXGa
rpymloDg5MMe7c079H59nCIRC7O5StXRZmiR6skdZDz/gMHMEZ15saQDYM2i3x/A
obbbjINl7MuW576YeHhfGzQhzYf4+41PuGSIyJaHQ7fuvL3l4jWbhYngcVP2WYBi
xudVKQei0eW7W2m5dGAeLD6IDC35PnxUvoWc69iRgDpn6OADkZ/bx4XD3Dx2Rqb9
6tkU4g9zIrlOG+ywlGzlBpDjKctC9AhBqh5SUJuM3WMGUfPUcTYfx+ENblfvTRAT
iz0gJJu62CumioMAx9aQGe3Dnr0f8m5COsPs79WmP+3g8Pm0LwswMx3agQEM5QWV
QFBh0189DNfwHR6GVt03EfxYUZx8Mm7uz+QyNfMJY/2fFPmJnqN4VctegWUn/FYw
8h0JTHjBC2dIAZwoQCvN
=1ESQ
-----END PGP SIGNATURE-----