Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 10:56 AM, Michał Górny <mgorny@g.o> wrote:
> Dnia 2014-09-14, o godz. 10:33:03
>
> With git, we can finally do stuff like preparing everything and pushing
> in one go. Rebasing or merging will be much easier then, since
> the effective push rate will be smaller than current commit rate.

While I agree that the ability to consolidate commits will definitely
help with the commit rate, I'm not sure it will make a big difference.
It will turn a kde stablereq from 300 commits into 1, and do the same
for things like package moves and such.  However, I suspect that the
vast majority of our commits are things like bumps on individual
packages that will always be individual commits.  Maybe insofar as one
person does a bunch of them they can be pushed at the same time,
but...

Looking at https://github.com/rich0/gentoo-gitmig-2014-02-21 it seems
like we get about 150 commits/day on busy days.  I suspect that isn't
evenly distributed, but you may be right that it will just work out.

>>
>> Actually doing the conversion is basically a solved problem.  If this
>> were actually the blocker I'd be all for just sticking the history in
>> a different repo and starting from scratch with a new one.
>
> Was the resulting tree actually verified? How long does the conversion
> take? Can it be incremental, i.e. convert most of it, lock CVS, convert
> the remaining new commits?

The tree has been verified.  The verification approaches so far are
neither 100% thorough nor realtime in operation.  However, I think we
have a working migration process and I don't really see the need to do
a double-check at the time of the actual migration.

ferringb was able to do conversions in about 20min with a decent SSD
and a 32-core system.  His migration scripts can migrate categories in
parallel.  I haven't personally tried to run them myself, but I
believe robbat2 and patrick have experimented with them.  If there is
revived interested I can see if I can set them up to run in a chroot
with some documentation so that anybody can run it and satisfy
themselves that it works, assuming somebody else doesn't have such a
chroot ready to go.  If finding a host to run it on is a problem I'm
sure we could get the Trustees to spring for some time on EC2 or
whatever.  There is no reason that this couldn't be as simple as
extracting a tarball, bind-mounting a cvs repo inside, and firing off
the scripts.

I do not believe it can be made to be incremental.  But, the runtime
should be in keeping with your hour-or-two of downtime suggestion.  I
suspect a fair bit of the downtime will taken just to transfer the
copy of the cvroot to the migration server, and transfer the resulting
git tree to wherever it needs to go and get all the back-end scripts
running/etc.

>
> Are you willing to champion that, then? :)
>

Well, I'm in for what it matters.  I don't have root on any infra
boxes if that is what you're looking for.  :)

--
Rich