[gentoo-scm] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 8:03 AM, Michał Górny <mgorny@g.o> wrote:
>
> I'm quite tired of promises and all that perfectionist non-sense which
> locks us up with CVS for next 10 years of bikeshed.

While I tend to agree with the sentiment, I don't think you're
actually targeting the problems that aren't already solved here.

> Of course, that assumes infra is
> going to cooperate quickly or someone else is willing to provide the
> infra for it.

The infra components to a git infrastructure are one of the main
blockers at this point.  I don't really see cooperation as the issue -
just lack of manpower or interest.

>
> I can provide some testing repos once someone is willing to provide
> the hardware.

We already have plenty of testing repos (well, minus all the back-end stuff).

>
> 1. send announcement to devs to explain how to use git,

This is one of the blockers.  We haven't actually decided how we want
to use git.

Sure, everybody knows how to use git.  The problem is that there are a
dozen different ways we COULD use git, and nobody has picked the ONE
way we WILL use it.

This isn't as trivial as you might think.  We have a fairly high
commit rate and with a single repository that means that in-between a
pull-merge/rebase-push there is a decent chance of another commit that
will make the resulting push a non-fast-forward.

People love to point out linux and its insane commit rate.  The thing
is, the mainline git repo with all those commits has exactly one
committer - Linus himself.  They don't have one big repo with one
master branch that everybody pushes to.  At least, that is my
understanding (and there are certainly others here who are more
involved with kernel development).

>
> 2. lock CVS out to read-only,
>
> 3. create all the git repos, get hooks rolling,
>
> 4. enable R/W access to the repos.
>
> With some luck, no more than 2 hours downtime.

I agree that the actual conversion should be able to done quickly.

> On top of user sync repo rsync is propagated. The rsync tree is populated
> with all old ChangeLogs copied from CVS (stored in 30M git repo), new
> ChangeLogs are generated from git logs and Manifests are expanded.

So, I don't really have a problem with your design.  I still question
whether we still need to be generating changelogs - they seem
incredibly redundant.  But, if people really want a redundant copy of
the git log, whatever...

> Main developer repo
> -------------------
>
> I was able to create a start git repository that takes around 66M
> as a git pack (this is how much you will have to fetch to start working
> with it). The repository is stripped clean of history and ChangeLogs,
> and has thin Manifests only.
>
> This means we don't have to wait till someone figures out the perfect
> way of converting the old CVS repository. You don't need that history
> most of the time, and you can play with CVS to get it if you really do.
> In any case, we would likely strip the history anyway to get a small
> repo to work with.

We already have a migration process that coverts the old CVS
repository, generating both a shallow repository that lacks history
and a full repository that contains all of history. Additionally,
these two are consistent - that is the last branch of the full
repository has the same commit ID as the base of the shallow
repository.  Basically we generate the full history and then trim out
99% of it so that the commit in the shallow repository points to a
parent that isn't in the packed repository.

Actually doing the conversion is basically a solved problem.  If this
were actually the blocker I'd be all for just sticking the history in
a different repo and starting from scratch with a new one.

>
> I think we should also merge gentoo-news & glsa & herds.xml into
> the repository. They all reference Gentoo packages at a particular
> state in time, and it would be much nicer to have them synced properly.
>

I can see the pros/cons here, but I don't personally have an issue
with merging them.  As has been brought up elsewhere herds.xml may
just go away.

If somebody can come up with a set of hooks/scripts that will create
the various trees and the only thing that is left is to get infra to
host them, I think we can make real progress.  I don't think this is
something that needs to take a long time.  The pieces are mostly there
- they just have to be assembled.

--
Rich