Re: [gentoo-scm] cvs/irker/git thread from -dev

[Rich Freeman]

On Mon, Oct 1, 2012 at 9:46 PM, Michael Mol <mikemol@×××××.com> wrote:
> This was on -dev a while ago. Was there nothing additional?

Not yet - there was a bunch of good conversation on irc over the last
few hours.

Note that this list is archived on gmane, and it is probably worth
reviewing for past discussions.

>
> Diego had a fair point about having to checkout a crapton of git
> history. I think that could be significantly improved with a seed
> tarball on a torrent, particularly if the seed tarball is updated
> annually (or biannually).

Initial distribution will be via a tarball of some kind (not sure if
it is just a tarball of the repository, or in an importable git
format) to save on load.  It would be available with or without
history.  I'm not actually certain that you can't commit without the
full history.  I would think that as long as git could fast-forward
from whatever gentoo considers the head to whatever you consider the
head you'd be fine.

From IRC the big issues at the moment are that we know the current
migration drops some old content from the history.  It is also hard to
validate the content.

Validation is tricky because CVS tends to store per-file history, and
git tends to store a history of the head which incorporates all the
files within.  So, traversing each in a logical order makes it really
hard to compare them.  However, this really seems like a good problem
for something like MapReduce.

If you traverse git for each commit you should be able to output:
path/file hash date/time committer message

Then you'd sort that by the first three fields, and drop all lines
where the first two fields don't change.  That then gives you a sorted
commit history per file.  Then you could traverse cvs and get the same
(the output of cvs log per-file should do that).  Then you compare the
two lists.

You could do the dumps for all the git commits in parallel, and
sorting is parallel as well (basically automatic with mapreduce I
think).  For CVS you should be able to do each file in parallel, and
again the sorting is parallel.

Note I have zero experience with MapReduce.  However, it seems like a
really fun project.  I need to learn more about the "logical" model
behind CVS.  I understand it fairly well for git and that is a thing
of beauty.  You don't even need to calculate the hashes for git since
they're already stored in the tree.

Rich