Re: [gentoo-scm] Git Validation - Update

[Michael Haggerty]

First I'd like to make two comments about your general approach:

1. It seems like you are doing an awful lot of work and using a lot of
movable parts to reduce the switchover time.  Just how much traffic does
your repo get?  Is it really worth the effort to reduce the downtime by
a couple of hours?

2. It seem to me to be overkill to insist on a repository validation
between the conversion and switching the repository live.  I would suggest

    do {
        Adjust conversion scripts
        Copy CVS repo
        Do test conversion
        Validate converted repository
    } until (validation perfect);

    Switch CVS repo to read-only
    Do final conversion
    Switch git repo live
    Validate converted repository at leisure for your peace of mind

Because realistically, you are not going to get a perfect conversion
during your test followed by a corrupt one a few days later when there
are only a few more commits in the CVS repository.  The point of testing
is to perfect the migration scripts and establish confidence in the
tools, at which point it would be extraordinary for the final conversion
to have invisible problems that would be detected by the validation but
which weren't there during the test conversions.

But, if you really want to optimize the testing...

On 10/16/2012 07:59 PM, Rich Freeman wrote:
> [...]
> Next step is to get cvs into a similar format.  My initial thoughts:
>
> 1.  Just run cvs log in the root, chop it up at file boundaries,
> base64 encode each blob, and dump that into a text file one file per
> line.
> 2.  Distribute processing of each file, turning it into one line per
> commit with all the info my git program dumps, save the file hash.
> 3.  Distribute reading those lines, checking out that one version of
> one file, calculating the hash, and outputting the full info.
>
> Steps 1/2 might be cheaper to just combine since you have to scan the
> whole thing to chop it up and the parsing can't be THAT expensive.
>
> If there are libs to make any of this easier I'm all ears, but it
> seems like there isn't much out there - nothing like pygit2.

Obviously there's a lot of code in cvs2git for parsing CVS files and
extracting metadata and/or the revision fulltext from them.  It's pretty
straightforward really; the only slightly subtle thing is the expansion
of RCS keywords (like $Revision$ etc).  The code in cvs2git has the
advantages that it is pure Python and that it can generate all of the
CVS revision fulltexts via one parse of the RCS file, so it is vastly
faster than running CVS once per revision.

Depending on your level of paranoia, you might not trust cvs2git to
extract information from RCS files that you want to use to validate
cvs2git.  OTOH the parsing of the CVS files and the recreation of the
fulltext is not the complicated part of cvs2git.  The complicated part
is inferring the original project-wide commits from the individual file
histories recorded by CVS.  But given that your project didn't do any
branching, even that should be pretty straightforward.

> Once I have both I can start working on validation rules and perhaps
> get feedback to the conversion team.  We'll need to work out what does
> and doesn't count as OK.  We're doing transformation of data during
> migration, so I need to take that into account.  Either the logic goes
> into the compare function, or the logic goes into the dump side so
> that the compares work out the same.  Timestamps might force us to do
> logic during compare anyway.

What kind of data transformations are you doing during the migration?

Michael

-- 
Michael Haggerty
mhagger@××××××××.edu
http://softwareswirl.blogspot.com/