Re: [gentoo-scm] CVS -> git, list of where non-infra folk can contribute

[Brian Harring]

On Thu, Oct 04, 2012 at 11:34:00PM +0200, Michael Haggerty wrote:
> On 10/04/2012 09:38 PM, Brian Harring wrote:
> > On Thu, Oct 04, 2012 at 05:45:40PM +0200, Michael Haggerty wrote:
> >> Another big boost for cvs2git is to use the ExternalBlobGenerator, which
> >> extracts revision contents to a "blob" file in CollectRevsPass using an
> >> external script, thereby making OutputPass much faster.
> >
> > This still sequential, or can it effectively run in parallel?  I'm
> > just wondering if the 'faster' is via parallelism, or bypassing a
> > slower native python implementation.
>
> It currently does the blob generation in one separate process in
> parallel with the activities that are happening in the main cvs2svn
> process.  (I remembered incorrectly; the blobs are collected during
> FilterSymbolsPass, not CollectRevsPass.)
>
> I think it would be relatively straightforward to generalize this to N
> blob-generation processes, but I think that blob generation had already
> ceased to be the bottleneck with only 1 process.  However, your
> conversion is unusual (e.g., linear history, lots of RAM) so it could be
> that you will hit different limits than my tests did.
>
> Robin definitely had some version of the ExternalBlobGenerator code back
> in 2009.

Yeah, I think that patch is what I'm remembering for divergence.


> > If memory servers, where we hit major issues was in dealing w/
> > encoding- crap like names, some bad latin-1 sort of data.  Will have
> > to re-run this and check.
>
> This could be fixed in cvs2git with a reasonable amount of effort.  I
> already did a lot of restructuring to allow EOL handling and keyword
> expansion options to be specified via properties (see
> doc/properties.txt) and to make the properties available earlier in the
> conversion.  I think the main work that remains is to pass the
> properties to the generate_blobs.py script and to teach the script to
> massage the revision contents before writing them to the blobfile.
>
> > Re: pgsql, good to here- they don't have our whacked tree structure,
> > but I'm sue the encoding sort of issues we had, they probably had a
> > couple of.
> >
> > As for .cvsignore, at least for gentoo-x86 (repo in discussion), we've
> > got none of that in use- so non issue, but for our other repos, yeah,
> > it'll come up.  Manually translating and stacking a commit on top of
> > the results is perfectly acceptable (your tools already cover 99%
> > after all).
>
> I think .gitignore files would be pretty easy, too.  It's another one of
> those things that I never got around to before being distracted by other
> projects (I hack mostly on git itself now).
>
> >> More details and/or bug reports would be much appreciated if the
> >> problems still exist in the 2.4.0 release (which is just out, though it
> >> is well-used code with hardly any recent changes).
> >
> > Honestly, I couldn't give you the details; that code I started from in
> > '10 was inherited; it was already a year old branching/release, w/
> > local modifications before I got started (I do recall verifying that
> > the modifications were necessary, else things failed miserably).
>
> Is the code that you were using visible somewhere?

Robin would know, but my suspicion is 'no'; your description of an 
early blob generator sounds right.

Offhand, I can tell the optimizations taht were done beyond that I 
upstreamed (or you/I agreed to leave them out); sole exception was 
some snakeoil inlining/usage, but that wasn't a massive gain (5% or 
so).

I'm offline till sunday/monday, but I'll fire off a run during then- 
barring it failing, will report stats when I get back from vacation.

~brian