Re: [gentoo-scm] Bizarre Python Issue with Validation

[Rich Freeman]

On Sun, Oct 14, 2012 at 8:08 PM, Brian Harring <ferringb@×××××.com> wrote:
> Probably not the answer you want... but I suggest you valgrind it.  If
> the behaviour is differing for Repository()[some-hash], that makes me
> think there is a ref counting bug afoot.

Ugh.

>
> Also, wtf is dumbo?

It is a simplified framework for doing parallel jobs that can run on
hadoop.  That said, after finding some documentation on just running
python on hadoop directly I'm considering just going that route
(should be easy to change).

> And can you clarify how this is doing it's
> validation, rough runtime, etc?

See my earlier posts on this list.  Basically the goal is to break
both git/cvs into sorted lists of files and their individual histories
and then compare them.  They would be one commit per line.  Due to how
Hadoop works I'm just using csv with a few fields in base64.

If you're looking at the python file I linked to, keep in mind that
what it does is takes in a list of trees/blobs in such a format,
decends one level on each, removes duplicates, and outputs a new list
of trees/blobs.  That would be itereated.  Since each commit typically
only changes a few files at the bottom of the tree it the list would
not be expected to grow much on each iteration (all but one entry at
the top level of the tree would be a duplicate from one commit to the
next).  However, if several things did change they'd end up in the
list.

This treatment allows each record to be treated entirely in parallel.
You start out with 1.2M of them which is how many commits there are in
the git repository (at least the one I'm testing on), grows 5-40x
after the map step, and shrinks to just over 1x on the reduce step.
When you run a step and either end up with all blobs or unchanged
output you know you're done (not sure what the fastest way to detect
that condition is - the list isn't THAT big so grep is probably good
enough on the interim files).

> Minimally, it's going to need to go
> parallel- I rebuilt the conversion bits (parallelized along category
> lines, then reintegrating after the fact), and it's around 50m for
> run- validation being equivalent would definitely be preferable.

That was my concern as well.  The solution is parallel and I'm looking
to run it on hadoop.  In theory the cvs side should be easier to do -
I'm running into more issues with the tools than the software.  It
works just fine on a few thousand commits with my last design.
Actually, once I ditch the dumbo framework I might see how it does
without a cluster - I was running into RAM issues before and the new
design would operate as a pipe.  As long as you chop up the input file
at line boundaries and feed it to the map step and concatenate/sort
the output of the map step to feed it to reduce anything that runs the
script in parallel would work.

In fact, one of my concerns is just the time it takes to transfer that
2GB git repository over to a cluster and get everything running -
especially if you've got the conversion time down to 50m (I heard we
were at 8h previously).  I was hoping to have the testing done in
under an hour.

Shouldn't take more than a few days for me to get around to modifying
my script to not require dumbo and just testing it out on datasets in
serial.  If I can get the speed up we might just be able to run it on
whatever host you're using for the conversion and that would eliminate
a lot of hassle.

>
> @mhagger; details of that I'll share in a bit- roughly, it's
> exploiting the fact gentoo-x86 /never/ has cross category commits (no
> one does repo wide commits, although detection/merging of that I'm
> checking for).

Well, my validation only checks that the per-file history is correct,
which I think is all cvs really lets you do anyway.  So, it won't
distinguish between a->a' then b->b' vs (a,b)->(a',b').  However, if
your code is somehow losing changes it will certainly find that.
Also, if the timestamps on two commits are the same it will not
distinguish between the order of those two commits, but I don't think
that info is really in cvs either.  I haven't gotten far enough to
discover if there are any issues with time resolution/conversion/etc.

I'm certainly interested in seeing what you came up with.  I take it
that you're writing something from scratch here?

Rich