Re: [gentoo-scm] Bizarre Python Issue with Validation

[Brian Harring]

On Sat, Oct 13, 2012 at 06:12:14PM -0400, Rich Freeman wrote:
> In my validation script I'm running into a bizarre issue.
>
> This script fails if run under dumbo:
> https://github.com/rich0/gitvalidate/raw/master/gitdump/iteratetree.py
>
> The dumbo command line was:
> dumbo start iteratetree.py -input /outfile.csv -output /out
>
> Outfile.csv just had 3 entries in it for testing (generated by
> parsetrees.py in that directory).
>
> It failed with a KeyError on the line:
> tree=repository[unicode(filehash)];
>
> The KeyError error message contained the correct hash.
>
> To eliminate variables I confirmed that replacing that line with this
> one generates the same error message:
> tree=Repository("/data/git/gentoo-x86")[unicode('5789de7076bfc534787fb3b072652810ca1ad197')]
>
> (all I did was hard-code everything for one entry to cut out all the
> other parsing logic).
>
> However, the following script works just fine:
> #!/usr/bin/python
>
> from pygit2 import Repository,GIT_OBJ_TREE;
> tree=Repository("/data/git/gentoo-x86")[unicode('5789de7076bfc534787fb3b072652810ca1ad197')]
> print tree
>
> Those are the exact same statements from the failing script, with the
> same hard-coded values.  Running those commands interactively works
> fine also, as does stepping through the script logic.
>
> The main difference is how the execution flow reaches that point.  In
> the simple script it is just outright run.  In the case of the failing
> one it is in an iterator that is invoked by dumbo.
>
> Has anybody seen this kind of behavior in python - where an interator
> fails if it is called as part of a map statement but the same
> statements otherwise work normally?  Perhaps I'm committing some
> obscure (to me) python sin.

Probably not the answer you want... but I suggest you valgrind it.  If 
the behaviour is differing for Repository()[some-hash], that makes me 
think there is a ref counting bug afoot.

Also, wtf is dumbo?  And can you clarify how this is doing it's 
validation, rough runtime, etc?  Minimally, it's going to need to go 
parallel- I rebuilt the conversion bits (parallelized along category 
lines, then reintegrating after the fact), and it's around 50m for 
run- validation being equivalent would definitely be preferable.

@mhagger; details of that I'll share in a bit- roughly, it's 
exploiting the fact gentoo-x86 /never/ has cross category commits (no 
one does repo wide commits, although detection/merging of that I'm 
checking for).

Cheers-
~harring