Re: [gentoo-portage-dev] Cache rewrite backport

[Brian Harring]

On Wed, Oct 12, 2005 at 03:49:44AM +0200, Bastian Balthazar Bux wrote:
> Brian Harring ha scritto:
> > On Wed, Oct 12, 2005 at 12:01:12AM +0200, Bastian Balthazar Bux wrote:
> >
> >>Sorry, but here the results are not those expected:
> >
> > .51.22 vs .53_rc5... try with a vanilla .53_rc5 please
>
> here they are, also added a test with a dirty trick to precharge the
> portage dir and see what happen. Look like there is a small improvement.
> Now it's late.
>
> ==== time emerge --metadata; 1st run; 2.0.53_rc5 vanilla
> real    9m44.449s
> user    4m51.034s
> sys     0m24.754s
>
> ==== time emerge --metadata; 2nd run; 2.0.53_rc5 vanilla
> real    2m50.932s
> user    0m12.597s
> sys     0m3.836s
>
> ==== time emerge --metadata; 3rd run; 2.0.53_rc5 vanilla
> real    1m55.445s
> user    0m12.501s
> sys     0m3.416s
>
> ==== tar -c /usr/portage/* >/dev/null & time emerge --metadata
> ==== ; 4th run; 2.0.53_rc5 vanilla
> real 1m10.275s
> user 0m13.377s
> sys  0m4.740s
>
>
> ==== time emerge --metadata; 1st run; 2.0.53_rc5 patched
> real 4m30.186s
> user 0m12.757s
> sys  0m9.921s
>
> ==== time emerge --metadata; 2nd run; 2.0.53_rc5 patched
> real 4m41.021s
> user 0m12.597s
> sys  0m9.297s
>
> ==== time emerge --metadata; 3rd run; 2.0.53_rc5 patched
> real 4m44.544s
> user 0m12.521s
> sys  0m9.457s
>
> ==== tar -c /usr/portage/* >/dev/null & time emerge --metadata
> ==== ; 4th run; 2.0.53_rc5 patched
> real 4m12.131s
> user 0m13.661s
> sys  0m10.329s
>
> >
> >
> >
> >>==== time emerge --metadata; 1st run; 2.0.51.22-r3
> >>real    2m24.419s
> >>user    0m12.329s
> >>sys     0m3.644s
> >>
> >>==== time emerge --metadata; 2nd run; 2.0.51.22-r3
> >>real    1m17.700s
> >>user    0m12.257s
> >>sys     0m2.976s
> >>
>
> [snip]
> the 2.0.51.22-r3 ones are still much faster on "real", please shade a
> light into my ignorance
Cache had to have been mostly full already; note the 4m51 for .53_rc5; 
.22-r3 would display the same if the cache was invalid (going from 
cache rewrite patch to .53_rc5 vanilla invalidates the local cache).

So... pretty much I'm ignoring the first .53_rc5 run, 2nd/3rd match up 
somewhat with .53.22-r3; main difference that comes to mind is that 
.53_rc5's --metadata code had a collection of extra checks/steps 
thrown in to protect against a lot of annoying tracebacks that were 
rearing there heads, and EAPI was added which would result in 
rewriting the cache entry on the fly.

Don't think it's the case though due to no matching user increase; 
difference in sys pretty much points at some extra IO occuring 
somewhere.


> > Meanwhile, thanks for testing; contrary to other results, but _any_
> > regression I'm after.
> > ~harring
>
> No, thank you to work on this, every time I've tryed to dive in portage
> I needed some day of hospital.

I'd suggest hitting the jim bean personally.  Replace the pounding 
headache with something you at least control... ;)

> look also at this additional try:
>
> ==== cp -a cache/* /dev/shm/
> ==== mount -obind /dev/shm /usr/portage/metadata/cache/
> ==== tar -c /usr/portage/* >/dev/null & time emerge --metadata
> ==== ; Nth run; 2.0.53_rc5 patched
>
> real 3m43.653s
> user 0m12.937s
> sys  0m9.817s

The copy instead of update accounts for that, which shouldn't occur 
with the experimental-4 patch in the other email.

> IMHO the "real" time of an emerge --metadata could be improved acting in
> two ways:
> 1) preload as much as possible data (stats included) from disk before to
> parse it
> 2) separating disk read  from disk writes (i.e many disk read followed
> by many disk writes followed ...)

I actually tried threading the bugger a while back... the improvement 
wasn't quite what I was hoping for, partially due to hitting issues 
with the global interpretter lock in python.
That said, could attempt it again, code for it is mostly 
straightforward

> Fetching virtual information from a cache save thousand of disk reads
> # find /var/db/pkg/ -name PROVIDE \
>   \( -exec echo -n {}\: \; -and -exec cat {} \; \) \
>   | egrep -v "PROVIDE:$"

a central cache of providers for the vdb would certainly make 
python -c'import portage' a helluva lot faster I'd expect.

Any takers to prototype it?
~harring