Re: [gentoo-portage-dev] Cache rewrite backport

[Bastian Balthazar Bux]

Brian Harring ha scritto:
> On Wed, Oct 12, 2005 at 12:01:12AM +0200, Bastian Balthazar Bux wrote:
>
>>Sorry, but here the results are not those expected:
>
> .51.22 vs .53_rc5... try with a vanilla .53_rc5 please

here they are, also added a test with a dirty trick to precharge the
portage dir and see what happen. Look like there is a small improvement.
Now it's late.

==== time emerge --metadata; 1st run; 2.0.53_rc5 vanilla
real    9m44.449s
user    4m51.034s
sys     0m24.754s

==== time emerge --metadata; 2nd run; 2.0.53_rc5 vanilla
real    2m50.932s
user    0m12.597s
sys     0m3.836s

==== time emerge --metadata; 3rd run; 2.0.53_rc5 vanilla
real    1m55.445s
user    0m12.501s
sys     0m3.416s

==== tar -c /usr/portage/* >/dev/null & time emerge --metadata
==== ; 4th run; 2.0.53_rc5 vanilla
real 1m10.275s
user 0m13.377s
sys  0m4.740s


==== time emerge --metadata; 1st run; 2.0.53_rc5 patched
real 4m30.186s
user 0m12.757s
sys  0m9.921s

==== time emerge --metadata; 2nd run; 2.0.53_rc5 patched
real 4m41.021s
user 0m12.597s
sys  0m9.297s

==== time emerge --metadata; 3rd run; 2.0.53_rc5 patched
real 4m44.544s
user 0m12.521s
sys  0m9.457s

==== tar -c /usr/portage/* >/dev/null & time emerge --metadata
==== ; 4th run; 2.0.53_rc5 patched
real 4m12.131s
user 0m13.661s
sys  0m10.329s

>
>
>
>>==== time emerge --metadata; 1st run; 2.0.51.22-r3
>>real    2m24.419s
>>user    0m12.329s
>>sys     0m3.644s
>>
>>==== time emerge --metadata; 2nd run; 2.0.51.22-r3
>>real    1m17.700s
>>user    0m12.257s
>>sys     0m2.976s
>>

[snip]
the 2.0.51.22-r3 ones are still much faster on "real", please shade a
light into my ignorance

>
>
> Wasn't expecting a massive improvement, although wasn't sure as hell
> wasn't expecting a 3x increase in sys. :)
>
> Should've seen a large tweak for the first .53_rc5 run also, since it
> (essentially) would be a forced rewrite of the cache due to INHERITED
> vs _eclasses_ key changes (moving eclass_cache into the backend).

look like it is :)

>
> Not running anything additional via /etc/portage/modules I'd bet, but
> asking to verify also...

good bet, /etc/portage/modules does not exist on this box

>
> Meanwhile, thanks for testing; contrary to other results, but _any_
> regression I'm after.
> ~harring

No, thank you to work on this, every time I've tryed to dive in portage
I needed some day of hospital.

look also at this additional try:

==== cp -a cache/* /dev/shm/
==== mount -obind /dev/shm /usr/portage/metadata/cache/
==== tar -c /usr/portage/* >/dev/null & time emerge --metadata
==== ; Nth run; 2.0.53_rc5 patched

real 3m43.653s
user 0m12.937s
sys  0m9.817s

IMHO the "real" time of an emerge --metadata could be improved acting in
two ways:
1) preload as much as possible data (stats included) from disk before to
parse it
2) separating disk read  from disk writes (i.e many disk read followed
by many disk writes followed ...)

some idea?

Fetching virtual information from a cache save thousand of disk reads
# find /var/db/pkg/ -name PROVIDE \
  \( -exec echo -n {}\: \; -and -exec cat {} \; \) \
  | egrep -v "PROVIDE:$"

not the same for use, it's only called seldom (at least on the run>=2)
# find /var/db/pkg/ -name USE \
  \( -exec echo -n {}\: \; -and -exec cat {} \; \) \
  | egrep -v "USE:$"

read behind the ebuilds ...

bla bla bla

cheers, francesco

P.S.
find ... PROVIDE ... isn't the best, because PROVIDE are multiline
files, but I don't know how to join the lines
-- 
gentoo-portage-dev@g.o mailing list