Re: [gentoo-user] slocate masked

[Alan McKinnon]

Apparently, though unproven, at 23:00 on Wednesday 17 November 2010, Paul 
Hartman did opine thusly:

> On Wed, Nov 17, 2010 at 2:35 PM, Mick <michaelkintzios@×××××.com> wrote:
 
> > Why is the second time so much faster?  The size of the derived db was
> > the same on both occasions.
>
> I guess caching like Volker said too. What happens if you do something
> like this twice:
>
> sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb

Now I'm intrigued. I did some quick and nasty tests.

First, mlocate's updatedb. No measures taken to invalidate caches etc:

# time updatedb
real    0m39.265s
user    0m2.245s
sys     0m0.228s


Then unmerge mlocate, emerge slocate, delete all dbs, run slocate's updatedb 
twice:

# rm /var/lib/[ms]locate/*db
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    1m35.365s
user    0m5.941s
sys     0m0.383s
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb     
real    1m34.929s
user    0m5.925s
sys     0m0.377s

slocate seems quicker than the few tests I'd already done with mlocate and has 
no optimizations to re-use existing correct data in the db. Now unmerge 
slocate, merge mlocate, do not delete dbs and run mlocate's updatedb twice:

# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    3m50.574s
user    0m7.277s
sys     0m0.361s
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    1m5.830s
user    0m2.088s
sys     0m0.173s

Second run definitely quicker as it only has to read the fs, not write the 
entire index as well. But that initial run ... The old slocate db was still 
around, possibly affecting the first run, so delete both db's and run 
mlocate's updatedb twice:

# rm /var/lib/[ms]locate/*db
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    3m51.592s
user    0m7.249s
sys     0m0.350s
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    1m7.662s
user    0m1.997s
sys     0m0.159s

Almost identical to the prior test, so the presence of slocate's db has no 
effect on mlocate. Then I realized I hadn't measured how long they took to 
reindex a largely cache'd fs so I tried that with both, deleting the db's at 
each test:

slocate:
# rm /var/lib/[ms]locate/*db
rm: cannot remove `/var/lib/[ms]locate/*db': No such file or directory
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    1m34.341s
user    0m5.929s
sys     0m0.397s
# time updatedb
real    0m2.454s
user    0m0.855s
sys     0m1.569s

mlocate:
# rm /var/lib/[ms]locate/*db
# sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
real    3m54.792s
user    0m7.215s
sys     0m0.350s
# time updatedb
real    0m0.538s
user    0m0.302s
sys     0m0.232s

0.5 second vs 2.5 seconds. Wow.

Conclusions:

1. mlocate is slow at building it's db from scratch - about 250% as long as 
slocate on the same task. 
2. mlocate is faster at reindexing a largely-unchanged fs - it does it in 
about 66% of the time slocate took.
3. mlocate is insanely quick at reindexing a db that is in cache.

#1 is are - most systems will only do it once
#3 is silly and does not represent anything close to reality
#2 is pretty realistic and a 33% performance boost is significant

I have no idea where the speed increase in #3 comes from. This is an ext4 fs - 
does ext4 keep an in-memory hash of inodes it reads? It seems to me that would 
be a very clever and very useful thing for an fs to do.


-- 
alan dot mckinnon at gmail dot com