Re: [gentoo-user] slocate masked

[Mick]

On Wednesday 17 November 2010 21:55:28 Alan McKinnon wrote:
> Apparently, though unproven, at 23:00 on Wednesday 17 November 2010, Paul
>
> Hartman did opine thusly:
> > On Wed, Nov 17, 2010 at 2:35 PM, Mick <michaelkintzios@×××××.com> wrote:
> > > Why is the second time so much faster?  The size of the derived db was
> > > the same on both occasions.
> >
> > I guess caching like Volker said too. What happens if you do something
> > like this twice:
> >
> > sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
>
> Now I'm intrigued. I did some quick and nasty tests.
>
> First, mlocate's updatedb. No measures taken to invalidate caches etc:
>
> # time updatedb
> real    0m39.265s
> user    0m2.245s
> sys     0m0.228s
>
>
> Then unmerge mlocate, emerge slocate, delete all dbs, run slocate's
> updatedb twice:
>
> # rm /var/lib/[ms]locate/*db
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    1m35.365s
> user    0m5.941s
> sys     0m0.383s
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    1m34.929s
> user    0m5.925s
> sys     0m0.377s
>
> slocate seems quicker than the few tests I'd already done with mlocate and
> has no optimizations to re-use existing correct data in the db. Now
> unmerge slocate, merge mlocate, do not delete dbs and run mlocate's
> updatedb twice:
>
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    3m50.574s
> user    0m7.277s
> sys     0m0.361s
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    1m5.830s
> user    0m2.088s
> sys     0m0.173s
>
> Second run definitely quicker as it only has to read the fs, not write the
> entire index as well. But that initial run ... The old slocate db was still
> around, possibly affecting the first run, so delete both db's and run
> mlocate's updatedb twice:
>
> # rm /var/lib/[ms]locate/*db
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    3m51.592s
> user    0m7.249s
> sys     0m0.350s
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    1m7.662s
> user    0m1.997s
> sys     0m0.159s
>
> Almost identical to the prior test, so the presence of slocate's db has no
> effect on mlocate. Then I realized I hadn't measured how long they took to
> reindex a largely cache'd fs so I tried that with both, deleting the db's
> at each test:
>
> slocate:
> # rm /var/lib/[ms]locate/*db
> rm: cannot remove `/var/lib/[ms]locate/*db': No such file or directory
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    1m34.341s
> user    0m5.929s
> sys     0m0.397s
> # time updatedb
> real    0m2.454s
> user    0m0.855s
> sys     0m1.569s
>
> mlocate:
> # rm /var/lib/[ms]locate/*db
> # sync; sh -c "echo 3 > /proc/sys/vm/drop_caches"; time updatedb
> real    3m54.792s
> user    0m7.215s
> sys     0m0.350s
> # time updatedb
> real    0m0.538s
> user    0m0.302s
> sys     0m0.232s
>
> 0.5 second vs 2.5 seconds. Wow.
>
> Conclusions:
>
> 1. mlocate is slow at building it's db from scratch - about 250% as long as
> slocate on the same task.
> 2. mlocate is faster at reindexing a largely-unchanged fs - it does it in
> about 66% of the time slocate took.
> 3. mlocate is insanely quick at reindexing a db that is in cache.
>
> #1 is are - most systems will only do it once
> #3 is silly and does not represent anything close to reality
> #2 is pretty realistic and a 33% performance boost is significant
>
> I have no idea where the speed increase in #3 comes from. This is an ext4
> fs - does ext4 keep an in-memory hash of inodes it reads? It seems to me
> that would be a very clever and very useful thing for an fs to do.

No. 3 is what made me sent my first post.  I was almost convinced that I did 
something wrong, because no sooner had I hit return it completed.

I've deleted the database and rebooted.  This is what I'm getting now on the 
first run:

# time updatedb

real	2m30.729s
user	0m0.723s
sys	0m9.070s

My database is small, this is a relatively slim installation:

# ls -la /var/lib/mlocate/mlocate.db 
-rw-r----- 1 root locate 9326688 Nov 17 22:14 /var/lib/mlocate/mlocate.db
-- 
Regards,
Mick