Re: [gentoo-portage-dev] cache subsystem replacement

[Brian Harring]

On Tue, Nov 15, 2005 at 01:13:58AM +0900, Jason Stubbs wrote:
> Was talking with a guy yesterday who mentioned he had 10 line patch that sped
> up current portage a lot with regard to updating metadata. I asked him to
> send it to me and here it is:
>
> --- -   2005-10-29 18:49:15.156173000 +0900
> +++ /usr/lib/portage/pym/portage_db_cpickle.py  2005-10-08 11:13:37.000000000
> +0900
> @@ -61,6 +61,9 @@
>                 return False
>                         
>         def sync(self):
> +               return
> +
> +       def realsync(self):
>                 if self.modified:
>                         try:
>                                 if os.path.exists(self.filename):
> @@ -74,6 +77,6 @@
>                                 pass
>         
>         def close(self):
> -               self.sync()
> +               self.realsync()
>                 self.db = None;

Ok, your mail client is screwing stuff up here ;)

The problem with the trick above is that, yeah, it delays syncs, but 
it also means if portage shuts down uncleanly _ever_, the entire 
eclass db of the old cache format is invalidated.  
All of it.  
Back to square one.
Massively bad thing, obviously.

This is why the default sync rate of cache classes in the rewrite is 
1 also, it updates every time a change is pushed to it.

> I remembered seeing sync_rate when glancing through the new cache stuff and
> then had a look into mirror_cache(). Playing with trg_cache.sync(x), I got
> the following numbers.
>
> x        total #1  total #2  total #3  median sys
> 1          13.651    13.451    13.727       2.712
> 10         13.413    13.412    13.645       2.538
> 100        13.605    13.498    13.405       2.700
> 1000       13.673    13.726    13.748       2.839
> 10000      14.541    14.054    13.447       2.743
> 100000     13.973    13.951    14.512       2.881
> 1000000    13.583    13.622    13.935       2.669
>
> Command run was:
>
> rm -rf /var/cache/edb/dep/*; time emerge -q metadata
>
> So what does changing the sync_rate actually do? Ease seeks? Should I re-run
> these tests with a reboot in between? (And what happened to the 4 seconds I
> was getting with earlier patches? Bug fixes turn quantity into quality? :)

Umm... 4 seconds?  Eh?

Regarding what the sync_rate does, if the target cache supports 
batched updates (think rdbms), it is capable of delaying upto N 
modifications prior to pushing the change out.

a cdb/cpickle cache backend would want to use this fex.

Meanwhile, why you're not seeing any variation- I'm pretty much 
positive you're using a cache that autocommits, meaning delayed 
sync'ing isn't possible.  Autocommit == can't batch, so sync rate 
isn't used/valid.

The only cache in the rewrite that doesn't autocommit is the sqlite 
implementation (which coincidentally is why sync rate exists; inserts 
into sqlite are !@*#!@#*ing slow).
~harring