Re: [gentoo-portage-dev] [PATCH] rsync metadata cache patch (obsoletes metadata transfer on sync)

[Brian Harring]

On Sat, Jan 28, 2006 at 11:24:18AM -0600, Andrew Gaffney wrote:
> Zac Medico wrote:
> >I have reimplemented the previous patch as a normal cache module that adds
> >a writable layer on top of the pre-generated metadata.  If you'd like to
> >try this out (with portage-2.1_preX), simply copy metadata_overlay.py into
> >/usr/lib/portage/pym/cache/ and add portdbapi.auxdbmodule =
> >cache.metadata_overlay.database to /etc/portage/modules.
> >
> >I haven't tested this much, but like the previous patch, this shouldn't
> >cause any harm. Feedback would be appreciated.
>
> Portage didn't explode and dep calculation is pretty damn quick with an
> empty /var/cache/edb/dep/,

Shouldn't be any difference here, if you are seeing a difference 
please time it and post the stats here- it should actually be slightly 
slower then a straight flat_hash cache since it's 2 lookups worst case 
instead of single lookup.

If you're seeing a difference, indicative of one of the 
following
1) I'm a moron and am missing something, 
2) innefficiency in flat_hash pulls (since you should be using 
metadata for most of the queries, the actual underlying ro cache)
3) something is funky.

> Is it normal for /var/cache/edb/dep/${PORTDIR}/ to be re-created
> (it's still empty) when this new module is in use?

Flat_hash does that up front so it doesn't have to do tests for the 
directory on __setitem__ calls; so yes, normal :)


Additional issue here zach pointed out is that the current flat_list 
cache format used for rsync metadata/cache doesn't actually bundle 
eclass mtimes, so stale detection for eclasses won't work via this.

Part of the reason the metadata class exists (and can handle 
flat_list/flat_hash internally on it's own)- until flat_hash is the 
rsync tree format, staleness detection won't work for running straight 
from $PORTDIR/metadata/cache

~harring