[gentoo-portage-dev] portage python module and caching behaviour

[Jean-Christophe Petkovich]

I'm currently working on solving a strange bug in a daemon like process
that uses layman and the portage python module to determine whether or
not the daemon requires updates (and if so, self updates). It's highly
unlikely that what I'm doing falls under the normal portage usecase, but
I would like some help understanding how the caching mechanism works in
the portage module. I'm aware that the portage python module uses _some_
kind of caching or lazyness to store queries it makes on the portage
tree, but I'm having some trouble figuring out exactly how it works.

I'm updating the portage tree in the background while periodically
running a check of the most recently available version of a particular
package. At some point, portage appears to cache the result of my query
and stop hitting the disk, and from that point forward, it never hits it
again until the process is killed or something outside that process
calls some kind of emerge command. I know from reading portage's source
that there is some kind of lazyness or caching going on, but I'm curious
why it doesn't seem to happen right away. It seems to me like I can keep
querying the disk happily for some amount of time before that process
begins to receive the same results per query every time. Since it seems
to be happening only after some event that occurs over time, it is
basically impossible to reproduce. Can anyone enlighten me?

Here is an overview of what the process does:

1. daemon spins up
2. checks if it needs an update
2.1 if it needs an update, do so and restart the daemon goto 1
2.2 if no update is required go to 3
3. do some work, goto 2

Step 2 utilizes layman for syncing an overlay, and then uses the portage
python module for checking if the newest version is newer than the
installed version with calls like the following:

    # to get currently installed version of the daemon
    installed_versions = portage.db[portage.root]['vartree'].dbapi.match(pattern)
    
    # to find the most recent version of the daemon
    available_versions = portage.db[portage.root]['porttree'].dbapi.match(pattern)
    

I've seen a method in the dbapi called flush_cache that sounds
promising. If it indeed flushes the cache that I'm running into, it
should fix this problem I would imagine.

Please let me know if this question is inappropriate for this list,

Best Regards,

-- 
Jean-Christophe Petkovich