Re: [gentoo-portage-dev] emerge threaded

[Brian Harring]

On Mon, Apr 24, 2006 at 07:55:17PM -0700, Brian wrote:
> That however could change for the next major version of portage.  I have
> not attempted to try an internal build in the imported pkgcore, but it
> may well work ok.  That is something that is yet to be tested.  Brian
> Harring does not know how pkgcore will handle being threaded.  It is too
> early in it's development.

Locking has been setup in a few spots (not enabled mostly, fex vdb 
repo locking)- what's not in place (and matters) is that internally, 
pkgcore uses pkgcore.util.caching pretty heavily- used to generate 
unique instances that are reused.

For example, if you get the package instance sys-apps/portage-2.0.51 
from repo A and shove it in a list (thus holding onto it in memory), 
if you grab sys-apps/portage-2.0.51 from repo A *again*, you get the 
same instance back (literally same instance).

So... generation of the instance (or lookup of the old instance) is 
a critical section- have to ensure that updates to the dict of what 
pkgs are in memory (whether removal or addition of pkgs) is atomic so 
you don't get N copies of pkg A.

Fortunately, python's GIL protects deals with the "pkg a is being 
unalloc'd while it's about to be handed back"- GIL + ref counting 
prevents that from being an issue.

Aside from that... the only other spots where locking is required is
1) build directory usage- don't want two instances executing 
configure/make in the same directory fex, mainly because ebuild 
execution may mangle things in install phase (which would break things 
for compile phase).
2) sync locking.  Debatable on that one (best you can manage is 
invalidating any pkgs from that repo in memory, which can cause 
issues).
3) cache updates (debatable also).  Downside of races there is that 
cache regen for a node occurs twice (rather then once), which needs to 
be weighed against cost of locking overhead...

Usual locking requirements stable has (distdir fex) applies also.
~harring