[gentoo-portage-dev] Possible bug in portdbapi.xmatch()

[Brian]

Sorry, this is a long one :(

A user on the forums has been having problems with porthole-0.5.0 only
on his desktop, his laptop has been working fine.  It has been the
dev-util/ecletex package causing porthole to stall when building the
descrition database for searches.

Testing on my machine has similar results.  Porthole is unable to get
the description and other data because it cannot get an ebuild.  It is
difficult to explain in short terms.  Anyway porthole makes frequent
calls to portage for info rather than cache the info again, since the
xmatch() is a caching function.  Porthole's portagelib.py (expanded for
added debug print statements):

def get_versions(full_name, include_masked = True):
    """Returns all available ebuilds for the package"""
    # Note: this is slow, especially when include_masked is false
    criterion = include_masked and 'match-all' or 'match-visible'
    v = portage.portdb.xmatch(criterion, str(full_name))
    utils.debug.dprint("PORTAGELIB: get_versions(); criterion = %s, package = %s, v = %s" %(str(criterion),full_name,str(v)))
    return  v #portage.portdb.xmatch(criterion, str(full_name))

Data returned in portholes debug terminal with the print filter set:

big_squirt trunk # ./porthole -l -d ecletex
PORTHOLE: importing MainWindow
PORTAGE setting =  backends.portagelib
PORTAGELIB: portage version = 2.1.2_rc4
import complete : <module 'backends.portagelib' from '/home/brian/trunk/backends/portagelib.pyc'>
opt, arg  -l  <type 'str'>
opt, arg  -d ecletex <type 'str'>
Debug printing is enabled
('Debug print filter set to ', 'ecletex')
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = ['dev-util/ecletex-0.0.3']
PACKAGE: get_versions(); v = ['dev-util/ecletex-0.0.3']
PACKAGE: get_latest_ebuild(); checking hard masked vers = ['dev-util/ecletex-0.0.3']
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = []
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = []
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = []
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = []
PACKAGE; get_properties(): No ebuild found for dev-util/ecletex!
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = []
PACKAGE: get_versions(); criterion = match-all, package = dev-util/ecletex
PORTAGELIB: get_versions(); criterion = match-all, package = dev-util/ecletex, v = []
READERS: DescriptionReader(); No description for ecletex


This is consistent behaviour for this package as well as
dev-util/eclipse-cdt.  I have browsed around the tree in porthole and
found numerous other packages that do work for masked, hard-masked, or
stable, single ebuild packages, multiple ebuild packages, etc..  It was
happening on portage-2.1.2_rc3-r2.  I synced, upgraded to -2.1.2_rc4 ,
restarted everything and have the same results.


So for another test I opened a new terminal entered python, imported
portage and tried repeated calls to xmatch().  It worked correctly!

>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>> portage.portdb.xmatch('match-visible', str('dev-util/ecletex'))
[]
>>> portage.portdb.xmatch('match-all', str('dev-util/ecletex'))
['dev-util/ecletex-0.0.3']
>>>


Anyway following porthole's program flow to the
package.get_latest_ebuild():

    def get_latest_ebuild(self, include_masked = False):
        """Return latest ebuild of a package"""
        # Note: this is slow, see get_versions()
        # Note: doesn't return hard-masked packages by default, unless in package.unmask
        # unstable packages however ARE returned. To return the best version for a system,
        # taking into account keywords and masking, use get_best_ebuild().
        if self.full_name == "None":
            return ''
        vers = self.get_versions()
        utils.debug.dprint("PACKAGE: get_latest_ebuild(); versions: " + str(vers)) 
        if include_masked:
            utils.debug.dprint("PACKAGE: get_latest_ebuild(); trying portage_lib.best() of versions: " + str(vers)) 
            return portage_lib.best(vers)
        if self.latest_ebuild == None:
            utils.debug.dprint("PACKAGE: get_latest_ebuild(); checking hard masked vers = " + str(vers)) 
            for m in self.get_hard_masked(check_unmask = True):
                while m in vers:
                    vers.remove(m)
            self.latest_ebuild = portage_lib.best(vers)
        return self.latest_ebuild

I discovered that portage.portdbapi.xmatch() returned a pointer to
myval.  It seems porthole while removing the hard masked packages from
the list it was actually removing them from portages cache.

Changing:  vers = self.get_versions()
      To:  vers = self.get_versions()[:] # make a copy

solved the problem.  The part I do not understand is why was it only
screwing up for those packages (there may be more, I did not search all
of them out) and not other similar masked packages.

-- 
Brian <dol-sen@×××××.net>

-- 
gentoo-portage-dev@g.o mailing list