[gentoo-dev] Questions/Suggestions about the metadata databases

[Georgi Georgiev]

Hello list,

I have a couple of questions related to the modules that handle the "databases"
in /usr/portage/metadata/cache and /var/cache/edb/dep at the moment. 

First I'd like to know what the purpose of the following code in
/usr/bin/emerge is:

        if os.path.exists(myportdir+"/metadata/cache") and updatecache_flg:
                print "\n>>> Updating Portage cache...  ",
                os.umask(0002)
                cachedir = os.path.normpath(portage.settings["PORTAGE_CACHEDIR"])
                if cachedir in ["/",    "/bin", "/dev",  "/etc",  "/home",
                                "/lib", "/opt", "/proc", "/root", "/sbin",
                                "/sys", "/tmp", "/usr",  "/var"]:
                        print "!!! PORTAGE_CACHEDIR IS SET TO A PRIMARY ROOT DIRECTORY ON YOUR SYSTEM."
                        print "!!! This is ALMOST CERTAINLY NOT what you want: "+str(cachedir)
                        sys.exit(73)
                if os.path.exists(cachedir):
                        portage.spawn("rm -Rf "+cachedir+"/*",portage.settings,free=1) 
                else:
                        os.mkdir(cachedir)


I am talking about the last four lines. Searching for "-Rf" in /usr/bin/emerge
finds it right away. Is this really necessary?

Another question. Why has there been no talk about the database modules
anywhere?  Turns out one can use db databases (or some other format if they
don't have db) by doing:

# echo 'portdbapi.auxdbmodule="portage_db_anydbm.database"' >> /etc/portage/modules

I couldn't find anything in the forums by searching for anydbm, or anywhere
else for that matter. Isn't the code stable? It sure looks good enough. The
only problem is that "emerge metadata" takes forever to convert the flat
database from /usr/portage/metadata/cache/ to anydbm format in
/var/cache/edb/dep/.  However, if rsync distributes not only the flat database,
but also the .anydbm files, it would be pretty easy to simply copy the files
that are in the same format as what the user decides to use on their system, and
not to do the time-consuming database-by-database, key-by-key copy. For
bandwidth and space concerns, the irrelevant databases can be excluded from the
rsync process. Something like:

# if we want the anydbm files
rsync --exclude=/metdata/cache/*/

# if we want the flat database
rsync --exclude=/metdata/cache/*.anydbm

Using db database files in /var/cache/edb/dep/ really speeds up things.

Another concern I have is that there is no easy way to set the locations for
the databases.  Currently the paths $PORTDIR/metadata/cache and
/var/cache/edb/dep/ are hard-coded. I decided to get a faster database and
coded a mysql module, but the configuration currently is like this:

# cat /etc/portage/modules.mysql
/usr/portage/metadata/cache = "modules.mysql.meta"
/var/cache/edb/dep/         = "modules.mysql.aux"

# cat /etc/portage/modules.myslq.meta
host	= 'localhost'
user	= 'portage'
password= ''
db	= 'portage'
table	= 'metadata'

# cat /etc/portage/modules.myslq.aux
host	= 'localhost'
user	= 'portage'
password= ''
db	= 'portage'
table	= 'auxdb:ols-dell'
table	= 'metadir'

Using the mysql module is faster than the anydbm databases. Plus, it is not
necessary to have the auxdb on localhost. Would there be interest in this
thing? Should I attach it to bugzilla?

-- 
\    Georgi Georgiev   \  The Golden Rule is of no use to you whatever \
/     chutz@×××.net    /  unless you realize it is your move. -- Frank /
\   [phone]   \  Crane                                        \