Re: [gentoo-performance] Re: portage performance

[Sri Gupta]

On Fri, Jun 11, 2004 at 11:02:53PM -0400, Jerry McBride wrote:
> Jesse Guardiani wrote:
>
> > Hello,
> >
> > First off, sorry if this isn't the right place!
> > I didn't see anything more appropriate though, except
> > perhaps Portage-dev, but I'm not a developer.
> >
> > I'm a FreeBSD user switching to Gentoo for my home
> > system. Linux 2.6 is a LOT more stable than any
> > FreeBSD 5.x-RELEASE kernels right now, and Linux
> > has much better support for Wine and such.
> >
> > One of the most obvious differences between FreeBSD
> > ports and Gentoo Portage is how LONG it takes to
> > search for ports/packages with Portage.
> >
> > I have 525 ports installed on my FreeBSD laptop right
> > now, and probably only half of that installed on the
> > Gentoo machine, but the gentoo machine crunches a lot
> > more doing a --search than the FreeBSD machine does
> > with a pkg_version -vs 'name'.
> >
> > And forget about --searchdesc! That takes ages!
> >
> > Is portage not hash indexed or something?
> >
>
> The main problem is this.... the portage database is a file system based
> mechanism. That is to say, when you update, search or otherwise use emerge,
> the program is busy as hell leafing through thousands of files looking for
> what you want..
>
> "Hashed", "indexed"??? Not even close....
>
> Portage is begging for a decent database engine and I pray it's sql
> based....
>
> The other hitch is that portage is written in an interpreted language,
> Python. Python is an excellent programming language, no doubt about it, but
> performance isn't one of it's highlights. Moving portage to C would be so
> much better....

Try using a JIT-style python compiler.

#emerge psyco
#nano -w `which emerge`

Add the bit from try: to pass in like so:

----
import os,sys
os.environ["PORTAGE_CALLER"]="emerge"
sys.path = ["/usr/lib/portage/pym"]+sys.path

try:
        import psyco
        psyco.full()
except ImportError:
        pass

import emergehelp,xpak,string,re,commands,time,shutil,traceback,atexit,signal,s...
---

use tabs for the indented bits.  you can also try the experimental 
database support (I'm using it, seems fine)

# mkdir /etc/portage
# echo 'portdbapi.auxdbmodule="portage_db_anydbm.database"' > /etc/portage/modules 
# echo 'eclass_cache.dbmodule="portage_db_anydbm.database"' >> /etc/portage/modules 

Both of these speed up portage quite a bit...

-Sri