Gentoo Archives: gentoo-performance

From: Alec Warner <warnera6@×××××××.edu>
To: gentoo-performance@l.g.o
Subject: Re: [gentoo-performance] gentoo-performance
Date: Fri, 20 Jan 2006 06:34:22
In Reply to: Re: [gentoo-performance] gentoo-performance by Jeremy Brake
Hash: SHA1

Jeremy Brake wrote:
> How about speeding up the wait time on updating the portage cache after > a sync.. even on my AMD 64 3500 it takes a number of minutes to chug > through.. > are there any known ways to "vrrmmm" this up a little? > > Jeremy
Well the current problem is that in the 2.0 portage branch the cache code sucks. This is fixed in ~arch portage ( the 2.1_pre series ). For you users that don't want to upgrade to unstable, you should be able to use cdb to speed up the process. Setting RSYNC_EXCLUDES will not speed up the second half of the --sync ( the --metadata portion ). Explanation: The Portage Tree has a ton of ebuilds in it, when you do something like emerge -pv <blar> portage used to have to go read each ebuild and be like "oh what are the USE flags, the dependencies, etc.." This takes a lot of time, especially for something like emerge -uD world that may touch thousands of packages. So to mitigate this issue portage has a "metadata cache" where it stores the ebuild metadata so it doesn't have to source each ebuild every time. This gives performance increases ( reportedly ) of 100x speed-ups. However, generating the metadata is time consuming, even on decent hardware it will probably take 30-45 minutes. To not piss the users off, Gentoo calculates this metadata serve-side and serves it to you during sync. The Rsync portion of emerge --sync is pretty much everything before the "updating metadata cache". This should be pretty standard as far as time goes on all boxes. However the second portion is where portage has to take the generated server-side cache and kind of "meld" it into your already existant local cache ( stored in /var/cache/edb/dep for those of you whom are curious ). This didn't used to take a bunch of time..until KDE split their packages from KDE-monolith to KDE-meta. The switch probably won't help much in this area either. Particularly the slowdown at 50-52% is due to this portion of the tree. There is a new cache system in the 2.1 branch of portage, or using cdb, or perhaps even an sqlite back-end (patches and modules are out there), will help until such time as the 2.1 branch is stablized ( probably not too soon ). - -Alec Warner (antarus) Disclaimer: I'm not a developer (yet) but I've worked with the portage team for some time. Release dates, features, and other things are not gauranteed to be correct just because I said so :) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Thunderbird - iQIVAwUBQ9CExWzglR5RwbyYAQLJEhAAj7i2cRVgv1iBfi61mGkn9q1t/K5JEJcv zK07bTmMDirviessdKiZVSufXdWV79tW0MDEVqmGI9t+V+uwM77aFbge1VSkEaQB RWetuQL8tQRUX1KvQ+ItScVtbqKIAQe/Jn+BwSim05jF3+fF15Z6EpPPKypNLQxK 2ef/bcJ91Gctv0xcd6j943uOPFDCt05Ahe06/pQ0wgGdnAcKLOy/RVwDOVfprXim AwiVsU4aCxRI056RkEj1VuSwSYQEa91WKpTGv81lZVkRW8LRxkgc7UAydMYGjOY5 1prjv2Koyly5ycVvxshKVmLfuaqByY9bBnlklyKDdFW1ZD1udCflCLxmz3GuTCsm /FHce+Y9smqq3sF1wV7DYXu9vTnLdBqVBlDq4cEtd5XdgQm3TJ5rUGF2Mepicyhg Bg4ibDExDB5eKWCnGU3ioSCd8TY9cdR9ZXxpm6JLGfr9ztog0/vScsIZbj3dS4RH WqGklvW0F9N6NjP26WJwtQsmSIZpSJU4yPxneZOdQxGOUfdNPQap+qO+rZaitPKW yWaikaiSuecPSul0ithpGnCFttPHrBLyNKNl7aom04Bht4KSdaqzriIL/RylWzHW OgazUV9RuVdI8oEx+q3zKzOgvG3dXeL7HNdhH+j9noIyGVa8ouNHWMGfFUc1baxV 7eyB1buSeQY= =vWqe -----END PGP SIGNATURE----- -- gentoo-performance@g.o mailing list


Subject Author
Re: [gentoo-performance] gentoo-performance (sync speedups) lnxg33k <lnxg33k@×××××.com>
Re: [gentoo-performance] gentoo-performance Jeremy Brake <gentoolists@×××××××××××.nz>