On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny <mgorny@g.o> wrote:
> On Mon, 19 Sep 2011 10:39:11 +0200
> Corentin Chary <corentin.chary@...> wrote:
>
>> ## Also update eix database, because we use eix internaly
>> ## Bottleneck: disk and cpu
>> ##Time: 30mn ~ 1h
>> eix-update
>
> Using egencache to keep caches for overlays will make eix updates much
> faster.
>
> Here's my code for it (it uses overlays in /usr/portage/local):
>
> cd /usr/portage/local && \
> for O in */; do
>        echo "${O}"
>        egencache --jobs=8 --update --update-use-local-desc --rsync \
>                --repo=$(cat ${O}profiles/repo_name)
> done
>

Just done that, and indeed, it's much faster.

Also I switched to PostgreSQL, and since that avoids a lot of deadlock
situations when making queries in parallel, I'm now able to do that:

eix --only-names -x | gparallel --eta --load 8 --jobs 400%
--max-args=64 python manage.py scan-metadata

And now it takes less than 30mn instead of more than 1h, and more
importantly, it scales.

-- 
Corentin Chary
http://xf.iksaif.net