Gentoo Archives: gentoo-dev

From: Corentin Chary <corentin.chary@×××××.com>
To: "Michał Górny" <mgorny@g.o>
Cc: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] euscan proof of concept (like debian's uscan)
Date: Fri, 23 Sep 2011 09:57:06
Message-Id: CAHR064h4_MUxOOmBgTZ9RY+nb2yZybaf2JNjmWr+XQMuFyVUxw@mail.gmail.com
In Reply to: Re: [gentoo-dev] euscan proof of concept (like debian's uscan) by "Michał Górny"
1 On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny <mgorny@g.o> wrote:
2 > On Mon, 19 Sep 2011 10:39:11 +0200
3 > Corentin Chary <corentin.chary@×××××.com> wrote:
4 >
5 >> ## Also update eix database, because we use eix internaly
6 >> ## Bottleneck: disk and cpu
7 >> ##Time: 30mn ~ 1h
8 >> eix-update
9 >
10 > Using egencache to keep caches for overlays will make eix updates much
11 > faster.
12 >
13 > Here's my code for it (it uses overlays in /usr/portage/local):
14 >
15 > cd /usr/portage/local && \
16 > for O in */; do
17 >        echo "${O}"
18 >        egencache --jobs=8 --update --update-use-local-desc --rsync \
19 >                --repo=$(cat ${O}profiles/repo_name)
20 > done
21 >
22
23 Just done that, and indeed, it's much faster.
24
25 Also I switched to PostgreSQL, and since that avoids a lot of deadlock
26 situations when making queries in parallel, I'm now able to do that:
27
28 eix --only-names -x | gparallel --eta --load 8 --jobs 400%
29 --max-args=64 python manage.py scan-metadata
30
31 And now it takes less than 30mn instead of more than 1h, and more
32 importantly, it scales.
33
34 --
35 Corentin Chary
36 http://xf.iksaif.net