1 |
On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny <mgorny@g.o> wrote: |
2 |
> On Mon, 19 Sep 2011 10:39:11 +0200 |
3 |
> Corentin Chary <corentin.chary@×××××.com> wrote: |
4 |
> |
5 |
>> ## Also update eix database, because we use eix internaly |
6 |
>> ## Bottleneck: disk and cpu |
7 |
>> ##Time: 30mn ~ 1h |
8 |
>> eix-update |
9 |
> |
10 |
> Using egencache to keep caches for overlays will make eix updates much |
11 |
> faster. |
12 |
> |
13 |
> Here's my code for it (it uses overlays in /usr/portage/local): |
14 |
> |
15 |
> cd /usr/portage/local && \ |
16 |
> for O in */; do |
17 |
> echo "${O}" |
18 |
> egencache --jobs=8 --update --update-use-local-desc --rsync \ |
19 |
> --repo=$(cat ${O}profiles/repo_name) |
20 |
> done |
21 |
> |
22 |
|
23 |
Just done that, and indeed, it's much faster. |
24 |
|
25 |
Also I switched to PostgreSQL, and since that avoids a lot of deadlock |
26 |
situations when making queries in parallel, I'm now able to do that: |
27 |
|
28 |
eix --only-names -x | gparallel --eta --load 8 --jobs 400% |
29 |
--max-args=64 python manage.py scan-metadata |
30 |
|
31 |
And now it takes less than 30mn instead of more than 1h, and more |
32 |
importantly, it scales. |
33 |
|
34 |
-- |
35 |
Corentin Chary |
36 |
http://xf.iksaif.net |