Gentoo Logo
Gentoo Spaceship




Note: Due to technical difficulties, the Archives are currently not up to date. GMANE provides an alternative service for most mailing lists.
c.f. bug 424647
List Archive: gentoo-dev
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Headers:
To: Michał Górny <mgorny@g.o>
From: Corentin Chary <corentin.chary@...>
Subject: Re: euscan proof of concept (like debian's uscan)
Date: Fri, 23 Sep 2011 11:55:57 +0200
On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny <mgorny@g.o> wrote:
> On Mon, 19 Sep 2011 10:39:11 +0200
> Corentin Chary <corentin.chary@...> wrote:
>
>> ## Also update eix database, because we use eix internaly
>> ## Bottleneck: disk and cpu
>> ##Time: 30mn ~ 1h
>> eix-update
>
> Using egencache to keep caches for overlays will make eix updates much
> faster.
>
> Here's my code for it (it uses overlays in /usr/portage/local):
>
> cd /usr/portage/local && \
> for O in */; do
>        echo "${O}"
>        egencache --jobs=8 --update --update-use-local-desc --rsync \
>                --repo=$(cat ${O}profiles/repo_name)
> done
>

Just done that, and indeed, it's much faster.

Also I switched to PostgreSQL, and since that avoids a lot of deadlock
situations when making queries in parallel, I'm now able to do that:

eix --only-names -x | gparallel --eta --load 8 --jobs 400%
--max-args=64 python manage.py scan-metadata

And now it takes less than 30mn instead of more than 1h, and more
importantly, it scales.

-- 
Corentin Chary
http://xf.iksaif.net


References:
euscan proof of concept (like debian's uscan)
-- Corentin Chary
Re: euscan proof of concept (like debian's uscan)
-- Corentin Chary
Re: euscan proof of concept (like debian's uscan)
-- Michał Górny
Navigation:
Lists: gentoo-dev: < Prev By Thread Next > < Prev By Date Next >
Previous by thread:
Re: euscan proof of concept (like debian's uscan)
Next by thread:
masking mailwrapper
Previous by date:
Re: Packages up for grabs: virtual/{cron,dev-manager,inetd,libc,linux-sources,man,os-headers,package-manager,skkserv,ssh,w3m}
Next by date:
zlib breakage


Updated Jun 29, 2012

Summary: Archive of the gentoo-dev mailing list.

Donate to support our development efforts.

Copyright 2001-2013 Gentoo Foundation, Inc. Questions, Comments? Contact us.