Gentoo Archives: gentoo-dev

From: Corentin Chary <corentin.chary@×××××.com>
To: gentoo-dev@l.g.o
Subject: Re: [gentoo-dev] euscan proof of concept (like debian's uscan)
Date: Mon, 19 Sep 2011 08:40:55
In Reply to: Re: [gentoo-dev] euscan proof of concept (like debian's uscan) by Dirkjan Ochtman
On Mon, Sep 19, 2011 at 9:35 AM, Dirkjan Ochtman <djc@g.o> wrote:
> On Mon, Sep 19, 2011 at 00:27, "Paweł Hajdan, Jr." > <phajdan.jr@g.o> wrote: >> Okay, I think this is pretty cool and we should find it a new home in >> the Gentoo infrastructure. >> >> I was thinking about with the repo at >>;a=summary >> >> I can act as a proxy committer and reviewer for that code. Could you >> break it up into some smaller parts (preferably backend first) and send >> to me for review (if you're interested)? >> >> How long does it take to generate the reports? > > +1 I think it would be good to run this on Gentoo infra, and I > wouldn't mind helping out. > > Bikeshedding: not sure "reports" is the best name for this, as reports > implies something more static?
Here is how it works, each week I launch this script on lt server. I've got ~30 trees installed with layman. The server is an AMD X2 4600+ with 4GB of RAM and two 80GB HD in raid1 using ext4. My network bandwidth is 20Mbps down 1Mbps up. #!/bin/sh ## Setup some vars to use local portage tree export PATH=${HOME}/euscan/bin:${PATH} export PYTHONPATH=${HOME}/euscan/pym:${PYTHONPATH} export PORTAGE_CONFIGROOT=${HOME}/local export ROOT=${HOME}/local export EIX_CACHEFILE=${HOME}/local/var/cache/eix ## Go to euscanwww dir cd ${HOME}/euscan/euscanwww/ ## Update local trees ## Bottleneck: disk and network bandwidth ## Time: less than 30mn emerge --sync --root=${ROOT} --config-root=${PORTAGE_CONFIGROOT} ROOT="/" layman -S --config=${ROOT}/etc/layman/layman.cfg ## Also update eix database, because we use eix internaly ## Bottleneck: disk and cpu ##Time: 30mn ~ 1h eix-update ## Scan portage (packages, versions) ## Bottleneck: disk and cpu ## Time: < 15mn ## Note: this script uses eix to get a list of packages and versions python scan-portage --all --purge-versions --purge-packages ## Scan metadata (herds, maintainers, homepages, ...) ## Bottleneck: disk ## Time: 1h ~ 1h30 ## Note: this script uses gentoolkit to fetch metadata python scan-metadata --all --progress ## Scan uptsream packages ## Bottleneck: disk, network bandwidth and latency, cpu ## Time: up to 6h ## Note: euscan is called on each package. euscan has a slow startup caused by gentoolkit/portage. ## gparallel is used here to limit the load caused by euscan, and to launch up to 16 euscan instances at a time on this machine ## this part is the longest, but scale very well eix --only-names -x | gparallel --load 4 --jobs 800% euscan >> ${HOME}/logs/euscan-upstream.log python scan-upstream --feed --purge-versions < ${HOME}/logs/euscan-upstream.log ## Update counters (6) ## Time: some minutes ## Bottleneck: cpu ## Note: this script could probably be implemented faster using raw SQL queries python update-counters
> Also not sure how much it has to do > with QA. > How much of it constitutes the backend, in your opinion? It seems > there are two parts, right now: > > 1. euscan script, to find new versions for a single package > 2. the django www app, including storage for the version data
Yes, exactly. Here is how the tree is structured currently: euscan script bin/ -- contains the euscan python "binary" pym/ -- contains most of the code used by the euscan script pym/euscan/handlers -- contains specific site handlers (rubygems, pypi, pecl, pear, ..) euscanwww django app euscanwww/ -- contains all the stuff for the django application, all the django application needs is a working portage tree and euscan available in the $PATH
> IMO it would be nice to have a somewhat generic REST-style service > exposing the data, and build a simple UI on top of that. In > particular, I have different ideas about what the UI should look like, > so it would be nice if different people could experiment (and/or > integrate in other services like
I already added some very dummy json formating (note that it also exposes internal key id, which is bad, but I just wanted to experiment). All you need is to append "/json" to an url. For example: - - This could be a lot better, we just need to define an API and the implementation will be easy. A first step would be to make an ebuild for euscan, and another for euscanwww so that anyone can easilly install it and play with it. Feel free to ping me on irc, I'm on #gentoo-sunrise, my nickname is "iksaif". -- Corentin Chary


Subject Author
Re: [gentoo-dev] euscan proof of concept (like debian's uscan) "Michał Górny" <mgorny@g.o>