Re: [gentoo-dev] Re: Gentoo Janitor scripts

[Corentin Chary]

On Wed, Feb 22, 2012 at 1:20 PM, Mart Raudsepp <leio@g.o> wrote:
> On K, 2012-02-22 at 09:48 +0100, Corentin Chary wrote:
>> I did a quick script to count most used prefixes in SRC_URI yesterday
>> (https://github.com/iksaif/portage-janitor/blob/master/mirrors.py)
>>
>> Here is the (filtered) result:
>>
>> $ eix --only-names | python mirrors.py --count
>> 960     http://dev.gentoo.org
>> 372     http://xorg.freedesktop.org
>> 372     http://xorg.freedesktop.org/releases
>> 372     http://xorg.freedesktop.org/releases/individual
>> 306     http://pear.php.net
>> 306     http://pear.php.net/get
>> 256     http://oss.tresys.com
>> 255     http://oss.tresys.com/files
>> 255     http://oss.tresys.com/files/refpolicy
>> 225     http://hackage.haskell.org/packages
>> 225     http://hackage.haskell.org/packages/archive
>> 225     http://hackage.haskell.org
>> 206     http://ftp.xemacs.org
>> 201     https://github.com
>> 196     http://ftp.xemacs.org/pub
>> 196     http://ftp.xemacs.org/pub/xemacs
>> 193     http://ftp.xemacs.org/pub/xemacs/packages
>> 181     http://gstreamer.freedesktop.org
>> 181     http://gstreamer.freedesktop.org/src
>> 175     http://launchpad.net
>> 175     http://linuxgazette.net
>> 143     http://github.com
>> 130     http://pear.horde.org
>> 130     http://pear.horde.org/get
>> 101     http://savannah.nongnu.org/download
>> 101     http://savannah.nongnu.org
>> 100     http://get.qt.nokia.com
>> 97      ftp://sources.redhat.com/pub
>> 97      ftp://sources.redhat.com
>> 96      http://get.qt.nokia.com/qt
>> 95      http://get.qt.nokia.com/qt/source
>> 90      http://download.gna.org
>> 75      http://pecl.php.net
>> 75      http://pecl.php.net/get
>> 72      http://components.ez.no/get
>> 72      http://components.ez.no
>> 69      https://fedorahosted.org
>> 67      http://www.phrack.org/archives
>> 67      http://www.phrack.org/archives/tgz
>> 67      http://www.phrack.org
>>
>>
>> From that output we can easilly find out new entries to
>> thirdpartymirrors, for example:
>> gentoo-dev    http://dev.gentoo.org
>> xorg             http://xorg.freedesktop.org
>> gna              http://download.gna.org
>> pecl             http://pecl.php.net
>> pear             http://pear.php.net
>> github          https://github.com http://github.com
>> xemacs       http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
>> launchpad    http://launchpad.net
>> redhat         ftp://sources.redhat.com/pub/ (and probably others !)
>> etc...
>>
>> The good part is that once you've modified thirdpartymirrors with new
>> mirrors, running mirrors.py --all will generate a big patch for all
>> your ebuilds to use those new mirrors !
>
> If you want this, then you should better figure out actual upstream
> mirroring systems and their list of mirrors they would want us to use.
> Until such, this seems to be just for shortening SRC_URI addresses when
> an upstream tarball domain name or path repeats, and that's definitely
> not what thirdpartymirrors is for.

Yes, of course, that was just a quick example, not something definitive.

But lets some examples:
- http://xorg.freedesktop.org, it's easy to find mirror for that one,
http://ftp.x.org/pub/ for example
- github: packages seems to use http and https, this script can help
to standardize the url used
-  "960     http://dev.gentoo.org": that's a lot of package hosted
here, is that really right ?

And still, thirdpartymirrors have some entries with only one "mirror"
and I believe factorizing SRC_URIs is a good thing (if something
changes, you just patch thirdpartymirros, not hundreds of ebuilds).

-- 
Corentin Chary
http://xf.iksaif.net