Re: [gentoo-user] Re: What is the meaning of "mirror" URI scheme in ebuilds?

[David Haller]

Hello,

On Tue, 27 Mar 2018, Ian Zimmerman wrote:
>On 2018-03-27 22:00, David Haller wrote:
>
>> So, using your example, portage looks up the mirror for mirror://gnu
>> by looking in /etc/portage/mirrors (if it exists) and
>> /usr/portage/profiles/thirdpartymirrors for a line with the word "gnu"
>> at the beginning, and then tries the mirrors specified there. I.e.:
>
>Thanks, this helps.  I have a related follow-up question.
>
>Sometimes, the SRC_URI in fact consists of multiple URIs.  What is
>confusing about that is that in some cases the individual URI clearly
>refer to copies of the same file (I'm guessing this provides redundant
>locations where to download it from),

Yes.

>while in other cases they're clearly different files (for example
>upstream patches to be downloaded separately). There are even ebuilds
>when _both_ cases happen at the same time, such as
>media-libs/freetype-2.9 :
>
>SRC_URI="mirror://sourceforge/freetype/${P/_/}.tar.bz2
>        mirror://nongnu/freetype/${P/_/}.tar.bz2
>        utils?  ( mirror://sourceforge/freetype/ft2demos-${PV}.tar.bz2
>                mirror://nongnu/freetype/ft2demos-${PV}.tar.bz2 )
>        doc?    ( mirror://sourceforge/freetype/${PN}-doc-${PV}.tar.bz2
>                mirror://nongnu/freetype/${PN}-doc-${PV}.tar.bz2 )"
>
>So, WTH is the semantics of this?  How does portage know which URIs are
>for the main tarball and which are the extras?  It has to know that to
>avoid downloading the main tarball twice.

I, too, am not sure how that's handled in portage, much less what the
spec (PMS) say about it, but practical observation leads me to
believe, that for each $(basename $URI) (i.e. each file), portage
tries to get them from the URIs specified, in this case, it tries to
get the freetype ${P/_/}.tar.bz2 tarball from a mirror specified in
the sourceforge or nongnu lists of /etc/portage/mirrors resp. 
${PORTAGE_DIR}/profiles/thirdpartymirrors and then the ft2demos and
-doc- tarballs from (in this case) the same mirror-lists.

Basically, it seems portage build a list of (think '$(basename $URI)')

    file_to_get -> mirrorlist_or_direct_URI

i.e. here:

    freetype-${VER}.tar.bz2     -> URI_LIST
    ft2demos-${VER}.tar.bz2     -> URI_LIST
    freetype-doc-${VER}.tar.bz2 -> URI_LIST

and for each of those it tries the specified URIs (here
mirror://sourceforge and mirror://nongnu mirrorlists). You quite often
also see a "direct" non-mirror:// URI at the end (as a "last resort"
or so).

Anyway, that's my take on it.

And yes (referring to the bug mentioned by Michael in his answer),
quite some mirrors in the (current) thirdpartymirrors list are
outdated, which is why I copied that to /etc/portage/mirrors and
massaged those mirrorlists I noticed "broken" or "slow", either adding
my favored mirrors up front or replacing the complete list with some
chosen "known-good" mirrors.

But for the most-used mirror:// URIs in ebuilds I've come across,
besides the core gentoo list, it's mainly sourceforge or gnu or so. 
For the core "gentoo" mirrors, you can manage that mirrorlist via the
GENTOO_MIRRORS="" variable in make.conf, manually or via mirrorselect.

For me, it's rather easy, there's some "big", reliable "local" mirrors
(mostly at universities and related institutions) that mirror most
opensource software, from core projects (kernel, gnu) and various
projects (X, CTAN, CPAN) to various distros (SuSE, Gentoo, Debian, and
whatnot)... Weirdly enough, neither "thirdpartymirrors" nor
mirrorselect nor https://www.gentoo.org/downloads/mirrors/#DE list the
biggest mirror for gentoo-portage and -distfiles in .de, probably .eu ;)

*adding that one to my mirror-list*

HTH,
-dnh

-- 
Ceci n'est pas une .signature.