Re: [gentoo-dev] New distfile mirror layout

[Michał Górny]

On Tue, 2019-10-29 at 15:33 +0100, Fabian Groffen wrote:
> On 29-10-2019 15:17:38 +0100, Ulrich Mueller wrote:
> > > > > > > On Tue, 29 Oct 2019, Michał Górny wrote:
> > > On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote:
> > > > > What if the file is hosted at a non-standard tcp port upstream
> > > > > (like http://example.org:8080/)? The devmanual says that it _must_
> > > > > be manually uploaded to /space/distfiles-local/ in such cases.
> > > > Or another example, app-emacs/vhdl-mode-3.38.1, where (incompetent,
> > > > or nasty?) upstream blocks wget for some reason, but other methods
> > > > (e.g., curl, firefox) work? How would I get the file onto the mirrors
> > > > there?
> > > If I were you, I would've explicitly mirrored the file anyway.
> > > If upstream blocks wget, then users who do not use GENTOO_MIRRORS will
> > > also suffer due to it.
> >
> > All what I'm saying is that there can be unusual circumstances where
> > manual uploading of a file is useful. So please don't take that
> > possibility away.
>
> In addition, there are currently files there that aren't referenced from
> ebuilds.  Prefix uses these files during bootstrap, local mirrors are
> often much faster than dev.g.o.
>
> If the files don't get mirrored anymore, I guess I can create a dummy
> ebuild that has the files in SRC_URI.

Ok, this is something I wasn't aware of.  I agree that dummy ebuild
should not be necessary here.  However, I'm also not sure if distfiles-
local is really the proper way either, especially that I don't see such
files on woodpecker right now.

I don't think the matter is urgent right now, so let's ponder on it
a bit.  In particular, I think we should have a clear indication of who
added which files, when, what for and where they came from.  Those are
precisely the things that the current distfiles-local approach misses.

> If the files get mirrored, but put in a subdir based on the filename
> hash, the original query endpoint on distfiles.g.o changes, much like
> the SRC_URI approach.
>
> Now I can use distfiles.prefix.b.n which redirects to the distfiles.g.o
> URL with subdir for most part I think, but it's sub-optimal from my
> point of view.  Calculating the hash is not always feasible due to the
> lack of b2sum or other means.  Hence my earlier request to have such
> official translation service on Gentoo hardware.
>
> (I just wrote a small wsgi script that calculates the hash and generates
> the redirect from Python, served via uwsgi/nginx, but there should be
> many ways to achieve the same goals, if and only if a blake2b
> implementation were available for it.)
>

This is also something that needs thinking.  I personally don't mind
having one but it would be nice if it was able to account for geodns
and such.

-- 
Best regards,
Michał Górny