Re: [gentoo-dev] New distfile mirror layout

[Michał Górny]

On Sun, 2019-10-20 at 05:21 -0400, Joshua Kinard wrote:
> On 10/20/2019 04:32, Michał Górny wrote:
> > On Sun, 2019-10-20 at 04:25 -0400, Joshua Kinard wrote:
> > > Why is having a max ~24k files in a directory a bad idea?  Modern
> > > filesystems are more than capable of handling that.
> > >
> > >   - ext4: unlimited files in a directory
> > >   - xfs: virtually unlimited (hard limit of 2^64-1 total files per volume)
> > >   - ntfs: 4,294,967,295
> > >
> > > And 24k is a bit more than 1/3rd of all distfiles that we currently have.
> >
> > For the same reason having ~60k files in a directory was a problem.
> > There is really no point in changing anything if you change BIG_NUMBER
> > to SMALLER_BIG_NUMBER.
>
> That doesn't answer my question.  Why is it a problem?  What criteria are
> you using to decide that 24k is a "smaller big number"?  Is there some issue
> highlighted by the mirror admins where having 24k files in a single
> directory offers no significant relief versus the current 60k files?

IIRC Robin set the goal as:

| the number of files in a single directory should not exceed 1000, [1]

I don't recall how that number was chosen but it's probably pretty
arbitrary.  In any case, I can notice the difference between working
with a listing of 1k files and 24k files, on the hardware running
masterdist.

> > > Under which scenario do you wind up with 24k files in a single directory?  I
> > > consider the tex package an outlier in this case (one package should not be
> > > the sole dictator of policy).
> >
> > Three versions of TeXLive living simultaneously.  If one package falls
> > completely out of bounds, no problem is solved by the change, so what's
> > the point of making it?
>
> The problem in this case is with texlive, not our current, or future,
> distfiles methodology.

Is it?  Are you suggesting we should ban upstream from using multiple
distfiles with similar prefix?  What about other potential packages that
may suffer from the same problem in the future?  Go packages have a good
potential, given that majority of them starts with 'github.com'.

> Has anyone looked at how other distros deal with texlive?

Other distros don't mirror original distfiles.

>   Has anyone complained or filed a bug to texlive developers
> upstream about their excessive amount of distfiles and the burden it places
> on distro maintainers?

You believe it to be a problem.  Don't expect others to bother upstream
with your preferences.


[1] https://www.gentoo.org/glep/glep-0075.html#algorithm-for-splitting-distfiles

>
-- 
Best regards,
Michał Górny