[gentoo-mirrors] Distfile mirror layout switch; please enable --hard-links

[Michał Górny]

Hi,

TL;DR: we've started switching to the new distfile layout.  Mirrors will
notice the presence of symlinks corresponding to the dual-layout
throughout the transitional period.  At some point in the future we will
have to switch the direction of symlinks, so please enable --hard-links
(-H) in your rsync calls if you haven't already.


I am pleased to announce that after initial internal testing, we've
successfully deployed the new distfile mirror layout.  The new layout
splits distfiles into 256 directories, solving the problem of having
60000+ files in a single directory.  However, the mirrors will really
start benefiting from this only once the old layout is gone.

To reduce the impact on mirrors to the bare minimum, we've chosen to go
with a hybrid symlink+hardlink approach.  Throughout the majority of
transitional period, the compatibility between two layouts will be
provided by symlinks.  This means the total number of file entries
on the mirror will double but the additional bandwidth and disk space
use will be limited to the possible minimum.

All newly downloaded files already use the new layout for regular files,
and create symlinks in the old layout.  However, existing files are
symlinked the other way.  Before the old layout is removed, we will have
to reverse the direction of those symlinks.

For the time being, we're just going to let normal distfile rotation
(i.e. fetching of new files and removal of old files) take care of
reducing the number of links needing transition.  Nevertheless, at some
point in the future we will have to start converting the remaining
files.

We will be using hardlinks to make it possible for the conversion to
occur without refetching everything.  Therefore, once again we urge you
to enable the --hard-links (-H) option to rsync if you haven't done that
already.  According to my current plan, we will use a shifting
transition window approach to limit the impact both on mirrors
and on master mirror.

If you have any questions, please do not hesitate to ask.

-- 
Best regards,
Michał Górny