| 1 |
Hi, |
| 2 |
|
| 3 |
as suggested by Mike in http://bugs.gentoo.org/show_bug.cgi?id=123335, |
| 4 |
here's my proposal for changing the layout of the distfiles tree: |
| 5 |
|
| 6 |
|
| 7 |
This is the current state: |
| 8 |
|
| 9 |
mirror:/storage/gentoo/data/source/distfiles# ls | wc -l |
| 10 |
22543 |
| 11 |
mirror:/storage/gentoo/data/source/distfiles# ls -l ../ | grep distfiles |
| 12 |
drwxr-xr-x 3 gentoo gentoo 950272 Mar 6 06:08 distfiles |
| 13 |
mirror:/storage/gentoo/data/source/distfiles# |
| 14 |
|
| 15 |
|
| 16 |
People who want to browse the files "by hand" are usually stumped by the |
| 17 |
large output (the directory listing lighttpd creates is currently 4.2MB |
| 18 |
in size) and generation of those listings causes an excessive strain on |
| 19 |
the server. |
| 20 |
|
| 21 |
Plus the creation/deletion of files doesn't scale too well on |
| 22 |
filesystems, which store directory entries in linked lists (ext2/3, |
| 23 |
probably the common bsd filesystems), since the list has to be traversed |
| 24 |
for each file deleted/created. |
| 25 |
|
| 26 |
Introducing an additional directory hierarchy should fix this, and is |
| 27 |
the common solution for this problem for various projects, be it debian |
| 28 |
[1], cpan [2], slackware [3], etc. |
| 29 |
|
| 30 |
|
| 31 |
One migration scenario for a better future: |
| 32 |
|
| 33 |
Create subdirectories named after the first letter of each file and move |
| 34 |
the files in their respective directories. |
| 35 |
|
| 36 |
Either sym- or hardlink the files from the current distfiles |
| 37 |
root-directory to the specific directory where they reside in. (Check |
| 38 |
with the mirror admins first (depending on the chosen linktype) if rsync |
| 39 |
hardlink support is enabled or their web/ftp servers allow/follow symlinks) |
| 40 |
|
| 41 |
Adapt the build scripts so that they look for the files in their new |
| 42 |
location. |
| 43 |
|
| 44 |
Change the scripts which fetch the files for distfiles so that they save |
| 45 |
them under the new location. |
| 46 |
|
| 47 |
Wait a few weeks... (months? years? decades?) until the last user has |
| 48 |
updated and/or a clean upgrade-path exists, which doesn't rely on the |
| 49 |
old file locations. |
| 50 |
|
| 51 |
Drop the sym/hardlinks. |
| 52 |
|
| 53 |
|
| 54 |
After the change we'd have (with the current set of files) 63 |
| 55 |
subdirectories, the largest one containing 1775 files (letter 'g'), |
| 56 |
which is a definitive improvement over the current situation. |
| 57 |
|
| 58 |
Full list can be seen at http://mirror.inode.at/gentoo-listing.txt . |
| 59 |
|
| 60 |
|
| 61 |
best regards, |
| 62 |
Michael Renner - admin of gentoo.inode.at/rsync1.at.gentoo.org |
| 63 |
|
| 64 |
[1] http://debian.inode.at/debian/pool/main/ |
| 65 |
[2] http://www.slackware.at/data/slackware/slackware/ |
| 66 |
[3] http://cpan.inode.at/modules/by-authors/id/ |
| 67 |
-- |
| 68 |
gentoo-dev@g.o mailing list |
Archives