| 1 |
On Wed, Oct 13, 2004 at 09:01:06AM +0200, Spider wrote: |
| 2 |
> > For real benefits, reducing the number of files, or using a filesystem |
| 3 |
> > that performs tail packing reduces the amount of disk seek that must |
| 4 |
> > be done, really increases performance given the number of small files. |
| 5 |
This is still applicable to your method as well. |
| 6 |
|
| 7 |
The one thing that your (previously known) method does bring out is that |
| 8 |
reducing the I/O required really helps. |
| 9 |
|
| 10 |
> Well, here's another method ;) |
| 11 |
> |
| 12 |
> /root/portage.img on /usr/portage type ext2 (rw,noatime,loop=/dev/loop0) |
| 13 |
> -rw-r--r-- 1 root root 293M Oct 12 23:17 /root/portage.img |
| 14 |
> /root/portage.img 257M 195M 62M 77% /usr/portage |
| 15 |
> |
| 16 |
> |
| 17 |
> some varied interesting things from tune2fs -l |
| 18 |
> Filesystem features: dir_index sparse_super |
| 19 |
> Inode count: 300144 |
| 20 |
> Block count: 300000 |
| 21 |
> Free blocks: 62825 |
| 22 |
> Free inodes: 154512 |
| 23 |
> Block size: 1024 |
| 24 |
> Fragment size: 1024 |
| 25 |
Pack it into a loopback reiserfs instead, way better performance. For |
| 26 |
an even bigger boost, put the loop file into tmpfs or use some other |
| 27 |
direct memory scheme. |
| 28 |
|
| 29 |
See: |
| 30 |
http://dev.gentoo.org/~robbat2/fastcvstest |
| 31 |
|
| 32 |
I developed the above when I was working on super-fast CVS repositories, |
| 33 |
as I needed my client to not be the bottleneck ;-). My record for a |
| 34 |
complete CVS checkout of gentoo-x86 (over the network to a remote |
| 35 |
client), stands at 65 seconds. This is quite a bit more work than an |
| 36 |
rsync checkout as well. |
| 37 |
|
| 38 |
Provided you can assure only a single client is using the loopback |
| 39 |
system, here is a very good way of keeping it fast, but not needing the |
| 40 |
network traffic of a full checkout: |
| 41 |
portage loop file is usually on disk, when a sync is needed: |
| 42 |
1. umount loop file |
| 43 |
2. copy loop file to /dev/shm or other fast place |
| 44 |
3. mount loop file again (from new location) |
| 45 |
4. run updates to loop filesystem ('cvs up; emerge metadata' or 'emerge sync') |
| 46 |
5. umount loop file, copy back to disk |
| 47 |
6. mount loop file again |
| 48 |
|
| 49 |
The optimal reiserfs mount options are approximately: |
| 50 |
noexec,nosuid,nodev,noatime,nodiratime,nolog |
| 51 |
|
| 52 |
Your performance may vary with nolog, I use it for the workload of the |
| 53 |
CVS server tmpdir, which is a very frequent creation of 50,000 tiny |
| 54 |
files [for every checkout/update]. |
| 55 |
|
| 56 |
Solar has been doing work on putting the contents of the tree into a |
| 57 |
read-only squashfs filesystem and distributing that. |
| 58 |
|
| 59 |
-- |
| 60 |
Robin Hugh Johnson |
| 61 |
E-Mail : robbat2@××××××××××××××.net |
| 62 |
Home Page : http://www.orbis-terrarum.net/?l=people.robbat2 |
| 63 |
ICQ# : 30269588 or 41961639 |
| 64 |
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 |
Archives