| 1 |
Proposal, create snapshots of portage as squashfs iso, to be used in place |
| 2 |
of /usr/portage directory. |
| 3 |
prior art: see #1 |
| 4 |
|
| 5 |
Already working here: one central server which keep the portage full tree |
| 6 |
and that after a sync create "portage.sqsh" a squashfs 4.0 iso. |
| 7 |
|
| 8 |
Advantages are mainly: |
| 9 |
- cleaner root directory (ext4: du -sh /usr/portage ~= 600M | find |
| 10 |
/g/portage | wc -l ~= 130000) |
| 11 |
- faster `emerge --sync` with fast connections |
| 12 |
- faster `emerge -uDpvN world` |
| 13 |
- less cpu/disk load on the server (if not serving from memory) |
| 14 |
|
| 15 |
Disadvantages |
| 16 |
- need to mount portage, or to use autofs |
| 17 |
- need kernel >= 2.6.30 |
| 18 |
- bigger rsync transfer size (?= 2x) #2 |
| 19 |
- bigger initial transfer size, lzma snapshot currently weight 30.8M, |
| 20 |
portage.sqsh 45M |
| 21 |
|
| 22 |
How it's done here: |
| 23 |
Currently on the dispatcher the following run after every emerge --sync: |
| 24 |
|
| 25 |
mksquashfs /usr/portage /srv/portage.sqsh \ |
| 26 |
-noappend -no-exports -no-recovery -force-uid 250 -force-gid 250 |
| 27 |
|
| 28 |
The clients run from cron the following: |
| 29 |
umount /g/portage 2>/dev/null \ |
| 30 |
; cp /srv/server/portage.sqsh /var/tmp/portage.sqsh \ |
| 31 |
&& mount /usr/portage |
| 32 |
|
| 33 |
/etc/fstab: |
| 34 |
/srv/server/portage.sqsh /usr/portage squashfs loop,ro,noauto 1 1 |
| 35 |
|
| 36 |
some real data: |
| 37 |
|
| 38 |
stats for a portage sync, one day |
| 39 |
|
| 40 |
Number of files: 136429 |
| 41 |
Number of files transferred: 326 |
| 42 |
Total file size: 180345216 bytes |
| 43 |
Total transferred file size: 1976658 bytes |
| 44 |
Literal data: 1976658 bytes |
| 45 |
Matched data: 0 bytes |
| 46 |
File list size: 3377038 |
| 47 |
File list generation time: 0.001 seconds |
| 48 |
File list transfer time: 0.000 seconds |
| 49 |
Total bytes sent: 47533 |
| 50 |
Total bytes received: 4120255 |
| 51 |
|
| 52 |
sent 47533 bytes received 4120255 bytes 124411.58 bytes/sec |
| 53 |
total size is 180345216 speedup is 43.27 |
| 54 |
|
| 55 |
stats for a portage.sqsh sync, one day |
| 56 |
|
| 57 |
Number of files: 1 |
| 58 |
Number of files transferred: 1 |
| 59 |
Total file size: 46985216 bytes |
| 60 |
Total transferred file size: 46985216 bytes |
| 61 |
Literal data: 8430976 bytes |
| 62 |
Matched data: 38554240 bytes |
| 63 |
File list size: 27 |
| 64 |
File list generation time: 0.001 seconds |
| 65 |
File list transfer time: 0.000 seconds |
| 66 |
Total bytes sent: 48096 |
| 67 |
Total bytes received: 8454837 |
| 68 |
|
| 69 |
sent 48096 bytes received 8454837 bytes 5668622.00 bytes/sec |
| 70 |
total size is 46985216 speedup is 5.53 |
| 71 |
|
| 72 |
|
| 73 |
|
| 74 |
#1 http://forums.gentoo.org/viewtopic-p-2218914.html |
| 75 |
http://www.mail-archive.com/gentoo-dev@g.o/msg05240.html |
| 76 |
|
| 77 |
#2 May be mitigated by mksquashfs '--sort' option, to be tested |
| 78 |
|
| 79 |
- francesco (vivo) |
Archives