Gentoo Archives: gentoo-dev

From: "Michał Górny" <mgorny@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas
Date: Fri, 17 Jan 2014 16:27:48
Message-Id: 20140117172730.0c504246@pomiot.lan
1 Hello, all.
3 I'm using squashfs to hold my Gentoo repositories on all of my systems
4 for some time. As you probably know, this allows me to save space while
5 keeping portage fast. However, it makes updating the tree quite
6 burdensome and time-consuming.
8 We're already hosting daily gx86 tarballs on our mirrors, and deltas
9 made using diffball. Those can be used with Zac's emerge-delta-webrsync
10 to get daily updates done with minimal network overhead. Sadly, it
11 takes the whole process even more time consuming :).
13 Therefore, I'd like to suggest an alternative solution that could help
14 out Gentoo users that use squashfs for gx86 and would like to be able
15 to get daily updates fast and easy.
17 The idea is to host -- along with the tarballs -- daily squashfs images
18 of gx86 in a chosen format. Additionally, the images would come with
19 deltas made using xdelta3 or a similar tool. Those deltas -- with
20 a slight download overhead -- would allow very fast updates
21 of the squashfs.
23 Now some numbers. I did some tests 'converting' late gx86 daily
24 tarballs to squashfs. I've used squashfs 4.2 with LZO compression
25 since it's quite good and very fast.
27 96M portage-20140108.sqfs
28 96M portage-20140109.sqfs
29 96M portage-20140110.sqfs
30 96M portage-20140111.sqfs
31 96M portage-20140112.sqfs
32 96M portage-20140113.sqfs
33 97M portage-20140114.sqfs
34 97M portage-20140115.sqfs
36 For deltas, I've used xdelta3 with max compression (-9) and djw
37 secondary compression (it gave ~0.1M smaller files than fgk
38 and ~0.5M gain than with no secondary compression).
40 4,9M portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
41 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
42 5,6M portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
43 8,9M portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
44 6,3M portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
45 7,8M portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
46 8,5M portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
48 As you can see, the deltas are quite large compared to the actual
49 changes. However, we could have expected that since we're diffing
50 a compressed filesystem. What's important, however, is that applying
51 it takes ~2.5 second on my 2 GHz Athlon64.
53 So, even with the extra download time, the update is much faster
54 than recreating the squashfs. And unlike some types of unionfs,
55 it doesn't come with extra runtime slowdown.
57 What do you think?
59 --
60 Best regards,
61 Michał Górny


File name MIME type
signature.asc application/pgp-signature