Gentoo Archives: gentoo-dev

From: "Michał Górny" <mgorny@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas
Date: Fri, 17 Jan 2014 16:27:48
Message-Id: 20140117172730.0c504246@pomiot.lan
Hello, all.

I'm using squashfs to hold my Gentoo repositories on all of my systems
for some time. As you probably know, this allows me to save space while
keeping portage fast. However, it makes updating the tree quite
burdensome and time-consuming.

We're already hosting daily gx86 tarballs on our mirrors, and deltas
made using diffball. Those can be used with Zac's emerge-delta-webrsync
to get daily updates done with minimal network overhead. Sadly, it
takes the whole process even more time consuming :).

Therefore, I'd like to suggest an alternative solution that could help
out Gentoo users that use squashfs for gx86 and would like to be able
to get daily updates fast and easy.

The idea is to host -- along with the tarballs -- daily squashfs images
of gx86 in a chosen format. Additionally, the images would come with
deltas made using xdelta3 or a similar tool. Those deltas -- with
a slight download overhead -- would allow very fast updates
of the squashfs.

Now some numbers. I did some tests 'converting' late gx86 daily
tarballs to squashfs. I've used squashfs 4.2 with LZO compression
since it's quite good and very fast.

96M	portage-20140108.sqfs
96M	portage-20140109.sqfs
96M	portage-20140110.sqfs
96M	portage-20140111.sqfs
96M	portage-20140112.sqfs
96M	portage-20140113.sqfs
97M	portage-20140114.sqfs
97M	portage-20140115.sqfs

For deltas, I've used xdelta3 with max compression (-9) and djw
secondary compression (it gave ~0.1M smaller files than fgk
and ~0.5M gain than with no secondary compression).

4,9M	portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
6,3M	portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
5,6M	portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
8,9M	portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
6,3M	portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
7,8M	portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
8,5M	portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw

As you can see, the deltas are quite large compared to the actual
changes. However, we could have expected that since we're diffing
a compressed filesystem. What's important, however, is that applying
it takes ~2.5 second on my 2 GHz Athlon64.

So, even with the extra download time, the update is much faster
than recreating the squashfs. And unlike some types of unionfs,
it doesn't come with extra runtime slowdown.

What do you think?

Best regards,
Michał Górny


File name MIME type
signature.asc application/pgp-signature