Re: [gentoo-project] Mirror request: SquashFS portage images and deltas

[Rick \\\"Zero_Chaos\\\" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/28/2014 07:49 AM, Michał Górny wrote:
> Hello,
>
> (I'm CC-ing mirror admins since this may impact space consumption
>  on mirrors, and gentoo-project to keep it public)
>
> As you may have noticed, I was playing with providing and efficiently
> updating SquashFS images for our users. A few notes of what has been
> decided so far:
>
> 1. the SquashFS images would be LZO-compressed (with LZ4 switch planned
> when the kernels with it are spread enough), since it provides very
> good compression with great decompression speed and low requirements,

I would suggest XZ over both LZO and LZ4 (and even LZ4HC).  Compression
time isn't really a major concern (imho) and the resulting files are
much smaller yet easy to decompress and with (reasonably) low ram
requirement for doing so.

- -Zero

PS>  If for whatever reason you choose to use lz4 I have support for it
in mksquashfs in my overlay and I would be happy to move it into gentoo
if there is a call for it.
>
> 2. the images would be updated through binary deltas made using
> xdelta3.
>
> 3. since the deltas are quite space-inefficient, we won't be downloading
> a few day-to-day patches but instead a single combined patch for each
> supported day back. IOW, we would have:
>
>   squashfs-20140127-20140128.delta
>   squashfs-20140126-20140128.delta
>   squashfs-20140125-20140128.delta
>   squashfs-20140124-20140128.delta
>   (...)
>
> Just to be clear, I don't want to replace the current snapshot+patch
> since it's more space-efficient. However, I'd like to provide
> an alternative for users for whom bandwidth is not a concern, but
> memory and/or time is.
>
> More details can be found in my article on SquashFS deltas [1].
>
>
> I'd like to raise the following questions:
>
> 1. Is infra@ ok with building the snapshots & deltas?
>
> 2. Do our mirror admins have any concerns wrt this?
>
> 3. How many days old snapshots do we want to support? That is, after
> how many days of not updating the user is supposed to download
> the whole new image rather than a patch?
>
>
> Now for the numbers:
>
> 1) build server use:
>
>   a) disk space:
>
>      ~100M for each snapshot (we need to keep the past snapshots
>        for delta updates)
>
>      6M-20M for early deltas, up to 40M for month-old delta
>
>      likely no extra space requirements if we hack the update in .tar
>      generation
>
>      for 30 days backlog, I suspect around 3G for snapshots
>      + 1G for deltas
>
>   b) memory:
>
>      no noticeable impact
>
>   c) CPU:
>
>      LZO compression has relatively small impact
>
>      around 7s heavy CPU use for xdelta3 (on i3 1.8 GHz)
>
>   d) time:
>
>      about 1 minute to build SquashFS
>
>      for delta gen: ceil(n / c) * 7 seconds,
>        where n = number of days, c = number of CPUs
>
>   e) uplink bandwidth:
>
>      mirrors will have to download all the deltas during sync,
>      e.g. 1G every day
>
> 2) mirror use:
>
>   a) disk space:
>
>      ~100M for the snapshot (it's enough to mirror the latest one)
>
>      6M-20M for early deltas, up to 40M for month-old delta
>
>      for 30 days of backlog, I suspect around 1G
>
>   b) uplink bandwidth:
>
>      every switching user will download ~100M initial image
>
>      users who sync daily will dl 6-10M every day
>
>      users who sync often will dl 20-40M a week
>
>      users who sync rarely will dl 40M-80M a month
>
>   c) downlink bandwidth:
>
>      depending on the no of days supported, for 30 days ~1G every day
>
> 3) client impact
>
>   a) disk space:
>
>      ~100M for current SquashFS (that is used by the package manager)
>
>      during update, 10M-40M for the delta (depending on how outdated
>      the image is)
>
>   b) downlink bandwidth:
>
>      10M-40M for the delta (for each update)
>
>   c) memory & CPU:
>
>      negligible
>
>   d) time:
>
>      fetching time + around 5 s for the update
>
>
> [1]:http://dev.gentoo.org/~mgorny/articles/using-deltas-to-speed-up-squashfs-ebuild-repository-updates.pdf
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJS6BR9AAoJEKXdFCfdEflKO84P/jffUBF5OcVNNNfpmPRuE36J
x5voPKDRWxJ5WwRB4cBt9yWKxqegrTIiFQV9v0i19zJR5JVsdRDb5zOGeDwOPZW/
x7pmDTM5WVoFefmVPBeENAjCMmxikenMBGiDjkkyYInbsIw87tThVFMJTm89hXdi
gvDQW5rgfc8eaBhRZH9V5WGEE/vvionCr20oMv+DbDlUNQxJ/+Ldf5v1wbejJEey
wwZBDbP/NwDsxn6G3ymya4+o/e89YaTq76Yk3Oaacm26Jq/oUWwMio90lkQPmKAp
LGYOKjjE9bVgJomklWgSKh2RKKNNPXy2KjTUl6TtRteVhFMYkSlYSNonUgoT6/iC
FlsjUQclIGtwRbPPUeveOD6jxhWHpXfFYd3lQL8NU0948hq7FTZSnS3zzEU2sw0Y
bCRS3pJNfOZbHgDr2Pv4BhG2wBCji7oudhQFd9fn78DrwNk32/wKP7GvUzEvdgWr
1R0JRJNcZVTbxiRiVFpox2+pxHdc7vMH4IHAsNJ2SG7BTJKZEduco0zAR50igNd8
+tgM2SQh5eht4/tR/1syyXfjYAq+rZTHklhSt6nQ1n+OlEaPnNnt6Q8TJlrMGBeR
ZNpiggnCUPq8Vtbaj6WqLsPZuV6lOsRPt/mh9r5QN3JYsjlpxaMlKm4tOiOhABZJ
TUyYNn7VB6CgAO9vQG/k
=fLkA
-----END PGP SIGNATURE-----