[gentoo-project] Mirror request: SquashFS portage images and deltas

[Michał Górny]

Hello,

(I'm CC-ing mirror admins since this may impact space consumption
 on mirrors, and gentoo-project to keep it public)

As you may have noticed, I was playing with providing and efficiently
updating SquashFS images for our users. A few notes of what has been
decided so far:

1. the SquashFS images would be LZO-compressed (with LZ4 switch planned
when the kernels with it are spread enough), since it provides very
good compression with great decompression speed and low requirements,

2. the images would be updated through binary deltas made using
xdelta3.

3. since the deltas are quite space-inefficient, we won't be downloading
a few day-to-day patches but instead a single combined patch for each
supported day back. IOW, we would have:

  squashfs-20140127-20140128.delta
  squashfs-20140126-20140128.delta
  squashfs-20140125-20140128.delta
  squashfs-20140124-20140128.delta
  (...)

Just to be clear, I don't want to replace the current snapshot+patch
since it's more space-efficient. However, I'd like to provide
an alternative for users for whom bandwidth is not a concern, but
memory and/or time is.

More details can be found in my article on SquashFS deltas [1].


I'd like to raise the following questions:

1. Is infra@ ok with building the snapshots & deltas?

2. Do our mirror admins have any concerns wrt this?

3. How many days old snapshots do we want to support? That is, after
how many days of not updating the user is supposed to download
the whole new image rather than a patch?


Now for the numbers:

1) build server use:

  a) disk space:

     ~100M for each snapshot (we need to keep the past snapshots
       for delta updates)

     6M-20M for early deltas, up to 40M for month-old delta

     likely no extra space requirements if we hack the update in .tar
     generation

     for 30 days backlog, I suspect around 3G for snapshots
     + 1G for deltas

  b) memory:

     no noticeable impact

  c) CPU:

     LZO compression has relatively small impact

     around 7s heavy CPU use for xdelta3 (on i3 1.8 GHz)

  d) time:

     about 1 minute to build SquashFS

     for delta gen: ceil(n / c) * 7 seconds,
       where n = number of days, c = number of CPUs

  e) uplink bandwidth:

     mirrors will have to download all the deltas during sync,
     e.g. 1G every day

2) mirror use:

  a) disk space:

     ~100M for the snapshot (it's enough to mirror the latest one)

     6M-20M for early deltas, up to 40M for month-old delta

     for 30 days of backlog, I suspect around 1G

  b) uplink bandwidth:

     every switching user will download ~100M initial image

     users who sync daily will dl 6-10M every day

     users who sync often will dl 20-40M a week

     users who sync rarely will dl 40M-80M a month

  c) downlink bandwidth:

     depending on the no of days supported, for 30 days ~1G every day

3) client impact

  a) disk space:

     ~100M for current SquashFS (that is used by the package manager)

     during update, 10M-40M for the delta (depending on how outdated
     the image is)

  b) downlink bandwidth:

     10M-40M for the delta (for each update)

  c) memory & CPU:

     negligible

  d) time:

     fetching time + around 5 s for the update


[1]:http://dev.gentoo.org/~mgorny/articles/using-deltas-to-speed-up-squashfs-ebuild-repository-updates.pdf

-- 
Best regards,
Michał Górny