Re: [gentoo-dev] Re: Bug #565566: Why is it still not fixed?

[Robin H. Johnson]

On Sat, Feb 27, 2016 at 02:14:12PM +0100, Luca Barbato wrote:
> On 24/02/16 01:33, Duncan wrote:
> > That option is there, and indeed, a patch providing it was specifically
> > added to portage for infra to use, because appending entries to existing
> > files is vastly easier and more performant than trying to prepend entries
> > and having to rewrite the entire file as a result.
> This sounds wrong in many different ways. The changelog files are tiny
> and makes next to no difference truncate+write or append.
Prior to seperating ChangeLog files into years, this was way worse:
a kernel bump present in any of gentoo-sources, hardened-sources,
vanilla-sources meant another 100k of data to sent. It's not a lot
overall, but here's some quick stats from one of our rsync servers, on
bytes sent.

Stats for Feb 25, from one of the 3 primary rsync.g.o servers, on the
'bytes sent' output from rsyncd.

rsyncd example output:
Feb 25 00:03:17 quetzal rsyncd[27280]: sent 4930260 bytes  received 32215 bytes  total size 408174052

3909 entries.

Min RAW size: 4833709 bytes [1]
Median RAW size: 22436094 bytes.
Mean RAW size: 45652781 bytes.
Sum of RAW size: 178456721459 bytes = ~166GiB (per day!)

The min possible transfer size is forcing an rsync with no changes; it
just sends the metadata about the files (path, mtime, size, etc).

Let's subtract that from all the rest of the entries, to get stats about
the data transfer.

Median data size: 17602385 bytes
Mean data size: 40819072 bytes

So, now the question:
If we use appending changelogs, the large changelogs only differ by a
few hundred bytes. If we instead have to rewrite them, it's 50k+ per
changelog.

For each 50k changelog, the median transfer would get 0.25% larger.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@g.o
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85