Re: [gentoo-dev] ChangeLog - Infra Response; update 2015/11/11, potential impact to 30min rsync cycle

[Robin H. Johnson]

On Thu, Nov 05, 2015 at 12:54:06PM +0100, Alexis Ballier wrote:
> It's not perfectly clean but I don't see any problem here:
> ChangeLog-2015 : all ChangeLog from CVS
> ChangeLog: autogenerated from git
FYI, this was implemented.

For reference, the old CVS changelogs are now taken from HEAD of this
repo:
https://gitweb.gentoo.org/data/gentoo-changelogs.git/

mgorny and I have been poking at the generation issue, with the features
I requested now implemented, plus one patch I pushed up to portage-dev.

There are still some issues remaining.

I filed bugs for some of them:
565536 - need to exclude some commits/paths
565538 - need to exclude some lines
565540 - need parallel threads

However, the largest sticking point, even with parallel threads, is that
it seems the base ChangeLog generation is incredibly slow. It averages
above 350ms per package right now (at 19k packages in a full cycle, it's
a long time), but some packages can take up to 5 seconds so far.

Incremental processing does help this hugely, but isn't always
available.

Right now, I'm considering promising 30 minute syncs as a best case
interval; if changelog generation causes it to take longer, then a push
window WILL be missed. 

How often might this happen? Since we converted to Git, excluding the
initial large commits, there were three instances where it would have
added more than 10 minutes without the improvements I created bugs for.
Plus, any other changes that cause loss of timestamps/reference for
comparison will trigger a full run, at ~6 hours of delay.

(Yes, that's why there hasn't been an rsync update in the last 3 hours,
and won't be for another ~3 hours: because it's crunching to generate
ChangeLogs).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail     : robbat2@g.o
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85