Gentoo Archives: gentoo-dev

From: Francesco Riosa <vivo@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] ChangeLogs and rsync time
Date: Sun, 01 Jan 2006 20:38:44
Message-Id: 43B83D16.6060803@gentoo.org
1 We have currently 10371 ChangeLog files, > 25 MB totally .
2 1365 == 13% with size >= 4096 B, 12 MB totally
3
4 rsync from "emerge --sync" has "--whole-file" between its options, that
5 mean transfer the whole file if changed.
6
7 To make things worse the bigger ChangeLogs are (oh, surprise) those that
8 change frequently (also the age count), following there is the list of
9 first 25:
10
11 110062 ./x11-base/xorg-x11/ChangeLog
12 96906 ./sys-devel/gcc/ChangeLog
13 86916 ./sys-libs/glibc/ChangeLog
14 72429 ./net-www/apache/ChangeLog
15 65331 ./sys-apps/baselayout/ChangeLog
16 61801 ./media-video/mplayer/ChangeLog
17 57688 ./dev-db/mysql/ChangeLog
18 53938 ./sys-kernel/gentoo-sources/ChangeLog
19 53810 ./net-im/gaim/ChangeLog
20 53173 ./www-client/mozilla/ChangeLog
21 51891 ./dev-php/mod_php/ChangeLog
22 48127 ./dev-db/postgresql/ChangeLog
23 47014 ./sys-devel/binutils/ChangeLog
24 46742 ./kde-base/kdelibs/ChangeLog
25 45370 ./dev-lang/perl/ChangeLog
26 44998 ./sys-kernel/mm-sources/ChangeLog
27 41010 ./kde-base/kdebase/ChangeLog
28 37644 ./www-client/mozilla-firefox/ChangeLog
29 37524 ./net-fs/samba/ChangeLog
30 36411 ./mail-mta/postfix/ChangeLog
31 35269 ./app-office/openoffice-ximian/ChangeLog
32 34890 ./app-office/openoffice/ChangeLog
33 34773 ./sys-kernel/mips-sources/ChangeLog
34 33245 ./media-sound/xmms/ChangeLog
35 32769 ./dev-util/subversion/ChangeLog
36
37
38 The information contained in the ChangeLogs is essential, and it must be
39 kept, but, force the users to download all that data it's not optimal.
40
41 That said I can see only two ways to reduce the ChangeLog files (a
42 centralized one is obviously not viable)
43
44 1) bzip2 them in some way.
45
46 Pros:
47 - whole story is avaiable
48 - much lower file transfer size
49 Cons:
50 - grepping them need dedicated tools
51 - never ending cvs issues, may be needed an alternate non cvs tree
52 - vim already has a wrapper to read bz2 files, other editors does not
53 - also 247 bytes file will need to be zipped ?
54
55 2) "rotate" Changelogs, keeping only the last changes, until a size
56 of 4000 or [choose a preferred size here] bytes.
57 This would save only about 7Mb of data (max size < 4096).
58
59 Pros:
60 - still easily readable and parseable
61 - save download of data in the right points
62 - affect only 13% of the actual tree
63 Cons:
64 - need changes in repoman/echangelog to cut the ChangeLog in the
65 right position
66 - grepping of ChangeLog impossible (for cutted data)
67 - ChangeLog cutted at _no_ definite point in time (maybe yesterday)
68 - whole history only on viewcvs or with acks like a
69 sys-apps/ChangeLogs package
70
71 Thoughts ? It's doable in some way ?
72 --
73 gentoo-dev@g.o mailing list

Replies

Subject Author
Re: [gentoo-dev] ChangeLogs and rsync time Grobian <grobian@g.o>
Re: [gentoo-dev] ChangeLogs and rsync time Ciaran McCreesh <ciaranm@g.o>
Re: [gentoo-dev] ChangeLogs and rsync time Andrej Kacian <ticho@g.o>
Re: [gentoo-dev] ChangeLogs and rsync time "Peter Volkov (pva)" <pva@g.o>