Gentoo Archives: gentoo-portage-dev

From: Ritter Adam <adam@××××××.hu>
To: gentoo-portage-dev@l.g.o
Subject: [gentoo-portage-dev] quicker emerge sync
Date: Thu, 22 Jul 2004 21:55:12
1 Hi!
3 I implemented a program that can be used to incrementally mirror the portage tree.
4 It's much more efficient than the current rsync algorithm for small changes in the tree.
5 It creates incremental patch files with a timestamp and you can download the latest timestamps.
6 It takes about 40 secs to create the patch file on my machine (it should be quicker on a server I think),
7 so it could easily be run in about every hour. The patch files can be easily concatenated so the older
8 files could be stored daily.
10 The patch files are text files and they contain these commands:
11 - create file (needed directories are automagically created)
12 - create symlink
13 - delete file
14 - delete empty directory (I didn't want to test for every directory if it's empty when deleting a file)
15 - delete symlink
17 It's robust: it's not a problem if you apply a patch more times.
19 The program is written in ruby and it's available at .
20 I tried to make its interface simple. To try it (better not as root!), copy the portage tree to another directory
21 (let's name it portage2; you can skip the distfiles directory). Delete some files from the copy.
22 type ediff diff /usr/portage portage2 > diff.epatch (it's a nice text file so you can examine it).
24 You can apply the patch with:
25 ediff patch portage2 <diff.epatch
27 Server/client usage is a bit more complicated:
29 Create a timestamp with
30 ediff timestamp ts (this is for the client side)
31 ts will contain something like 20041202_1232 (it's GMT).
33 Delete some files again if you have to.
34 Mirror it to portage3 if you deleted files randomly.
35 Create the patch file with
36 ediff cpdiff /usr/portage portage2 patchdir (server side command)
37 The patchdir directory sould be created, a patch is in the directory with a name something like 20041212_2330.epatch.gz
38 and the portage2 is the same as /usr/portage.
40 Now (client side) apply the patch with
41 ediff patch portage3 patchdir ts
42 (patchdir will be something like I wrote some ftp support but I was too lazy to
43 set up an anonymous ftp server for the patches so it probably doesn't work).
45 I hope some of you are interested in this little program. It's not only good for the end users (like me) but
46 it saves a lot of bandwidth and CPU time for the mirrors too. And the most important: people could be really up-to-date
47 and sync many times a day :)
49 It would be nice if you/I could test it on a mirror (I can't really test it at home because it would mean a lot of
50 rsync-ing).
52 Adam
55 --
56 gentoo-portage-dev@g.o mailing list