[gentoo-portage-dev] quicker emerge sync

[Ritter Adam]

Hi!

I implemented a program that can be used to incrementally mirror the portage tree. 
It's much more efficient than the current rsync algorithm for small changes in the tree.
It creates incremental patch files with a timestamp and you can download the latest timestamps.
It takes about 40 secs to create the patch file on my machine (it should be quicker on a server I think),
so it could easily be run in about every hour. The patch files can be easily concatenated so the older
files could be stored daily.

The patch files are text files and they contain these commands:
  - create file (needed directories are automagically created)
  - create symlink
  - delete file
  - delete empty directory (I didn't want to test for every directory if it's empty when deleting a file)
  - delete symlink

It's robust: it's not a problem if you apply a patch more times.

The program is written in ruby and it's available at http://www.cs.bme.hu/~adam/ediff .
I tried to make its interface simple. To try it (better not as root!), copy the portage tree to another directory
(let's name it portage2; you can skip the distfiles directory). Delete some files from the copy.
type ediff diff /usr/portage portage2 > diff.epatch (it's a nice text file so you can examine it).

You can apply the patch with:
  ediff patch portage2 <diff.epatch

Server/client usage is a bit more complicated:

Create a timestamp with
  ediff timestamp ts (this is for the client side)
ts will contain something like 20041202_1232 (it's GMT).

Delete some files again if you have to.
Mirror it to portage3 if you deleted files randomly.
Create the patch file with
  ediff cpdiff /usr/portage portage2 patchdir (server side command)
The patchdir directory sould be created, a patch is in the directory with a name something like 20041212_2330.epatch.gz
and the portage2 is the same as /usr/portage.

Now (client side) apply the patch with
  ediff patch portage3 patchdir ts
(patchdir will be something like ftp://gentoo.patches.org/patches.. I wrote some ftp support but I was too lazy to
set up an anonymous ftp server for the patches so it probably doesn't work).

I hope some of you are interested in this little program. It's not only good for the end users (like me) but
it saves a lot of bandwidth and CPU time for the mirrors too. And the most important: people could be really up-to-date
and sync many times a day :)

It would be nice if you/I could test it on a mirror (I can't really test it at home because it would mean a lot of
rsync-ing).

Adam


--
gentoo-portage-dev@g.o mailing list