[gentoo-server] strategies for upgrading packages in a large cluster of machines

[Kurt Lieber]

All --

Curious what various suggestions you can offer for the following problem:

I have a large cluster of machines that I will be building using Gentoo.  I
will be using partimage for the initial installation (build one gold server
and clone the rest from it) and cfengine for ongoing configuration
management of things like /etc/ files, etc.

The one area I'm stuck on is how to manage package upgrades.  I don't want
to have to run 'emerge -u world' on each machine.  In fact, I don't want to
have to compile things on each machine.  Ideally, I want to only have to
upgrade one machine and have those changes propogate out automatically to
the other machines from there.  (All hardware will be identical)

I've looked at rdist, but I've never used that before, so I'm not sure how
well it would work.

Another suggestion was maintaining a custom portage tree and running a
nightly 'emerge -u --usepkg world' on all the boxes.  Then, by making a
change to the portage tree (which the other servers would sync from), it
would propogate out to the servers automatically the next time the cron job
ran.  This is an option and probably the best one I have at this point, but
it seems somewhat fragile -- one slip-up in the portage tree and I'm hosed.
This also wouldn't work for installing new kernels

A third suggestion was simply to use rsync.  This has some undesirable CPU
overhead, however, as rsync recurses through the various directories.

A final solution would be to simply re-image machines each time we want to
upgrade them.  This is a rather brutal solution and requires on-site
presence, but I actually kind of like it, otherwise. 

So, what other suggestions do you have?

--kurt