Re: [gentoo-server] strategies for upgrading packages in a large cluster of machines

[Andrew Cooks]

I think this problem is actually quite common and not just restricted to
number crunching clusters.

We're running Gentoo on about 200 workstations at the University of
Pretoria and I've spent a fair amount of time trying to find the best
solution for keeping the machines in sync and up to date. It gets worse
when Linux is the secondary OS and the labs are run by non-Linux people.

We're using the system of creating a master image for each specific
hardware combination and distributing that image to all the
workstations. If I had any say in it, we would've used UDPCast from
http://udpcast.linux.lu/, but the suits like paying for "enterprise"
software like Ghost or Imagecast.

UDPCast, together with PXE booting could work nicely. You still install
everything on the workstations, but the boot loader is downloaded. This
way, you can swap the bootloader at any time to "cast" any or all
machines. UDPCast  also uses multicast and sector-by-sector copying, so
casting more machines don't take substantially longer and it doesn't
have to understand the file system. 

At the moment though, we're "casting" every few months, whenever the
other OS breaks or needs patching. Gentoo updates are made at this time
and smaller/critical updates are made with some home-grown scripts that
basically just looks in a specific NFS-mounted directory for scripts and
packages at boot time. This creates some extra work with managing config
files. I'll definitely have a look at cfengine to solve this.

We're considering mounting /usr/portage from a file server. You can
actually keep a full gentoo installation in a directory on a server and
cross compile binaries for the clients. Config files and large updates
are still a problem, but with the UDPCast + PXE approach, you can cast
an up to date and configured image whenever you want.

Hope this is worth something.

Andrew


On Wed, 2003-10-15 at 20:51, Kurt Lieber wrote:
> All --
>
> Curious what various suggestions you can offer for the following problem:
>
> I have a large cluster of machines that I will be building using Gentoo.  I
> will be using partimage for the initial installation (build one gold server
> and clone the rest from it) and cfengine for ongoing configuration
> management of things like /etc/ files, etc.
>
> The one area I'm stuck on is how to manage package upgrades.  I don't want
> to have to run 'emerge -u world' on each machine.  In fact, I don't want to
> have to compile things on each machine.  Ideally, I want to only have to
> upgrade one machine and have those changes propogate out automatically to
> the other machines from there.  (All hardware will be identical)
>
> I've looked at rdist, but I've never used that before, so I'm not sure how
> well it would work.
>
> Another suggestion was maintaining a custom portage tree and running a
> nightly 'emerge -u --usepkg world' on all the boxes.  Then, by making a
> change to the portage tree (which the other servers would sync from), it
> would propogate out to the servers automatically the next time the cron job
> ran.  This is an option and probably the best one I have at this point, but
> it seems somewhat fragile -- one slip-up in the portage tree and I'm hosed.
> This also wouldn't work for installing new kernels
>
> A third suggestion was simply to use rsync.  This has some undesirable CPU
> overhead, however, as rsync recurses through the various directories.
>
> A final solution would be to simply re-image machines each time we want to
> upgrade them.  This is a rather brutal solution and requires on-site
> presence, but I actually kind of like it, otherwise.
>
> So, what other suggestions do you have?
>
> --kurt
>
>