[gentoo-server] Best practices in managing large server groups

[Charles Duffy]

[Apologies if this is a duplicate post; I tried posting this through
GMANE yesterday without subscribing first, and it appears not to have
gone through; likewise when I posted several hours ago today from a 
different address than the one I subscribed].

All,

I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
application (for reasons of flexibility and licensing) (no, we don't
have an enterprise application budget -- just the reliability
requirements; yaaay, startups!). We're looking to be able to deploy and
manage hundreds of geographically distributed servers.

We have a QA department available to vet each configuration before it is
deployed to the field. We have infrastructure for tracking the progress
of code in svn from creation though QA to deployment; I'm anticipating
tracking a local overlay (containing all packages we use), make.conf,
/etc/portage/*, etc. through this system, autobuilding system images
(either to run virtualized or on real hardware) from the contents of
svn, building binary packages and deploying them to real hardware.

I'm interested in best practices, suggested tools, and/or 3rd party
experiences in this regard.

Some particular questions which come to mind:
  - Should I be using a custom profile or a standard profile with
overrides through make.conf, /etc/portage/* and the like?
  - What's the Right Way to create new system images ready to be loaded
onto a hard drive or run through a virtual machine? gentoo-buildhoster
looks interesting. I've seen Catalyst mentioned as a way to create
stage3 images, but what documentation I've been able to find doesn't
seem very much targeted for my use cases.
  - Any experiences with puppet? With out ratio of servers to staff,
automating configuration and administration is a priority. (We already
have an internal tool written with automating the server configuration
process in mind; it has some functionality puppet doesn't, and puppet
has functionality it doesn't; in theory, I'd like to extend puppet until
our internal tool becomes unnecessary, though I'll need to understand
puppet much better before I can think too hard about that).
  - Have any of 'yall been in the 100s-of-servers situation with
comparable requirements and come up with a different approach to
managing it? How did things work out?

Thank you!


-- 
gentoo-server@g.o mailing list