Re: [gentoo-server] Best practices in managing large server groups

[Ramon van Alteren]

Hi Charles

I've been looking for time to answer this more fully than the quick 
oneshot mail I send off earlier.
We run 600 servers on gentoo and started with a single server originally.

So yes it's definitely doable :)

I'll try and answer as much as possible, any questions feel free to mail.

Charles Duffy wrote:
> I'm looking at replacing SuSE SLES9 with Gentoo for an enterprise
> application (for reasons of flexibility and licensing) (no, we don't
> have an enterprise application budget -- just the reliability
> requirements; yaaay, startups!). We're looking to be able to deploy and
> manage hundreds of geographically distributed servers.
See above, what is your planned initial deployment ? Are you starting 
with a hundred or more servers or are you starting with just a couple ?
> We have a QA department available to vet each configuration before it is
> deployed to the field. We have infrastructure for tracking the progress
> of code in svn from creation though QA to deployment; I'm anticipating
> tracking a local overlay (containing all packages we use), make.conf,
> /etc/portage/*, etc. through this system, autobuilding system images
> (either to run virtualized or on real hardware) from the contents of
> svn, building binary packages and deploying them to real hardware.
Having a QA department to offload this work to is certainly a bonus :)
> I'm interested in best practices, suggested tools, and/or 3rd party
> experiences in this regard.
>
> Some particular questions which come to mind:
>  - Should I be using a custom profile or a standard profile with
> overrides through make.conf, /etc/portage/* and the like?
AFAIK you should be able to set all required stuff through overrides.
The point is to keep in mind the benefits of using gentoo and not try 
and work against the system.
Several people discussed running gentoo servers in produktion without a 
build toolchain (gcc etc.)
I have no comments to offer on how desirable this is, but if this is a 
goal/requirement for you I'd strongly suggest using a binary distro.
Gentoo shines as a ultra-configurable source-based distro, running it 
without a build toolchain and / or portage tree is certainly possible.
It would however take away much of the advantages of using gentoo, so 
why not switch to something else in that case.

Removing the portage tree has always been a weird question to me, nobody 
discusses removing the rpm-package database, why are people so keen on 
removing the portage tree ?
It takes roughly 550Mb of space which is quite a lot, but hardly a 
killing requirement given todays diskspace and hardware.
The linux kernel source tree is roughly in the same sizing category.
>  - What's the Right Way to create new system images ready to be loaded
> onto a hard drive or run through a virtual machine? gentoo-buildhoster
> looks interesting. I've seen Catalyst mentioned as a way to create
> stage3 images, but what documentation I've been able to find doesn't
> seem very much targeted for my use cases.
I would recommend catalyst-2. Although documentation is lacking, it 
isn't that hard to setup.
You're probably looking for the stage4 target if you want to build 
system images.
Rolling out gentoo on such a large scale, you need a repeatable system 
image build environment.

The bonus of catalyst is that you automatically get a binary package 
server in the process of generating your images.
Catalyst can be told to use a binary package cache, by carefully setting 
up your catalyst environment you can easily reuse that as source for 
your binpkg server.
I'm not familiar with gentoo-buildhoster, but since it's webpage [1] 
lists it as no longer maintained that would be a no-go area for me.

We combined catalyst with a pxe based boot environment, the quickstart 
installer [2] and puppet[3].
It allows us to provision a server within 30 minutes. That's 30 minutes 
from connecting the hardware to a switch and active in production.
This requires very little manual intervention, which we consider to be a 
good thing (tm)
And yes, that's concurrent, we believe it to be capable of roughly 30 
servers concurrent setup and that appears to be a pxe limitation.

If at all possible, try to build your deployment system thus that you 
can always easily wipe a server and reinstall.
We didn't originally and are refactoring to allow it now.
>  - Any experiences with puppet? With out ratio of servers to staff,
> automating configuration and administration is a priority. (We already
> have an internal tool written with automating the server configuration
> process in mind; it has some functionality puppet doesn't, and puppet
> has functionality it doesn't; in theory, I'd like to extend puppet until
> our internal tool becomes unnecessary, though I'll need to understand
> puppet much better before I can think too hard about that).
We are using puppet extensively. It works, although it's still rough 
around the edges which is as expected from such a young project.
The gentoo provider for puppet is in it's infancy. It works, but 
definitly needs work as well.

Apart from that puppet is a very versatile and powerful tool.
And most importantly it has a very active community of people around it.
They are actively exchanging recipes for server configuration, which is 
useful in itself but becomes extremely useful when combined with
the new module organization in puppet.

Many problems you will face in deploying and configuring such an amount 
of servers will have been solved in whole or in part by someone in the 
puppet community.

I would be curious what functionality is missing from puppet right now 
in your opinion ?
>  - Have any of 'yall been in the 100s-of-servers situation with
> comparable requirements and come up with a different approach to
> managing it? How did things work out?
We've grown very very fast and have tried different methods along the way.
To be honest, we're still in the process of moving most of the 
serverpark under puppet control (nearly 50% done)
And I actually do not expect to find a single set of tools that cope 
with all the issues that you face when deploying such an amount of servers.
Your situation might be different because you are starting with a single 
app.

One vital thing that is missing from the picture is an inventory database.
You need some sort of queryable database that stores servers, location, 
networking info, function, server-identifyable serial of some sort, 
hardware classes, deployment status etc.
Without that you're basically lost. We use a homebrew mysql based system 
with both a cli- and a webinterface.

Apart from that, you'll need lots of infrastructure:
* logservers
* monitoring
* statistics gathering
* backups
* scripts repository
* version controlled configs
* bug / issue tracking
* firewalling
* loadbalancing
* network monitoring and configuration system
* etc. etc. etc.
> Thank you!
You're welcome, I would like to see more people use gentoo in 
large-scale environments and am actively looking for possibilities to 
exchange experiences.
Feel free to contact me if you have any questions. I also hang out at a 
couple of irc-channels [4] with nickname Innocenti

Out of pure curiosity what is your staff to server ratio ?

Grtz Ramon

Senior System Administrator Hyves.nl

[1]http://badpenguins.com/source/
[2]http://dev.gentoo.org/~agaffney/quickstart.php
[3]http://puppet.reductivelabs.com/
[4]#puppet, #gentoo-server, #gentoo-cluster, #gentoo-amd64
-- 
gentoo-server@g.o mailing list