Gentoo Archives: gentoo-server

From: Thilo Bangert <bangert@g.o>
To: gentoo-server@l.g.o
Subject: Re: [gentoo-server] Best practices in managing large server groups
Date: Sat, 26 May 2007 09:57:38
Message-Id: 200705261153.32953.bangert@gentoo.org
In Reply to: Re: [gentoo-server] Best practices in managing large server groups by Ramon van Alteren
1 hi all,
2
3 > It's likewise nice to hear someone is planning the same as we do.
4
5 at work, we are currently planning for a massive expansion of our server
6 park and i am in the process of figuring it all out. not having done this
7 before i am probably about to fall into a bunch of holes - so any insight
8 into how you do stuff is of great help. thanks for all the information so
9 far.
10
11 what follows are the pieces that we've put together for now. much of this
12 has not been tested yet, but i'd wouldn't mind some feedback.
13
14 - use catalyst2 to build different role stage4 images.
15 - seed a binhost with the binary packages from catalyst
16 - add additional packages to the binhost (as needed)
17 - use quickstart to deploy stage4 image
18 - use enhost to add the host to a inventory db
19 - use puppet for configuration management
20 - use munin for resource and performance monitoring
21 - use nagios for service monitoring
22 - build a tool which creates nagios configuration from the inventory
23 - all syslog to a central loghost - using srlog2
24 - use nullmailer to forward email to a central smarthost
25 - higher level applications will be deployed using capistrano - possibly
26 drawing some of the configuration options from the inventory database.
27
28 we will not do backups of the nodes - the goal is to be able to reproduce
29 the setup faster (or at roughly the same speed) than a backup recovery.
30 application data will most likely be backed up to amazon S3 in (near)
31 real time.
32
33 we are currently looking into RT or bugzilla for issue tracking...
34
35 i'd love to hear what tools other people are using for these and similar
36 tasks or what comments people have to the setup outlined above.
37
38 > I would be very
39 > curious about the inventory system you are planning, if you're willing
40 > to shed some light on that I'd be grateful.
41
42 i haven't given this two much of a thought, just yet. enhost supplies its
43 data to ldap - so for a start we'll probably just use that. depending on
44 how the deployment procedure of the higher layers works out a relational
45 database may turn out to be better suited...
46
47 kind regards
48 Thilo