Re: [gentoo-user] another old box to update

[Alan McKinnon]

On 07/01/2015 22:30, Stefan G. Weichinger wrote:
> Am 07.01.2015 um 20:06 schrieb Tomas Mozes:
>
>> Strange, I only have successful stories with upgrading old gentoo
>> machines. If you have a machine which you update regularly then you know
>> all the issues during the time and so upgrading "per partes" leads to no
>> surprises but the same challenges you've handled before. But yes, it
>> takes time.
>>
>> Moreover, if you use configuration management like Ansible, you can even
>> automatically merge changes when applications ship new configuration.
>
> Thanks for that posting, it reminds me of some bigger issue I wanted to
> discuss here for quite a while now.
>
> Over the years I am now responsible for dozens of servers and VMs
> running gentoo linux ... and I wonder how to efficiently keep track of them.
>
> I learned my first steps with puppet and use it in a basic setup for my
> own machines in my LAN. It seems to work better for many identical
> servers, let's say in a hosting environment.
>
> The servers at my customers are somehow similar but not identical:
>
> different setups for services ... different update-cycles (which have to
> be synchronized and shortened as we have seen in this thread!) ...
>
> I look for a way of tracking all these systems:
>
> a) central database/repo with all the systems and how to access them:
>
> 	* unique system id
> 	* what IP, port, ssh-key, etc etc
>
> I use git for local tracking of /etc on most of my systems in the last
> years, but I did never really come up with a clever way how to
> centralize dozens of separate git-repos ... one repo per server pushed
> to one central git-home on of my inhouse servers?
>
> b) in addition tracking of let's say "rules" or "services":
>
> 	* which server runs e.g. apache? So if there is a new security warning
> out there for apache ... ask system "which servers/customers would need
> that update?"
>
> etc etc
>
> c) when was my last access to that server? Have I looked into it lately?
>
> (or more business-oriented:)
> Do I even have to / does the customer pay for that?)
> This should lead to some SLA-kind-of-thing, yes ... a bit off-topic for now.
>
> -
>
> Puppet is more oriented to push configs out to systems.
>
> Maybe a combination would apply ... puppet for building the basement,
> having stuff generalized (this path, that password/ssh-key ....)
>
> and some other components to track what has been done over time.
>
> I run OTRS  ( http://en.wikipedia.org/wiki/OTRS ) for my daily work and
> looked into their module "ITSM" (
> https://www.otrs.com/homepage/software/otrsitsm-features/ ) lately ...
> it allows to create "configuration items" (think: ITIL) etc, so far I
> think this is a bit of overkill and not really fitting the size of my
> business.
>
> I'd love to keep it simple and CLI-oriented:
>
> Gentoo allows to define (nearly?) everything via text-files, combined
> with the cleverness of git (and maybe puppet) this should give me a way of
>
> a) easily deploy new systems with configs according to some standards:
> 	I want these packages/users/paths/files ...
>
> b) track these systems: what boxes am I responsible for, what is out
> there and failing? ;-) (not talking monitoring here ... just what are my
> active systems out there)
>
> from there I should slowly get into defining new contracts with my
> clients including regular checks each 3 or 6 months ... what has to be
> done, are there any bigger updates to do (think udev, baselayout ...)
> and tell them if is possible to update the box within a few hours in
> parallel to normal work or if we need a bigger maintenance window.
>
> ---
>
> I am sure there are many other gentoo-users out there with similar
> challenges to face. And I am looking forward to your thoughts,
> experiences and recommendations!


In my opinion, ansible almost always beats puppet.

Puppet is a) complex b) built to be able to deal with vast enterprise
setups and c) has a definition language I never could wrap my brains
around. It always felt to me like puppet was never a good fit for my needs.

Then ansible hit the scene. Now ansible is designed to do what sysadmins
do anyway, to do it in mostly the same way, and to do it in an automated
fashion. It fits nicely into my brain and I can read a playbook at a
glance to know what it does.

Ansible deploys stuff and makes it like how you want it to be. It is
equally good at managing 1000 identical machines as 100 mostly the same
ones, or 20 totally different ones. How it manages this miracle is not
easy to explain, so I urge you to give it a test drive. Fool around with
a bunch of VMs to get a feel of how it works. A tip: keep it simple, and
use roles everywhere.

Ansible works nicely with other tools like vagrant and docker which
build and deploy your base images. Once you have a base machine with
sshd running and an administrative login account, ansible takes over an
manages the rest. It works really well.

On the business side of things, yes indeed you need to rationalize
things and what you offer to customers. There comes a point where you
the business grows and you just can't manage all these different things.
Mistakes get made, SLAs slip, and everyone gets annoyed.

On how to track all that real-world data you mentioned, I have a few
rules of my own. Monitor and track everything I can, get and store as
much info out of logs as I can. All the info you need is in there
somewhere. But how to get it is a problem highly specific to your
business. Maybe start some new threads each one with a specific
question, and watch for common solutions in the answers?

-- 
Alan McKinnon
alan.mckinnon@×××××.com