Re: [gentoo-user] Cross system dependencies

[J. Roeleveld]

On Saturday, June 28, 2014 09:23:17 PM thegeezer wrote:
> On 06/28/2014 07:06 PM, J. Roeleveld wrote:
> > On Saturday, June 28, 2014 01:39:41 PM Neil Bothwick wrote:
> >> On Sat, 28 Jun 2014 11:36:11 +0200, J. Roeleveld wrote:
> >>> I need a way to add dependencies to services which are provided by
> >>> different servers. For instance, my mail server uses DNS to locate my
> >>> LDAP server which contains the mail aliases. All these are running on
> >>> different machines. Currently, I manually ensure these are all started
> >>> in the correct sequence, I would like to automate this to the point
> >>> where I can start all 3 servers at the same time and have the different
> >>> services wait for the dependency services to be available even though
> >>> they are on different systems.
> >>>
> >>> All the dependency systems in the init-systems I could find are all
> >>> based on dependencies on the same server. Does anyone know of something
> >>> that can already provide this type of dependencies? Or do I need to
> >>> write something myself?
> >>
> >> With systemd you can add ExecStartPre=/some/script to the service's unit
> >> file where /some/script waits for the remote services to become
> >> available,
> >> and possibly return an error if the service does not become available
> >> within a set time.
> >
> > That method works for any init-system and writing a script to check and if
> > necessary fail is my temporary fall-back plan. I was actually hoping for a
> > method that can be used to monitor availability and, if necessary, stop
> > services when the dependencies disappear.
> >
> > --
> > Joost
>
> the difficulty is in identifying failed services.
> local network issue / load issue could mean your services start bouncing.
> the best way is to have redundancy so it doesn't matter as much

I know that. A proper system for this would have a configurable amount of 
retries with a wait-time in between.

> having said all of that::
>
> systemd will start servers and buffer network activity - how this works
> for non local services would be interesting to see.

It would, but I am not going to migrate my servers to something like systemd 
without a clear and proven advantage. For me, that currently does not exist.
It also would not work as not all the software I run will happily wait while 
the rest of the stack starts.
I would end up in a bigger mess thanks to timeout issues during startup.

> with openrc :
> you could on the DNS server have a service which is just a batch script
> that uses watches for pid / program path in "ps" which outputs ACK or
> NAK to a file in an NFS share  say /nfs/monitoring/dns

Yes, but in order to access the NFS share, I need DNS to be running. Chicken-
egg problem.

> then on the mail server you could have a service that polls
> /nfs/monitoring/dns for NAK or ACK
> you can then choose to have this service directly start your dependent
> services, or if you adjust /etc/init.d/postfix to have depends =
> "mymonitorDNS" which is an empty shell of a service. your watchdog
> service could stop / start the empty shell of a script mymonitorDNS, and
> then postfix depends on mymonitorDNS
> this would save you from "i've just stopped the mail server for
> maintenance and my watchdogservice has just restarted it due to a
> NAK>ACK event"

That is the problem I have with these watchdog services. During boot, I want 
it to wait. But it needs to understand not to start a service when I stopped 
it during runtime.
Otherwise it could prevent a clean shutdown as well...

> or...
> you could have a central master machine which has it's own services,
> watchdog and monitor... i.e. /etc/init.d/thepostfixserver start  /
> depends on thednsserver which just runs
> # ssh postfixserver '/etc/init.d/postfix start'
>
> or...
> puppet and it's kin

Last time I looked at puppet, it seemed too complex for what I need.
I will recheck it again.

Thanks,

Joost