RE: [gentoo-dev] logging in openntpd 20080406-r3+

[Paul B. Henson]

> From: Dirkjan Ochtman [mailto:djc@g.o]
> Sent: Friday, November 22, 2013 12:30 PM
>
> - Without -s, it can take a *very* long time to get close to an
> acceptable time error, whereas my initial expectation was that
> "starting my ntpd should fix the time error fairly quickly". But for
> me this, this is partly about starting ntpd while the machine is
> online, not just at boot.

In general, ntpd tries not to violate the presumption that time is monotonically increasing. Rather, it adjusts the clock rate such that your system time approaches real time; if your current time is too far behind, the clock runs faster, and each "second" takes less than a second, if your time is too far ahead, each "second" takes more than a second. However, if your time is very far off, that will take a considerable amount of time (heh heh) to synchronize. The -s option makes makes ntpd simply set the time to exactly whatever the current time is, regardless of what the system clock happens to say. This could be a huge jump, possibly into the "past" from the perspective of the system. Generally, this is only done at boot, typically before other processes are started that might need the correct time. Depending on what services your system is running, something might be quite unhappy if suddenly it is eight minutes earlier than it appeared to be when the process started.

> - Second, with -s, the boot delays can be quite long. I'm pretty sure
> I've seen delays that are quite a bit longer than 15s, probably in the
> case where there's no network or maybe where DNS doesn't resolve well;

I've tested a variety of scenarios, from the network interface being down/unplugged, providing invalid NTP servers, etc., and I haven't seen a delay longer than 15 seconds. If you look at the source code in ntpd.c:

        while ((ch = getopt(argc, argv, "df:nsSv")) != -1) {
[...]
                case 's':
                        lconf.settime = 1;

If you supply the -s option lconf.settime is set, and:

        if (!lconf.settime) {
                log_init(lconf.debug);
                if (!lconf.debug)
                        if (daemon(1, 0))
                                fatal("daemon");
        } else
                timeout = SETTIME_TIMEOUT * 1000;

rather than immediately daemonizing, it sets a 15 second timeout (SETTIME_TIMEOUT is defined to 15 in ntpd.h).

It then enters the main loop, where if a response is not received within 15 seconds:

                if ((nfds = poll(pfd, 1, timeout)) == -1)

                if (nfds == 0 && lconf.settime) {
                        lconf.settime = 0;
                        timeout = INFTIM;
                        log_init(lconf.debug);
                        log_debug("no reply received in time, skipping initial "
                            "time setting");
                        if (!lconf.debug)
                                if (daemon(1, 0))
                                        fatal("daemon");
                }

It backgrounds anyway and aborts the initial time set. I'm not saying there isn't some bug or scenario which would result in a longer delay, but if so, that is a bug in ntpd, not an issue to be worked around in the startup script.

> in any case, when you're trying to debug issues in a data center
> environment, waiting for a bunch of machines to come up is not much
> fun. (Or when you've had a machine go down and you're waiting to see
> if it comes up again.)

In my data center, NTP is considered a critical service and provisioned with fault tolerance. If a box trying to boot cannot reach an NTP server, that is as much of a problem as whatever is wrong with the box booting. I don't believe I've ever seen a boot delay caused by NTP on any of our production systems ever. If you can provide a reproducible failure mode where ntpd takes longer than 15 seconds to start I'd be willing to take a look and see what's going on. Ideally though, this should be reproducible on a system already running, not something only happening during boot, as it would be more difficult to debug the process at that state.

> Now, for my use case, it is not all that important that the time error
> is minimized before resuming the boot process, but I really wanted to
> minimize boot delays.

Then I advise you not to use the -s option, in which case there will never be a delay, no matter what.

> Also, I'm really not sure how the change to logging to stderr/file and
> running in debug mode helps with the boot delays.

Basically, the new startup script does something like:

/usr/sbin/ntpd -d [-s] 2>/var/log/ntpd.log &

The process is immediately put into the background and the startup sequence continues. This eliminates the boot delay, but at the cost of not actually setting the time before other processes are started (if the -s option is provided), using really kludgy logging, and always running the process in debug mode.

Personally, I think it should all be put back to the way it was to begin with, which was perfectly functional. If you want the time set at boot, use -s, and live with whatever delay that might cause in an environment with unreliable NTP service, if you don't want any boot delays, don't use -s, and live with a potentially inaccurate clock. Or, if you want the clock set more or less at startup, but don't want to have a delay, rather than starting ntpd in the run level, start it from a local.d script after everything else is started where any delay won't be an issue.

Perhaps a note could be added to the default conf.d file pointing out that adding the -s option will cause the boot to be delayed until the time is set or a timeout occurs...