[gentoo-amd64] Re: Recent clock problems

[Duncan]

Mark Knecht posted
<5bdc1c8b0602110941y446deb24qc519fcb40298136a@××××××××××.com>, excerpted
below,  on Sat, 11 Feb 2006 09:41:20 -0800:

> Hello,
>    Just in the last week or so my AMD64 machine has started to exhibit
> problems with the clock. Maybe it's a hardware problem, or possibly
> it's some new ntpd issue after updates. At boot time it seems to be
> coming up with semi-random times. This is causing me to ask a few
> questions and try to learn a bit more about how this actually works
> under Linux. Thanks in advance.

Those can be quite frustrating.  I'm not a master, but somehow, I've
managed to bluster my way thru a couple frustrating episodes.

> QUESTION 1: UTC vs. local
>
> In /etc/conf.d/clock I can choose UTC vs. local. Does this refer to
> the time I set in the hardware clock or something else? Nominally it's
> easier sitting here in California to set the hardware clock to
> California time. Is there any problem with doing this? If I understand
> the comments I should be using 'local' but the default seemed to be
> UTC.

UTC is effectively what used to be called GMT, Universal Time Coordinated
(I believe French, so looks backward to us English folks), Greenwich Mean
Time, the time standardized to London, UTC indicating as updated with
official leap-seconds, according to kdict (just looked it up to be sure to
get it right).

The continental US is of course 5-8 hours behind UTC, with an adjustment
for DST in the summer, naturally (that I thankfully don't have to worry
about here in AZ).

Traditionally, Unix has kept system time in UTC, while MSWormOS kept it in
local time, so if you are booting between the two, it can /really/ screw
things up.  Modern Linux has a toggle, but if the toggle gets out of sync
for some reason, you'll see it jumping back and forth by those several
hours, as you boot and reset it and reboot.  To get it synced back up can
be more difficult than you'd expect, because if you don't do all the steps
in the right order, it either doesn't take, or it takes, but just confirms
the offset once again.

> QUESTION 2: date
>
> date only sets a software clock, correct? Is this all the system uses
> after boot?

Short answer, yes, to both.  The hardware clock only gets reset when you
reset it, and it's not used for keeping time while the system is running.

> QUESTION 3: hwclock -w
>
> This is supposed to write the time in the system's software clock into
> hardware, correct?

The key word there is /supposed/.  Can you tell I've gotten rather
frustrated with it? =8^P

Here's a hint:  It's /much/ easier to figure out how to use the Gentoo
scripts and configuration, and call the initscript manually to reset the
hardware clock if necessary, than it is to get the sequence of steps
correct to set the hardware clock correctly, getting the toggle and the
local time offset correct all at the same time.  I don't know whether it's
because the hwclock command line is unnecessarily obtuse, or what, but
what one would /think/ would be a simple, logical process, just /isn't/!

So.  Set the Gentoo configuration, then turn off dhcpd if it's on and set
the (system/software) clock, then issue the command /etc/init.d/clock
restart, and it should first set the hardware clock from the software
clock (as if its shutting down), then set the software clock from the
hardware clock (as if the system is booting up).

The settings you need, at least in the baselayout-1.12.0-pre series, are
all in /etc/conf.d/clock.  baselayout-1.11.x may have had a couple of
these settings in either /etc/rc.conf or /etc/conf.d/rc, as I seem to
remember some setting there, but they aren't there now.

If you want to keep your hardware clock set for localtime, set
CLOCK="local" in the config file.  CLOCK_OPTS should normally remain
blank.  I'd certainly suggest you leave them so if this is at all useful
to you, as I had little enough success setting the thing right manually
that I wasn't /about/ to try and screw with things here, and if this is
useful, than you know no more than I do about it! =8^P

Perhaps the most important setting is CLOCK_SYSTOHC (system to hardware). 
You most likely want this set to "yes", as once you get things configured,
ntpd will be keeping things adjusted for you and you'll be able to let it
correct the hardware clock at shutdown.  In any case, you'll probably want
it set yes to allow you to properly set the hardware clock this time, then
you can turn it back to no if you wish.

Ensure the Alpha options are both set "no", as they should be unless you
screwed with them desperately trying to get things to work.

As I said above, once that's done, ensure dhcpd is off, and restart the
clock system service (it'll probably warn you that your stopping a boot
level service, but it shouldn't hurt, I do it here to set my clock).  The
clock boot script should then magically accomplish what was SO
FRUSTRATINGLY DIFFICULT to get right, manually.

Assuming there's nothing seriously wrong with your system, that should get
things back to being relatively accurate.  If it doesn't, there are a
couple of possible causes.  If the clock is going crazy, say running
double-speed when the system is up, it's a system timer problem.  You'll
need to find some way to regulate your kernel so it keeps time properly,
and dhcpd won't be able to help as the problem will be to drastic.  The
solution there involves possible kernel bootline switches, and/or finding
the proper driver for your chipset so the kernel has a reliable hardware
timer it can sync against.

If your hardware clock keeps resetting to January 1 of 2000, or some other
date, or if it goes random (not off by a fixed offset, which would
indicate UTC/local issues again/still) it's possible you have a battery
going dead, tho that really shouldn't be happening yet with amd64 boards. 
Maybe it was shorted out or something and lost a whole bunch of juice at
once, or maybe you have a bad timing capacitor or something.  This can be
bad news, because it often indicates you are about to start having other
problems, like the BIOS not retaining its settings over a shutdown, and
the like.  Get ready to buy a new mobo.  However, if it's just the clock
going bad, you can usually use ntp to correct the system once at boot and
then keep things regular while running.  It'll just be irritating if you
boot up for maintenance and don't go online or start ntp as your time
could be way off, but anyway...

> QUESTION 4:
>
> Does ntpd actually update the system clock, or is it another layer yet
> on top. If I use ntpd and then do hwclock -w does an accurate time get
> written to the hardware clock?

I don't like the term "system" clock in this context, because it's
ambiguous as to whether you are referring to hardware or software.  That's
why I used system/software clock or the like a couple times above, to make
it absolutely clear which I was referring to.

That said, "system clock" normally refers to the software clock that the
system uses while running, and yes, ntpd /does/ update it directly --
well, sort-of.  <g>  Don't worry, the confusion is normal and under the
circumstances entirely understandable, but there's a logical explanation
that makes sense once you understand what's going on.

Here's the deal.  ntpd was developed to be used in situations where
various running programs might be making critical assumptions about the
system time -- namely that it always moves forward, never backward.  To
have the time move backward, if the machine's an online server in some
applications, would be /very/ /bad/!  Thus, by default and if at all
possible (if the time isn't off by days or years), ntpd will normally
choose to /skew/ the time, adjusting how fast the machine advances time by
a few parts per million, rather than stepping it either forward or
backward by seconds/minutes/hours/days/whatever at a time.

On Linux, the kernel is actually responsible for keeping time, and has a
userland API that ntpd uses to skew the time, telling the  kernel to
measure say 999 998 instead of 1 000 000  ticks if the clock is slightly
fast, until real time catches up, or 1 000 005 ticks instead of 1 000 000,
if the clock is slightly slow and needs to catch up to real time.  So,
yes, ntpd /is/ adjusting the time, but ideally, not fast enough that
anything can really notice it.

Note also that ntpd keeps track of how far your system has drifted from
the rest of the world, each time it checks in, and uses that to tweak
system clock speed accordingly.  If you go adjusting the clock yourself
while ntpd is running, it's going to think something's drastically wrong
with the system that it gained or lost so much time so fast, and will
REALLY put the screws on that skew, thereby REALLY screwing up the system
for a few days, as ntpd adjusts the clock first one way, then the other,
until it figures out the real drift of the system once again.  This is why
I said turn off ntpd before you go screwing with the clock, even the
little bit of setting the hardware clock from the software clock and then
the software clock from the hardware clock, as you do by running
/etc/init.d/clock restart, because even the 100th of a second leap that
might happen setting the time between the two, would be enough to screw up
ntpd's calculations and set your time swinging that much more than it
should, for a day or two.  Once it knows how accurate your system is on
its own, ntpd can make it /really/ accurate, to single milliseconds within
the day, but it can't do it if you keep screwing with the time yourself! 
ALWAYS turn ntpd off when you go screwing with the clock, for /whatever/
reason!

FWIW, if you've been screwing with the clock enough to mix up ntpd and
just want it to start over from scratch, shut it down (of course), and
remove it's drift file, located by default on Gentoo at
/var/lib/ntp/ntp.drift (I think, mine's in /etc/ntp/drift, as set in
/etc/ntp.conf, but if the ntp.conf.orig I have here is correct, Gentoo's
default is that /var location).

OK, so what is used to do a time step, if ntpd normally does time skews?

That's ntp-client.  If you have ntp running on your system, you'll
probably want both ntpd and ntp-client in your default runlevel. 
ntp-client runs before ntpd, and steps the clock to /approximately/ the
correct time (within 100ms or so).  It's a single-shot run at boot, after
which ntpd starts, does it's connections to see how far it has to adjust,
and gets to work doing the skew.  If you look in the logs, probably 15
minutes to a couple hours  later (assuming it has had time to get
familiar with the system and how much it normally needs adjusted), it will
say it's properly synced to its satisfaction, having skewed the system
(software) clock by an appropriate degree, and then set out to maintain
that sync over the longer term.

Do note that there's an ntp-client config file in /etc/conf.d also, which
you can use to tweak settings if desired.  However, Gentoo's policy of
setting things up so they "just work" if possible, is as usual in force
here, and it should need no changes from you to work properly, once the
initscript has been added to the proper runlevel, of course.

Once you have it set up correctly, your system will boot and once the
internet connection is up, ntp-client will start and do any "gross"
adjustment necessary to the system time.  After it's done, ntpd will run
and keep your system very accurately synced with the rest of the world,
timewise.  When you shutdown, if you have clock configured to do so, it
will sync the hardware clock to the software clock that ntpd has been
managing so well while the system was up, keeping the hardware clock
fairly accurate as well, even when you only boot to no-net single-user
mode or whatever for system maintenance.

One more thing to cover:  If you suspend your system, you will want to
ensure that ntpd is shut down before suspend, and ntp-client and ntpd are
restarted after resume.  Again, this  ensures that ntpd is only tracking a
running system, so doesn't get any strange and unnecessary jerks in its
idea of how the system is keeping time, thereby keeping it as accurate as
possible, which is the idea, after all!  =8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@g.o mailing list