I have a rarely-on (media player) linux box that since the daylight savings changeover (I think) always boots with the time off by 1 hour. (I think it thinks it's 1 hour later than it really is.)
It's a modern(ish) Fedora 16 with the latest updates. I can set the clock correctly manually or with ntpdate and all is fine until reboot. Then it's wrong again. I am running ntpd on it to set the clock using my own internal ntpd server that does keep correct time. I did check that the timezone was set to Americas/Winnipeg and the daylight savings stuff all looks correct. I use "system clock is set to GMT".
Such a silly problem, and I'm stumped!
On 2012-11-20 08:51, Trevor Cordes wrote:
I have a rarely-on (media player) linux box that since the daylight savings changeover (I think) always boots with the time off by 1 hour. (I think it thinks it's 1 hour later than it really is.)
It's a modern(ish) Fedora 16 with the latest updates. I can set the clock correctly manually or with ntpdate and all is fine until reboot. Then it's wrong again. I am running ntpd on it to set the clock using my own internal ntpd server that does keep correct time. I did check that the timezone was set to Americas/Winnipeg and the daylight savings stuff all looks correct. I use "system clock is set to GMT".
Such a silly problem, and I'm stumped!
I don't think ntpdate and ntpd modify the system clock, only the runtime clock. So, my guess is that the two clocks are not in synch. Normally, RedHat-ish systems (including Fedora) write back the runtime clock setting to the system clock on shutdown, but perhaps this isn't happening for some reason.
Try the hwclock command with the appropriate options to see what the system clock is actually set at, and if it's different than what's reported by the date command, use hwclock to write the runtime clock to the system clock.
On 2012-11-20 Gilbert E. Detillieux wrote:
I don't think ntpdate and ntpd modify the system clock, only the runtime clock. So, my guess is that the two clocks are not in synch. Normally, RedHat-ish systems (including Fedora) write back the runtime clock setting to the system clock on shutdown, but perhaps this isn't happening for some reason.
Ya, that was it. I fixed the sys clock then wrote it to the hwclock and rebooted and it seems to be correct now, thanks!
So the big mystery is why is the Fedora shutdown not writing the correct time to the hwclock? I too thought that was automatic. I've never had this problem on other Fedora systems. Anyone know what program/rpm/script is responsible for this? Some quick greps of etc aren't finding me anything.
On 20/11/2012 12:57 PM, Trevor Cordes wrote:
On 2012-11-20 Gilbert E. Detillieux wrote:
I don't think ntpdate and ntpd modify the system clock, only the runtime clock. So, my guess is that the two clocks are not in synch. Normally, RedHat-ish systems (including Fedora) write back the runtime clock setting to the system clock on shutdown, but perhaps this isn't happening for some reason.
Ya, that was it. I fixed the sys clock then wrote it to the hwclock and rebooted and it seems to be correct now, thanks!
So the big mystery is why is the Fedora shutdown not writing the correct time to the hwclock? I too thought that was automatic. I've never had this problem on other Fedora systems. Anyone know what program/rpm/script is responsible for this? Some quick greps of etc aren't finding me anything.
In RHEL5, this was done in the start function of /etc/init.d/ntpd, after running ntpdate to initially sync with the time server, so not at shutdown time apparently. Also, it was only done if SYNC_HWCLOCK was set to yes in /etc/sysconfig/ntpd, and the default value seems to be no. In Fedora 17, the whole init.d/rc.d business is replaced with systemd, whose config files live in /lib/systemd/system. The ntpdate.service config file there sets ExecStart=/usr/libexec/ntpdate-wrapper, and that script seems to do the whole hwclock --systohc thing if SYNC_HWCLOCK=yes, this time in /etc/sysconfig/ntpdate, but again the default seems to be no. In both cases, the hwclock synching only happens if ntpdate returns 0, so no synching, even with everything setup correctly, if ntpdate fails. As far as I can tell, neither RHEL5 nor F17 call hwclock at any point during shutdown, which seems odd/bass-ackwards.
Finally, a bit of a source of frustration is that in Fedora, the ntpdate-wrapper script ignores any settings in /etc/sysconfig/clock, so that can cause some issues if hwclock doesn't pick the appropriate default. It's supposed to use the last value of --utc or --localtime that was used, which apparently is stored in /etc/adjtime, but in practice I've found that when switching between OSes on a multi-boot system, Fedora seems to get confused about whether you wanted to use local time or UTC.
On 2012-11-20 13:30, Gilles Detillieux wrote:
On 20/11/2012 12:57 PM, Trevor Cordes wrote:
On 2012-11-20 Gilbert E. Detillieux wrote:
I don't think ntpdate and ntpd modify the system clock, only the runtime clock. So, my guess is that the two clocks are not in synch. Normally, RedHat-ish systems (including Fedora) write back the runtime clock setting to the system clock on shutdown, but perhaps this isn't happening for some reason.
Ya, that was it. I fixed the sys clock then wrote it to the hwclock and rebooted and it seems to be correct now, thanks!
So the big mystery is why is the Fedora shutdown not writing the correct time to the hwclock? I too thought that was automatic. I've never had this problem on other Fedora systems. Anyone know what program/rpm/script is responsible for this? Some quick greps of etc aren't finding me anything.
In RHEL5, this was done in the start function of /etc/init.d/ntpd, after running ntpdate to initially sync with the time server, so not at shutdown time apparently. Also, it was only done if SYNC_HWCLOCK was set to yes in /etc/sysconfig/ntpd, and the default value seems to be no. In Fedora 17, the whole init.d/rc.d business is replaced with systemd, whose config files live in /lib/systemd/system. The ntpdate.service config file there sets ExecStart=/usr/libexec/ntpdate-wrapper, and that script seems to do the whole hwclock --systohc thing if SYNC_HWCLOCK=yes, this time in /etc/sysconfig/ntpdate, but again the default seems to be no. In both cases, the hwclock synching only happens if ntpdate returns 0, so no synching, even with everything setup correctly, if ntpdate fails. As far as I can tell, neither RHEL5 nor F17 call hwclock at any point during shutdown, which seems odd/bass-ackwards.
Actually, RHEL5 has the following line in /etc/init.d/halt...
[ -x /sbin/hwclock ] && action $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS
The value of CLOCKFLAGS is calculated earlier in the same script, based on the UTC and ARC variables set in the /etc/sysconfig/clock config file. A similar line exists in the halt script in RHEL6, except it no longer sources /etc/sysconfig/clock, but just has a hard-coded option.
That seems to have disappeared with the switch to systemd in Fedora. So, the best bet seems to be to configure ntpdate to work correctly, so you at least get the clock synch'ed on startup.
Finally, a bit of a source of frustration is that in Fedora, the ntpdate-wrapper script ignores any settings in /etc/sysconfig/clock, so that can cause some issues if hwclock doesn't pick the appropriate default. It's supposed to use the last value of --utc or --localtime that was used, which apparently is stored in /etc/adjtime, but in practice I've found that when switching between OSes on a multi-boot system, Fedora seems to get confused about whether you wanted to use local time or UTC.
On 20/11/2012 1:47 PM, Gilbert E. Detillieux wrote:
Actually, RHEL5 has the following line in /etc/init.d/halt...
[ -x /sbin/hwclock ] && action $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS
The value of CLOCKFLAGS is calculated earlier in the same script, based on the UTC and ARC variables set in the /etc/sysconfig/clock config file. A similar line exists in the halt script in RHEL6, except it no longer sources /etc/sysconfig/clock, but just has a hard-coded option.
That seems to have disappeared with the switch to systemd in Fedora. So, the best bet seems to be to configure ntpdate to work correctly, so you at least get the clock synch'ed on startup.
I missed that in my grepping. I was looking for the K* script links in /etc/rc.d/rc?.d, forgetting that when you're halting or shutting down, you're actually *starting* run level 0 or 6, so it's the S??halt or S??reboot link that gets run.
What I found for systemd on F17 is the /usr/lib/systemd/system-shutdown directory. According to this page: http://www.freedesktop.org/software/systemd/man/systemd-halt.service.html, "Immediately before executing the actual system halt/poweroff/reboot/kexec systemd-shutdown will run all executables in /usr/lib/systemd/system-shutdown/ and pass one arguments to them: either "halt", "poweroff", "reboot" or "kexec", depending on the chosen action. All executables in this directory are executed in parallel, and execution of the action is not continued before all executables finished." So, presumably one could add a little script there to run hwclock --systohc right before shutdown/reboot. And since the warring ideologues can't be counted on to put it there for you in the distro, I guess it's up to us pragmatists to do it ourselves. :-)
On 2012-11-20 Gilles Detillieux wrote:
fails. As far as I can tell, neither RHEL5 nor F17 call hwclock at any point during shutdown, which seems odd/bass-ackwards.
Ya, it looks like no one is touching hwclock at shutdown. To me it makes sense that on boot we read the hwclock, then we update with ntpdate/ntpd, ... system runs ..., then on shutdown we put our idea of the proper date (which if things are working should always be correct) back to hwclock for next time. This is how it used to work, but doesn't now.
I found: https://bugzilla.redhat.com/show_bug.cgi?id=816752 (long-winded)
In F16 (and newer) it looks like you need to enable both ntpd.service and ntpdate.service in systemd. On all my boxes I had just ntpd on. Time to change that.
Then set SYNC_HWCLOCK=yes in /etc/sysconfig/ntpdate
Looks like systemd is properly setup to run ntpdate before ntpd.
Thanks guys.
On 2012-11-20 14:24, Trevor Cordes wrote:
On 2012-11-20 Gilles Detillieux wrote:
fails. As far as I can tell, neither RHEL5 nor F17 call hwclock at any point during shutdown, which seems odd/bass-ackwards.
Ya, it looks like no one is touching hwclock at shutdown. To me it makes sense that on boot we read the hwclock, then we update with ntpdate/ntpd, ... system runs ..., then on shutdown we put our idea of the proper date (which if things are working should always be correct) back to hwclock for next time. This is how it used to work, but doesn't now.
I found: https://bugzilla.redhat.com/show_bug.cgi?id=816752 (long-winded)
Yeah, this is one of the things that bugs me about RedHat/Fedora (and maybe the whole Linux culture in general). Bug threads will sometimes degenerate into philosophical arguments between conflicting ideologies, and since an ideal solution isn't readily available, the default choice is to do nothing (or remove something that was working, but not ideal).
Sometimes, you just want to grab a pragmatist's clue-stick, and start whacking! ;)
In F16 (and newer) it looks like you need to enable both ntpd.service and ntpdate.service in systemd. On all my boxes I had just ntpd on. Time to change that.
Then set SYNC_HWCLOCK=yes in /etc/sysconfig/ntpdate
Looks like systemd is properly setup to run ntpdate before ntpd.
Yeah, this should work for startup, but unfortunately, doesn't address the issue of RTC drift during uptime. They really should fix the ntpd script to rewrite the RTC on shutdown, if SYNC_HWCLOCK is set. (And, ntpdate should really be merged back into ntpd's script, but that's a whole other nerd debate...)
Thanks guys.
You're welcome.
-
Gilbert E. Detillieux -
Gilles Detillieux -
Trevor Cordes