Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

swclock init: may send the clock backwards of current deptree reference. #675

Open
macmpi opened this issue Nov 28, 2023 · 11 comments · May be fixed by #678
Open

swclock init: may send the clock backwards of current deptree reference. #675

macmpi opened this issue Nov 28, 2023 · 11 comments · May be fixed by #678

Comments

@macmpi
Copy link

macmpi commented Nov 28, 2023

At first run (openrc-shutdowntime does not exist), swclock sets time according to openrc-run file mtime.

	if ! swclock "${swclock_file}" 2> /dev/null; then
	swclock --warn @SBINDIR@/openrc-run

Until then mtime reference for deptree was based on latest mtime found in RC_INITDIR, RC_CONFDIR, RC_CONF & al.
In some circumstances it may actually happen openrc-run mtime predates such calculated deptree date and therefore cause transitional havoc in service starts after swclock kicks-in (flock issues): one exemple is described there.

What would be the best approach to avoid that?

  • set same arbitrary mtime for all open-rc files?
  • add openrc-run in the list of deptree inspected locations to set initial mtime?
  • have swclock use another default openrc file from deptree inspected locations, to set its first-time time reference in consistent manner?
@macmpi
Copy link
Author

macmpi commented Nov 29, 2023

@williamh referring to original 3688c85
How about sth like this which will never be backwards of current time:

if ! swclock "${swclock_file}" 2> /dev/null; then
curr_mtime="$(mktemp -q)"
swclock --warn "${curr_mtime}"
[ -f "${curr_mtime}" ] && rm -f "${curr_mtime}"
fi

macmpi added a commit to macmpi/openrc that referenced this issue Dec 8, 2023
A default reference file mtime may send the clock backwards of current deptree mtime.
Fixes OpenRC#675
@vapier
Copy link
Member

vapier commented Dec 8, 2023

we shouldn't need to create a file just to get the current time. swclock can easily run gettimeofday() to get it itself.

also, it sounds like you think we should never roll the clock backwards ? clocks can easily be set incorrectly in the future and we need to pull it back.

@macmpi
Copy link
Author

macmpi commented Dec 8, 2023

Thanks for feedback.

Would it then make more sense to add openrc-run in the list of files whose mtime is being monitored to set deptree reference then? (RC_INITDIR, RC_CONFDIR, RC_CONF,...), or use another reference file which is already in the monitored list (lowest key change, could be self init.d/swclock)?

clocks can easily be set incorrectly in the future and we need to pull it back.

Sure, it may happen for particular reasons.
However, currently it happens every time at first boot depending on historic changes of packaged files: seems a bit nondeterministic and unwanted: depending on history of openrc-run file changes relative to other packaged files changes, clock may go forward or backwards at runtime, differently across releases, and eventually trigger deptree flock issues...
Alternatively packaged file mtime could be set as some arbitrary time (release date, time 0, etc...), but has some constraints.

Thoughts?

EDIT: in fact, whatever packaged file reference is taken, clock will be sent (slightly) backwards of current time anyway... We could just make it stable & consistent across release.
But fundamentally, the approach on missing shutdown file might just be to warn, and set current time as reference.

@vapier
Copy link
Member

vapier commented Dec 8, 2023

the whole point of swclock is that the system doesn't have a hwclock to keep track of time. it sounds like your system has a hwclock (otherwise, how would time be preserved across boots?), so why are you using swclock in the first place ?

@macmpi
Copy link
Author

macmpi commented Dec 9, 2023

No my system does not have hwclock, and uses swclock.

I came across those unexpected time-shifts, while looking at detailed rc logs trying to figure-out why some service eventually warned about flock issue at start on a fresh system.
The fact that swclock does affect service start inconsistently, depending on openrc files mtime is quite unexpected and unpredictable for user (i.e. using openrc-run as reference, which is not part of deptree's monitored files).

Then this gated the next question: "why would clock even go backwards in such boot process, if we don't pick a file that forces it (like previous shutdown or any other at user's choice)?"
Would seem logical to let time flow "naturally", unless instructed differently: if reference file does not exist, then just keep going.
Bug referenced at 3688c85 did not really asked for a "reasonable" file reference, but just referred to an annoying service error. It could not error-out but just warn instead in such situation.
(Or maybe deptree mtime would be a more appropriate reasonable file reference, should one be needed)
Makes sense, or am I missing something?

@vapier
Copy link
Member

vapier commented Dec 14, 2023

i don't know what scenario you're hitting. if your system lacks an RTC, then from a cold or warm boot, the time should be initialized to epoch (1 Jan 1970). the only thing to move it forward in time is swclock. so how do you have a clock that is "current" that swclock is in turn rewinding ?

@macmpi
Copy link
Author

macmpi commented Dec 14, 2023

Yes, agreed: maybe my "current" statement is a bit confusing.
To clarify, I attach detailed rc_log.txt, coming from an Alpine cold boot sample.
openrc is run 4 times: first within initramfs, then in standard init (sysinit, boot, default) runlevels.
In my test, I inserted my_service as last within boot runlevel (depends after *) which merely starts seedrng service.

Lines 16/17 deptree mtime is set in reference to /etc/init.d/watchdog (July 27th)
boot runlevel begins line 97
Lines 100/101 swclock sets system time in reference to openrc-run (July 6th - hence before deptree July 27th )
(BTW in any case, is NOT shutdown time as in this cold boot there is no previous shutdown time); previously system clock was just running from epoch (1 Jan 1970) as expected.
my_service kicks-in line 152
Line 163 we see the unexpected seedrng flock issue at start.
(BTW this issue will not happen if either my_service explicitly runs rc-update --update before starting seedrng, or if my_service is moved to default runlevel: this is what initially gated my close attention to swclock time setting).

So, back to my finer-tuned comment/questions related to swclock init on such cold boot situation:

  • do we really want to force swclock to a fall-back reference file mtime if no previous shutdown / user preference file exists? (in any case, log message should probably be clearer on which is actually used) ?
  • if we use such fall-back reference, is openrc-run the most reasonable choice as it depends on openrc packaged file history changes (we see it may gate uncontrolled changes wrt deptree)?
    Wouldn't using current deptree reference be more appropriate in such case (by construction latest mtime of all), or some other strategy?

Appreciate your feedback.

@mingzym
Copy link
Contributor

mingzym commented Dec 15, 2023

in your case the cold start looks like a first time start? in this case the you don't have the swclock last run timestamp, and also no good reference-able log/history file I believe.

if it is not clean shutdown we don't have the timestamp saved, then save timestamp at the start of swclock is useless, you may have some log file newer than it.

so, I think we can use epoch time in case it is first time run, and try to find a newest time when no timestamp file. we need to treat timestamp file as a trusted source, we should not create at start of swclock, as if it is a not-clean shutdown/reboot, you can not trust this timestamp.
or we can stamp both start and shutdown? remove the shutdown timestamp when write down the start timestamp. we always trust the shutdown timestamp, and if no shutdown timestamp, find the time newer than start timestamp, else EPOCH.

@macmpi
Copy link
Author

macmpi commented Dec 20, 2023

in your case the cold start looks like a first time start? in this case the you don't have the swclock last run timestamp, and also no good reference-able log/history file I believe.

Yes indeed.
How about using /run/openrc/starting/swclock (instead of openrc-run) as reference in such case?
(other options might also be /run/openrc/deptree or /run/openrc/depconfig & adding first value from /proc/uptime)

@macmpi
Copy link
Author

macmpi commented Jan 10, 2024

@vapier any recommendation on how to address that scenario?
Thanks.

@macmpi
Copy link
Author

macmpi commented Sep 13, 2024

@williamh would you have any recommendation on how to deal with this scenario on first (cold) boot?
(clock going back & forth gating flock issues, due to openrc-run file used as default timestamp...which mtime is packaging dependent).
Thanks for consideration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants