Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: startupProbe to avoid a long initial delay seconds for liveness probe #174

Merged
merged 1 commit into from
Feb 9, 2024

Conversation

a7i
Copy link
Contributor

@a7i a7i commented Jan 25, 2024

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Kubernetes Docs:

The kubelet uses startup probes to know when a container application has started. If such a probe is configured, liveness and readiness probes do not start until it succeeds, making sure those probes don't interfere with the application startup. This can be used to adopt liveness checks on slow starting containers, avoiding them getting killed by the kubelet before they are up and running.

# -- Probe frequency in seconds
periodSeconds: 10
periodSeconds: 20
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readiness probe must fail before liveness probe. otherwise container stops receiving traffic and restarts at the same time.

@@ -133,7 +145,7 @@ opencost:
# -- Probe path
path: /healthz
# -- Number of seconds before probe is initiated
initialDelaySeconds: 120
initialDelaySeconds: 10
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally you should run readiness as soon as the container comes up

@a7i a7i force-pushed the amir/startupprobe branch from 7ec9de4 to 7e5c172 Compare January 25, 2024 15:14
Copy link
Member

@lmello lmello left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we were having issues in our environment, when we reduced the liveness probe timeout.

This change solves the issue in a more efficent way.

In the worst case scenario kubernetes wait up to 160 seconds for container start, but it will proceed as soon as the container pass the probe.

Best case scenario if it starts fast it will wait only 10 seconds.

This allows kubernetes to wait for the container startup in a more efficient way, soon as the container starts, then it benefits of a faster liveness probe and readiness probe.

@a7i a7i force-pushed the amir/startupprobe branch from 7e5c172 to c2897d5 Compare January 25, 2024 23:07
@mattray mattray merged commit b33e55d into opencost:main Feb 9, 2024
1 check passed
@a7i a7i deleted the amir/startupprobe branch February 10, 2024 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants