feat: upgrade kubernetes client from 11.0.0 -> 24.2.0, implement List+Watch in KubeWatcher #32
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bodom0015
commented
Dec 18, 2023
Comment on lines
+29
to
+33
| #configuration = client.Configuration() | ||
| #api_batch_v1 = client.BatchV1Api(client.ApiClient(configuration)) | ||
| #api_v1 = client.CoreV1Api(client.ApiClient(configuration)) | ||
| api_batch_v1 = client.BatchV1Api() | ||
| api_v1 = client.CoreV1Api() |
There was a problem hiding this comment.
Overriding configuration has changed between versions, but this is not required for how we are using the K8S API client
bodom0015
changed the title
bodom0015
requested review from
matthewberry,
yashdeep97 and
samarthgupta1011
bodom0015
commented
Dec 19, 2023
Comment on lines
-73
to
+84
| # Resource version is used to keep track of stream progress (in case of resume) | ||
| # List all pods in watched namespace to get resource_version | ||
| namespaced_jobs: V1JobList = kubejob.api_batch_v1.list_namespaced_job(namespace=kubejob.get_namespace()) | ||
| resource_version = namespaced_jobs.metadata.resource_version if namespaced_jobs.metadata.resource_version else resource_version | ||
|
|
||
| # Then, watch for new events using the most recent resource_version | ||
| # Resource version is used to keep track of stream progress (in case of resume/retry) | ||
| k8s_event_stream = w.stream(func=kubejob.api_batch_v1.list_namespaced_job, | ||
| namespace=kubejob.get_namespace(), | ||
| timeout_seconds=timeout_seconds, | ||
| resource_version=resource_version) | ||
| resource_version=resource_version, | ||
| timeout_seconds=timeout_seconds) |
There was a problem hiding this comment.
Attempt to implement List+Watch pattern, as described here:
kubernetes-client/python#843 (comment)
This was referenced Jan 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Our Kubernetes Python client is horribly outdated
Intermittent HTTP 500 errors when connecting to Kube API. Once first encountered, this error loops endlessly
Approach
jobmgr.platform.moleculemaker.orgHow to Test
This image has been deployed to
job-manager-stagingCLEAN
MOLLI
Error Handling
With no way to reliably reproduce the error, all we can do is wait for a few days and watch the logs to see if the error surfaces again 😔
mmli1cluster:kubectl config use-context mmli1job-manager-staging:kubectl logs -f deploy/job-manager-staging -n staging