feat(opensearch): capture logs from Dask cluster pods #616
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #616 +/- ##
==========================================
- Coverage 75.57% 75.44% -0.13%
==========================================
Files 17 17
Lines 1883 1898 +15
==========================================
+ Hits 1423 1432 +9
- Misses 460 466 +6
|
Alputer
added a commit
to Alputer/reana-workflow-controller
that referenced
this pull request
Nov 26, 2024
Alputer
force-pushed
the
dask-live-logs
branch
from
553cb69 to
a3c6170
Compare
Alputer
force-pushed
the
dask-live-logs
branch
from
a3c6170 to
7d9e4c5
Compare
Alputer
added a commit
to Alputer/reana-workflow-controller
that referenced
this pull request
Jan 20, 2025
Alputer
added a commit
to Alputer/reana-workflow-controller
that referenced
this pull request
Jan 21, 2025
Alputer
force-pushed
the
dask-live-logs
branch
from
7d9e4c5 to
0330a34
Compare
tiborsimko
reviewed
Jan 22, 2025
| @@ -63,11 +63,13 @@ def __init__( | |||
| os_client: OpenSearch | None = None, | |||
| job_index: str = "fluentbit-job_log", | |||
| workflow_index: str = "fluentbit-workflow_log", | |||
| dask_index: str = "fluentbit-dask_log", | |||
There was a problem hiding this comment.
todo: For the release notes, please amend the commit log headline scope to opensearch and capitalise Dask:
feat(opensearch): capture logs from Dask cluster pods (#616)
|
|
||
| def fetch_dask_worker_logs(self, workflow_id: str) -> str | None: | ||
| """ | ||
| Fetch logs of the workers of a Dask cluster. |
There was a problem hiding this comment.
Perhaps we can document -- here or elsewhere in the code documentation or in the main commit log message body -- that the logs collected from Dask workers are "propagated" to all REANA job logs, so that if there are N jobs using the same cluster, the Dask logs are currently "multiplicated" and they show under each job log. So that we don't forget this by-product of the current approach. (And we could open an issues about this later.)
Alputer
added a commit
to Alputer/reana-workflow-controller
that referenced
this pull request
Jan 22, 2025
Currently, the logs collected from Dask workers and scheduler are propogated to all REANA job logs. It is not ideal since Dask logs are multiplicated for different steps which use the same Dask cluster and the multiplicated logs are shown for each step which might be confusing and not user-friendly. This change requires a larger architectural change and is deferred to a future commit. Closes reanahub#610
Alputer
force-pushed
the
dask-live-logs
branch
from
0330a34 to
e9e272e
Compare
Alputer
changed the title
Alputer
added a commit
to Alputer/reana-workflow-controller
that referenced
this pull request
Jan 22, 2025
Currently, the logs collected from Dask workers and scheduler are propogated to all REANA job logs. It is not ideal since Dask logs are multiplicated for different steps which use the same Dask cluster and the multiplicated logs are shown for each step which might be confusing and not user-friendly. Seperating Dask logs and job logs requires a larger architectural change and is deferred to a future commit. Closes reanahub#610
Alputer
force-pushed
the
dask-live-logs
branch
from
e9e272e to
16d9bc3
Compare
Alputer
added a commit
to Alputer/reana-workflow-controller
that referenced
this pull request
Jan 22, 2025
This commit collects logs from Dask scheduler and workers and propagates\nthem to all REANA jobs that are using the same Dask cluster. This is not\nideal, since Dask logs can become thusly duplicated for different\nworkflow steps of the workflow, which could be confusing for the user.\n\nHowever, when a user uses Dask to parallelise the workflow jobs, usually\nthe workflow steps are defined only within Dask, so this situation does\nnot occur. Hence we can afford doing this in usual real-life conditions.\n\nSeparating Dask scheduler and worker logs from regular Kubernetes job\nlogs would require a larger architectural change and is therefore\ndeferred to a future commit. Closes reanahub#610
Alputer
force-pushed
the
dask-live-logs
branch
from
16d9bc3 to
fc03fb9
Compare
This commit collects logs from Dask scheduler and workers and propagates them to all REANA jobs that are using the same Dask cluster. This is not ideal, since Dask logs can become thusly duplicated for different workflow steps of the workflow, which could be confusing for the user. However, when a user uses Dask to parallelise the workflow jobs, usually the workflow steps are defined only within Dask, so this situation does not occur. Hence we can afford doing this in usual real-life conditions. Separating Dask scheduler and worker logs from regular Kubernetes job logs would require a larger architectural change and is therefore deferred to a future commit. Closes reanahub#610
tiborsimko
force-pushed
the
dask-live-logs
branch
from
fc03fb9 to
51fad95
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the collection of logs for Dask pods, namely scheduler and workers.
Closes #610