Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip data parsing when squeue times out #142

Merged
merged 2 commits into from
Jan 13, 2025
Merged

Skip data parsing when squeue times out #142

merged 2 commits into from
Jan 13, 2025

Conversation

jordap
Copy link
Collaborator

@jordap jordap commented Jan 8, 2025

The collector crashes when squeue times out since the runShellCommand with the squeue query returns None.

This PR ensures squeue isn't timing out before parsing the results by checking data is not None.

Not tested in production.

@koomie koomie added the bug Something isn't working label Jan 9, 2025
@jordap jordap force-pushed the jorda/check-squeue branch 2 times, most recently from 1493c30 to f9af940 Compare January 10, 2025 18:21
@jordap jordap force-pushed the jorda/check-squeue branch from f9af940 to 3c24d33 Compare January 10, 2025 18:23
@jordap jordap requested a review from koomie January 10, 2025 18:27
@jordap jordap marked this pull request as ready for review January 10, 2025 18:28
@jordap
Copy link
Collaborator Author

jordap commented Jan 10, 2025

Btw, another option here is to increase the squeue timeout, which is always 1 second.

Not mentioning that in the warning message because 1) there is no configuration option to change it and requires changing the code, 2) longer squeue timeouts increase the chances of mismatched readings between the first and second squeue, and 3) timeout needs to be consistent with the scrape interval so that scrape_interval >= squeue_timeout * 2 (since we are executing squeue twice).

The last point also makes me think we should recommend the minimum scrape interval with squeue mode to be at least 2 seconds.

@koomie koomie added this to the 1.2 milestone Jan 13, 2025
@koomie koomie merged commit 02ee8a9 into main Jan 13, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants