Boefjes combined schedulers integration #4015
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpbruinsslot
changed the base branch from
poc/mula/combined-schedulers
to
main
jpbruinsslot
changed the base branch from
main
to
poc/mula/combined-schedulers
jpbruinsslot
commented
Jan 8, 2025
| # TODO: check | ||
| if not response: | ||
| logger.debug("Queue %s empty", queue_type.value) | ||
| time.sleep(10) |
There was a problem hiding this comment.
Should these be time.sleep(self.settings.poll_interval) ?
jpbruinsslot
commented
Jan 8, 2025
| if not p_item: | ||
| logger.debug("Queue %s empty", queue.id) | ||
| continue | ||
| # TODO: check |
There was a problem hiding this comment.
Check what kind of response the scheduler gives when the queue is empty and handle this.
jpbruinsslot
commented
Jan 8, 2025
Comment on lines
+84
to
+91
| def pop_item(self, scheduler_id: str) -> PaginatedTasksResponse | None: | ||
| response = self._session.post(f"/schedulers/{scheduler_id}/pop?limit=1") | ||
| self._verify_response(response) | ||
|
|
||
| return TypeAdapter(PaginatedTasksResponse | None).validate_json(response.content) | ||
|
|
||
| def pop_items(self, scheduler_id: str, filters: dict[str, Any]) -> PaginatedTasksResponse | None: | ||
| response = self._session.post(f"/schedulers/{scheduler_id}/pop", json=filters) |
There was a problem hiding this comment.
Important to note is that the pop endpoint changed to return multiple Tasks, additionally it allows for more complex filtering options. I've, for now limited the response to 1 Task to resemble how it currently works, but this can be updated accordingly to what is desired.
…-schedulers-integration
jpbruinsslot
commented
Jan 9, 2025
| if task_queue.qsize() > self.settings.pool_size: | ||
| time.sleep(self.settings.worker_heartbeat) | ||
| return | ||
| logger.debug("Popping from queue %s", queue_type.value) | ||
|
|
There was a problem hiding this comment.
add this back in
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Warning
Please check the SOURCE PR and the associated issue(s) to get the background and full context of this PR. This PR is to be merged into that source PR.
/queueshas been refactored to the/schedulers/{scheduler_id}/popendpoint how the task runner is popping off tasks. We don't have to iterate over all the organizational queues and check if items are still on the queue. We now can just pop off tasks from the queue.Issue link
Closes #3961