Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kill_switch parameter to BatchTask #30

Merged
merged 1 commit into from
Apr 15, 2024

Conversation

dominictarro
Copy link
Owner

@dominictarro dominictarro commented Apr 15, 2024

Closes #26

Created a kill_switch mechanism to let BatchTask quit before running all tasks. it is useful for cases where the number of tasks to map is very high and failures may suggest the entire flow ought to cancel.

from prefect import flow, task
from prefecto.concurrency import BatchTask, CountSwitch

@task
def my_task(a, b):
    return a + b

@flow
def my_flow():
    bt = BatchTask(my_task, size=2, kill_switch=CountSwitch(3))
    # Should stop on the third batch, when 5 + "c" raises a ValueError
    bt.map(a=[1, 2, 3, 4, 5, 6, 7, 8], b=["a", 1, 2, "b", "c", 3, 4, 5])

@dominictarro dominictarro added the enhancement New feature or request label Apr 15, 2024
@dominictarro dominictarro self-assigned this Apr 15, 2024
@dominictarro
Copy link
Owner Author

Debating adding a traceback that relates the failed task's error/reason to the KillSwitchError.

@dominictarro dominictarro merged commit 7cc6631 into v1 Apr 15, 2024
4 checks passed
@dominictarro dominictarro deleted the feat-batch-terminate-condition branch April 15, 2024 23:08
dominictarro added a commit that referenced this pull request Apr 19, 2024
* Refactor v1 prep (#28)

* Make versioneer unvendored

* Remove testing module

* Remove serializers module

* Remove filesystems module

* Update project build definitions to be pyproject.toml based

* Move package listing to more universally understood requirements.txt

* Move to ruff formatting

* Clean up docs

* Fix: set ruff formatting config

* Refactor static code analysis action

* Refactor package/docs build and publish

* Refactor unit test action

* Fix conftest to treat harness as context manager

* Remove support for Python 3.13

* Add Python 3.12 to PyPI classifiers

* Don't invoke pytest directly, instead through python module optional argument

* Fix dependency install during test action

* Split package and docs build and fix dependency install

* Pattern and functions for lazy loading blocks (#29)

* Add `kill_switch` parameter to `BatchTask` to prevent queueing excessive, failing tasks (#30)

* Update docs for v1 (#32)

* Clean up mkdocs and module/class docstrings

* Add kill switches to the  example on the home page
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant