Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for custom flow decorators to prefect deploy #14694

Merged
merged 11 commits into from
Jul 29, 2024

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented Jul 22, 2024

This PR alters how the client loads a flow from source code when dependencies are missing. Previously, the client would attempt to retrieve flow metadata from the @flow decorator and generate a parameter schema for the decorated function by traversing the abstract syntax tree (AST). This approach prevents the use of custom decorators that wrap the @flow decorator.

This PR instead adds a safe_load_flow_from_entrypoint function which is capable of loading the full flow after making modifications to the AST. safe_load_flow_from_entrypoint will use safe_load_namespace to avoid raising exceptions when dependencies are not present on the current machine when loading a flow. If the flow depends on types or objects from missing dependencies, the AST will be updated to remove the annotations, causing loading to fail. This can generate a different parameter schema if the necessary dependencies are present or not.

safe_load_flow_from_entrypoint is used as a fallback within load_flow_from_entrypoint, so the behavior should not be altered in most cases.

I recognize this implementation is complex and understand this PR might be rejected. If it is accepted, or you decide to go an alternative route without AST parsing, much of the old AST parsing implementation can be deprecated and/or removed in subsequent PRs.

Closes #14687

Example

Checklist

  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.
  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

@desertaxle desertaxle added the fix A fix for a bug in an existing feature label Jul 22, 2024
Copy link

codspeed-hq bot commented Jul 22, 2024

CodSpeed Performance Report

Merging #14694 will not alter performance

Comparing simplify-flow-parameter-extraction (140621b) with main (85bea69)

Summary

✅ 5 untouched benchmarks

@desertaxle desertaxle marked this pull request as ready for review July 23, 2024 14:49
@desertaxle desertaxle requested a review from cicdw as a code owner July 23, 2024 14:49
@github-actions github-actions bot added the bug Something isn't working label Jul 23, 2024
@cicdw cicdw requested a review from bunchesofdonald July 29, 2024 13:49
Copy link
Contributor

@bunchesofdonald bunchesofdonald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just the one question, assuming they get a reasonable error in the end this looks great.

with pytest.raises(ScriptError):
flow.fn()
# Should still be callable
assert flow() == "woof!"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if the flow uses the non-existent package? Like in this case, what if the flow was:

        from not_a_module import not_a_function
        from prefect import flow
        @flow(description="Says woof!")
        def dog():
            return not_a_function('dog')

Would the user still get a reasonable error message? This was the original reasoning for the placeholder flow variant, so that a user attempting to execute the flow in an environment where the package was unavailable would be told so.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, it'd raise a NameError since not_a_funtion isn't defined in the namespace the flow was loaded in. Do you think that's reasonable? I could wrap the flow's __call__ to reraise the ScriptError, but I don't want to accidentally obfuscate exceptions from the flow's function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name error is probably good enough for now, and we can revisit if it's confusing.

@@ -654,7 +650,7 @@ async def _run_single_deploy(
deploy_config["work_pool"]["job_variables"]["image"] = "{{ build-image.image }}"

if not deploy_config.get("description"):
deploy_config["description"] = flow_decorator_arguments.get("description")
deploy_config["description"] = flow.description
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm loving this, much cleaner.

@desertaxle desertaxle merged commit a364933 into main Jul 29, 2024
30 checks passed
@desertaxle desertaxle deleted the simplify-flow-parameter-extraction branch July 29, 2024 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working fix A fix for a bug in an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

prefect deploy doesn't respect arguments passed through custom decorators
2 participants