-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for custom flow decorators to prefect deploy
#14694
Conversation
CodSpeed Performance ReportMerging #14694 will not alter performanceComparing Summary
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just the one question, assuming they get a reasonable error in the end this looks great.
with pytest.raises(ScriptError): | ||
flow.fn() | ||
# Should still be callable | ||
assert flow() == "woof!" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if the flow uses the non-existent package? Like in this case, what if the flow was:
from not_a_module import not_a_function
from prefect import flow
@flow(description="Says woof!")
def dog():
return not_a_function('dog')
Would the user still get a reasonable error message? This was the original reasoning for the placeholder flow variant, so that a user attempting to execute the flow in an environment where the package was unavailable would be told so.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case, it'd raise a NameError
since not_a_funtion
isn't defined in the namespace the flow was loaded in. Do you think that's reasonable? I could wrap the flow's __call__
to reraise the ScriptError
, but I don't want to accidentally obfuscate exceptions from the flow's function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Name error is probably good enough for now, and we can revisit if it's confusing.
@@ -654,7 +650,7 @@ async def _run_single_deploy( | |||
deploy_config["work_pool"]["job_variables"]["image"] = "{{ build-image.image }}" | |||
|
|||
if not deploy_config.get("description"): | |||
deploy_config["description"] = flow_decorator_arguments.get("description") | |||
deploy_config["description"] = flow.description |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm loving this, much cleaner.
This PR alters how the client loads a flow from source code when dependencies are missing. Previously, the client would attempt to retrieve flow metadata from the
@flow
decorator and generate a parameter schema for the decorated function by traversing the abstract syntax tree (AST). This approach prevents the use of custom decorators that wrap the@flow
decorator.This PR instead adds a
safe_load_flow_from_entrypoint
function which is capable of loading the full flow after making modifications to the AST.safe_load_flow_from_entrypoint
will usesafe_load_namespace
to avoid raising exceptions when dependencies are not present on the current machine when loading a flow. If the flow depends on types or objects from missing dependencies, the AST will be updated to remove the annotations, causing loading to fail. This can generate a different parameter schema if the necessary dependencies are present or not.safe_load_flow_from_entrypoint
is used as a fallback withinload_flow_from_entrypoint,
so the behavior should not be altered in most cases.I recognize this implementation is complex and understand this PR might be rejected. If it is accepted, or you decide to go an alternative route without AST parsing, much of the old AST parsing implementation can be deprecated and/or removed in subsequent PRs.
Closes #14687
Example
Checklist
maintenance
,fix
,feature
,enhancement
,docs
.<link to issue>
"mint.json
.