-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow flow parameter schema generation when dependencies are missing #13315
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for prefect-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
desertaxle
force-pushed
the
fix/flow-flow-with-missing-dependencies
branch
from
May 13, 2024 17:34
a085d62
to
d34425f
Compare
desertaxle
force-pushed
the
fix/flow-flow-with-missing-dependencies
branch
2 times, most recently
from
May 17, 2024 17:00
df1f676
to
52f574a
Compare
desertaxle
commented
May 28, 2024
desertaxle
commented
May 28, 2024
desertaxle
force-pushed
the
fix/flow-flow-with-missing-dependencies
branch
from
May 29, 2024 01:28
d5c70ee
to
b51640f
Compare
@serinamarie Added the docstrings that we identified as missing during our sync review. |
serinamarie
approved these changes
May 29, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔥
desertaxle
added a commit
that referenced
this pull request
May 29, 2024
6 tasks
This was referenced Jun 12, 2024
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new method of generating a parameter schema for a function so users and use
prefect deploy
when not all dependencies are available on the current machine.Our current method of generating a parameter schema requires us to load the flow function itself and get the signature using the built-in
inspect
module. This causes issues when not all dependencies are present because loading the flow causes the entire module the flow is located in to be run and can result in Python raising anImportError
.This PR updates this approach to allow a parameter schema given an entrypoint that points to a function with the new
parameter_schema_from_entrypoint
function. This new function will generate ainspect.Signature
object and docstring dictionary from the source code directly and use it with the existing machinery for generating a parameter schema.Under the hood, the new functionality will load the source code at the provided entrypoint and parse it using the
ast
module. A new utility calledsafe_load_namespace
will load each module-scoped import one by one and create a namespace dictionary for downstream use. Custom class definitions will also be included in the namespace. Any import errors will be caught and logged, but will not stop execution. The created namespace is used as context when walking the AST for the given function so that custom typing can be correctly resolved. In the case that typing cannot be resolved, the generatedSignature
will use no annotation to prevent interrupting execution. This can result in overly broad typing if the annotations rely on external types that aren't installed at deployment time.The diff is big, but it's mostly tests.
Closes #9512
Example
Given code stored at
flow.py
:The following call will generate the expected parameter schema with or without
panadas
installed:The dictionary format looks like this:
Checklist
<link to issue>
"maintenance
,fix
,feature
,enhancement
,docs
.For documentation changes:
netlify.toml
for files that are removed or renamed.For new functions or classes in the Python SDK:
mkdocs.yml
navigation.