Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prototype] AssetExecutionContext partition methods given as named tuple #19037

Closed

Conversation

jamiedemaria
Copy link
Contributor

@jamiedemaria jamiedemaria commented Jan 4, 2024

Summary & Motivation

This PR is a prototype for a potential API for partition methods on the AssetExecutionContext. Rather than fetching individual properties about the partition from different methods, the user will fetch a PartitionInfo (name tbd) object from the context that will contain the relevant partition properties.

Hooli PR - dagster-io/hooli-data-eng-pipelines#58
DOP PR - https://github.com/dagster-io/internal/pull/7902

issues with this API design:
time_window on the PartitionInfo named tuple needs to be an optional field so that static partitions can have it set to None. this causes some linting errors in the code base when you want to access the start or end properties of the time window since the retrieved time_window may be None

How I Tested These Changes

@jamiedemaria
Copy link
Contributor Author

jamiedemaria commented Jan 4, 2024

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

This stack of pull requests is managed by Graphite.

@jamiedemaria jamiedemaria force-pushed the jamie/partition-methods-asset-key-based branch from 97d5e6f to f062952 Compare January 10, 2024 17:13
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from dbd4f00 to 90ceb0f Compare January 10, 2024 17:13
@jamiedemaria jamiedemaria marked this pull request as ready for review January 11, 2024 21:03
@jamiedemaria jamiedemaria force-pushed the jamie/partition-methods-asset-key-based branch from f062952 to 5331a95 Compare January 12, 2024 19:48
@jamiedemaria jamiedemaria changed the base branch from jamie/partition-methods-asset-key-based to jamie/partition-example-base January 12, 2024 19:49
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from 90ceb0f to e705b81 Compare January 12, 2024 19:49
@@ -33,7 +33,7 @@ def relativedelta(*args, **kwargs):
metadata={"partition_expr": "LastModifiedDate"},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file will not get merged. It's an example of what some of these apis will look like in use

@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 04fd627 to 0e1fc3e Compare January 26, 2024 20:05
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from 3d4bed9 to ab9a9eb Compare January 26, 2024 20:05
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 0e1fc3e to dc6edf0 Compare January 29, 2024 15:39
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from ab9a9eb to a44e758 Compare January 29, 2024 15:39
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from dc6edf0 to ccc4260 Compare January 29, 2024 16:30
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from a44e758 to 0fa4a8f Compare January 29, 2024 16:30
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from ccc4260 to f416c05 Compare January 29, 2024 17:26
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from 0fa4a8f to 25083e3 Compare January 29, 2024 17:26
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from f416c05 to 51d46aa Compare January 29, 2024 18:16
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from 25083e3 to e285806 Compare January 29, 2024 18:16
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 51d46aa to 2670364 Compare January 29, 2024 19:44
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from e285806 to fc01971 Compare January 29, 2024 19:45
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 2670364 to 2660075 Compare January 29, 2024 20:03
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from fc01971 to 5086c16 Compare January 29, 2024 20:03
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 2660075 to a5828e1 Compare January 30, 2024 17:23
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from 5086c16 to d6268c3 Compare January 30, 2024 17:23
@erinkcochran87 erinkcochran87 removed their request for review January 30, 2024 17:49
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from a5828e1 to f5be566 Compare January 30, 2024 20:19
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-2 branch from d6268c3 to c2da8af Compare January 30, 2024 20:19
@jamiedemaria jamiedemaria marked this pull request as draft February 6, 2024 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant