Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prototype] AssetExecutionContext partition methods directly on the context #19032

Closed

Conversation

jamiedemaria
Copy link
Contributor

@jamiedemaria jamiedemaria commented Jan 4, 2024

Summary & Motivation

This PR is a prototype for a potential API for partition methods on the AssetExecutionContext. It reduces the number of partition-related methods on the context by eliminating output methods (since the partition for an output will be the same for all outputs produced by the asset). The names of the methods are also updated to be asset-centric rather than input/output-centric and they will accept asset keys rather than input/output names.

Hooli PR - dagster-io/hooli-data-eng-pipelines#59
DOP PR - https://github.com/dagster-io/internal/pull/7903

How I Tested These Changes

@jamiedemaria
Copy link
Contributor Author

jamiedemaria commented Jan 4, 2024

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

This stack of pull requests is managed by Graphite.

)

@public
def upstream_partition_keys(self, key: CoercibleToAssetKey) -> Sequence[str]:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make all of these accept a CoercibleToAssetDep maybe? that way we can pass definitions around vs string matching on asset key? or maybe doing this is sufficiently good pattern

@asset(...)
def upstream():
   ...

@asset(deps=[upstream], ...)
def downstream(context):
      context.upstream_partition_keys(upstream.key)

allowing CoercibleToAssetDep could cause confusing scoping issues like this

@asset(...)
def upstream():
   ...

@asset(...)
def downstream(context, upstream):
      context.upstream_partition_keys(upstream.key) # upstream doesn't have a key in this case since it's a loaded value

@jamiedemaria jamiedemaria force-pushed the jamie/partition-methods-asset-key-based branch from 97d5e6f to f062952 Compare January 10, 2024 17:13
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 9704201 to 2e86bcf Compare January 10, 2024 17:13
@jamiedemaria jamiedemaria marked this pull request as ready for review January 11, 2024 21:04

@public
@property
def has_partition_key(self) -> bool: # TODO - maybe this one can be replaced by something else?
Copy link
Contributor Author

@jamiedemaria jamiedemaria Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from my understanding, this method is mostly used to check if a run is partitioned or not. In that case, I think a name like is_partitioned, is_partitioned_materialization/execution/run is better

@jamiedemaria jamiedemaria force-pushed the jamie/partition-methods-asset-key-based branch from f062952 to 5331a95 Compare January 12, 2024 19:48
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 2e86bcf to 592f799 Compare January 12, 2024 19:50
@jamiedemaria jamiedemaria changed the base branch from jamie/partition-methods-asset-key-based to jamie/partition-example-base January 12, 2024 19:50
@@ -33,7 +33,7 @@ def relativedelta(*args, **kwargs):
metadata={"partition_expr": "LastModifiedDate"},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file will not get merged. It's an example of what some of these apis will look like in use

@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 04fd627 to 0e1fc3e Compare January 26, 2024 20:05
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 6dde54c to f5f6756 Compare January 26, 2024 20:05
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 0e1fc3e to dc6edf0 Compare January 29, 2024 15:39
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from f5f6756 to 782de6a Compare January 29, 2024 15:39
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from dc6edf0 to ccc4260 Compare January 29, 2024 16:30
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 782de6a to 2334f74 Compare January 29, 2024 16:30
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from ccc4260 to f416c05 Compare January 29, 2024 17:26
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 2334f74 to 9d449ab Compare January 29, 2024 17:27
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from f416c05 to 51d46aa Compare January 29, 2024 18:16
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 9d449ab to 902a82b Compare January 29, 2024 18:16
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 51d46aa to 2670364 Compare January 29, 2024 19:44
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 902a82b to 7aa0437 Compare January 29, 2024 19:44
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 2670364 to 2660075 Compare January 29, 2024 20:03
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 7aa0437 to baa4e2f Compare January 29, 2024 20:03
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from 2660075 to a5828e1 Compare January 30, 2024 17:23
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from baa4e2f to 587de6b Compare January 30, 2024 17:23
@erinkcochran87 erinkcochran87 removed their request for review January 30, 2024 17:49
@jamiedemaria jamiedemaria force-pushed the jamie/partition-example-base branch from a5828e1 to f5be566 Compare January 30, 2024 20:19
@jamiedemaria jamiedemaria force-pushed the jamie/partition-api-1 branch from 587de6b to e6792e3 Compare January 30, 2024 20:19
@jamiedemaria jamiedemaria marked this pull request as draft February 6, 2024 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant