-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Summary & Motivation This splits up the existing customizing-automation-conditions page into two. The first is a slimmed-down version of the original page that stays much more focused on specific use cases. I also took the liberty of backing out of the original sin of directly embedding python into the mdx (now it's broken out into separate python files, at least for the new stuff). The second page is a more advanced conceptual guide for the ins and outs of AutomationConditions. This is not necessary information for the majority of users, but I think it is valuable to have in some place to give interested users a more detailed conceptual understanding of how the system fits together, especially if they have highly-specific use cases that require deep manipulation of the APIs. Also, some of the existing code examples in the cusotmizing-automation-conditions page are a bit complex, and will serve as good motivating examples for some lightweight chagnes to the core APIs to make those examples simpler ## How I Tested These Changes ## Changelog NOCHANGELOG
- Loading branch information
1 parent
c60f5d4
commit 12b54a1
Showing
10 changed files
with
325 additions
and
163 deletions.
There are no files selected for viewing
195 changes: 195 additions & 0 deletions
195
.../concepts/automation/declarative-automation/automation-conditions-explained.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,195 @@ | ||
--- | ||
title: "Advanced: Automation Conditions Explained | Dagster Docs" | ||
description: "Learn the details of how Automation Conditions are evaluated" | ||
--- | ||
|
||
# Automation conditions explained | ||
|
||
[Declarative Automation](/concepts/automation/declarative-automation) includes pre-built conditions to handle common use cases, such as executing on a periodic schedule or whenever an upstream dependency updates, but the core system is extremely flexible and can be tailored to your specific needs. | ||
|
||
By the end of this guide, you'll understand how <PyObject object="AutomationCondition" pluralize /> work and how to create your own custom conditions. | ||
|
||
--- | ||
|
||
## Prerequisites | ||
|
||
Before continuing, you should be familiar with: | ||
|
||
- [Asset definitions](/concepts/assets/software-defined-assets) | ||
- [Declarative Automation](/concepts/automation/declarative-automation) | ||
- [Customizing Automation Conditions](/concepts/automation/declarative-automation/customizing-automation-conditions) | ||
|
||
--- | ||
|
||
## How it works | ||
|
||
Each <PyObject object="AutomationCondition" /> consists of a set of **operands** and various **operators**. To create conditions that suit your needs, you can combine the operators and operands listed below. For example: | ||
|
||
```python | ||
from dagster import AutomationCondition | ||
|
||
in_progress_or_failed_parents = AutomationCondition.any_deps_match( | ||
AutomationCondition.in_progress() | AutomationCondition.failed() | ||
) | ||
``` | ||
|
||
This condition translates to **Any upstream dependencies (parents) are part of an in-progress run or failed during the latest run**. | ||
|
||
### Operands | ||
|
||
Operands are base conditions which can be true or false about a given target. For partitioned assets, the target will be a given partition of the asset. | ||
|
||
| Operand | Description | | ||
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | ||
| `AutomationCondition.missing` | Target has not been executed | | ||
| `AutomationCondition.in_progress` | Target is part of an in-progress run | | ||
| `AutomationCondition.execution_failed` | Target failed to be executed in its latest run | | ||
| `AutomationCondition.newly_updated` | Target was updated since the previous evaluation | | ||
| `AutomationCondition.newly_requested` | Target was requested on the previous evaluation | | ||
| `AutomationCondition.code_version_changed` | Target has a new code version since the previous evaluation | | ||
| `AutomationCondition.cron_tick_passed` | A new tick of the provided cron schedule occurred since the previous evaluation | | ||
| `AutomationCondition.in_latest_time_window` | Target falls within the latest time window of the asset’s <PyObject object="PartitionsDefinition" />, if applicable. | | ||
| `AutomationCondition.will_be_requested` | Target will be requested in this tick | | ||
| `AutomationCondition.initial_evaluation` | This is the first evaluation of this condition | | ||
|
||
### Operators | ||
|
||
The above conditions can be built into more complex expressions using the following operators: | ||
|
||
<table | ||
className="table" | ||
style={{ | ||
width: "100%", | ||
}} | ||
> | ||
<thead> | ||
<tr> | ||
<th | ||
style={{ | ||
width: "40%", | ||
}} | ||
> | ||
Operator | ||
</th> | ||
<th>Description</th> | ||
</tr> | ||
</thead> | ||
<tbody> | ||
<tr> | ||
<td> | ||
<code>~</code> (tilde) | ||
</td> | ||
<td> | ||
NOT; condition is not true; ex: <code>~A</code> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>|</code> (pipe) | ||
</td> | ||
<td> | ||
OR; either condition is true; ex: <code>A | B</code> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>&</code> (ampersand) | ||
</td> | ||
<td> | ||
AND; both conditions are true; ex: <code>A & B</code> | ||
</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>A.newly_true()</code> | ||
</td> | ||
<td>Condition A was false on the previous evaluation and is now true.</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>A.since(B)</code> | ||
</td> | ||
<td>Condition A became true more recently than Condition B.</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>AutomationCondition.any_deps_match(A)</code> | ||
</td> | ||
<td> | ||
Condition A is true for any upstream partition. Can be used with{" "} | ||
<code>.allow()</code> and <code>.ignore()</code> to target specific | ||
upstream assets. Refer to the{" "} | ||
<a href="#targeting-dependencies">Targeting dependencies</a> section for | ||
an example. | ||
</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>AutomationCondition.all_deps_match(A)</code> | ||
</td> | ||
<td> | ||
Condition A is true for at least one partition of each upstream asset. | ||
Can be used with <code>.allow()</code> and <code>.ignore()</code> to | ||
target specific upstream assets. Refer to the{" "} | ||
<a href="#targeting-dependencies">Targeting dependencies</a> section for | ||
an example. | ||
</td> | ||
</tr> | ||
<tr> | ||
<td> | ||
<code>AutomationCondition.any_downstream_condition()</code> | ||
</td> | ||
<td> | ||
Any <PyObject object="AutomationCondition" /> on a downstream asset | ||
evaluates to true | ||
</td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
### Composite conditions | ||
|
||
Finally, there are a set of pre-built conditions which make it easier to construct common combinations of the above conditions. | ||
|
||
| Condition | Description | | ||
| ------------------------------------------------- | -------------------------------------------------------------------------------------- | | ||
| `AutomationCondition.any_deps_updated` | Any dependencies have been updated since the previous evaluation | | ||
| `AutomationCondition.any_deps_missing` | Any dependencies have never been materialized or observed | | ||
| `AutomationCondition.any_deps_in_progress` | Any dependencies are part of an in-progress run | | ||
| `AutomationCondition.all_deps_updated_since_cron` | All dependencies have been updated since the latest tick of the provided cron schedule | | ||
|
||
--- | ||
|
||
## Evaluations | ||
|
||
Evaluation of each automation condition is handled by an <PyObject object="AutomationConditionSensorDefinition" />. By default, a sensor with the name `default_automation_condition_sensor` will be available in all code locations that have at least one asset with an `AutomationCondition`. This sensor will evaluate all available conditions every 30 seconds, and launch runs for any conditions that evaluate to true at that time. | ||
|
||
Because evaluations happen at discrete times, and not continuously, this means that many of the above conditions are defined in relation to these evaluation ticks. For example, `AutomationCondition.cron_tick_passed()` becomes true on the first evaluation after a cron schedule tick is passed. | ||
|
||
## Statuses and events | ||
|
||
There are two general categories of AutomationConditions: | ||
|
||
- **Statuses** are persistent states that are and will be true for some period of time. For example, the `AutomationCondition.missing()` condition will be true only if an asset partition has never been materialized or observed. | ||
|
||
- **Events** are transient and reflect something that may only be true for an instant. For example, the `AutomationCondition.newly_updated()` condition will be true only if an asset partition was materialized since the previous evaluation. | ||
|
||
Using the `<A>.since(<B>)` operator, you can create conditions that detect if one event has happened more recently than another. Think of this as converting two events to a status - in this case, `A has occurred more recently than B` - as this will stay true for some period of time. This operator becomes true whenever `<A>` is true, and will remain true until `<B>` is also true. Conversely, it can also be useful to convert statuses to events. For example, the default `eager()` condition ensures that Dagster only tries to materialize a missing asset partition once using the following sub-condition: | ||
|
||
```python | ||
import dagster as dg | ||
|
||
dg.AutomationCondition.missing().newly_true().since_last_handled() | ||
``` | ||
|
||
By using the `<A>.newly_true()` operator, you can turn the status of _"being missing"_ into a single event, specifically the point in time where an asset partition entered the _missing_ state. This is done because an asset partition will generally remain missing for several evaluations after a run is initially requested, as that run spins up and does the work required to materialize the asset. To avoid continually requesting partitions during this time, this condition is designed to only be true from the point in time that the partition becomes missing to the point in time that we request a run to materialize it. After that point in time, the event is considered to be "handled", and the subcondition will remain false. | ||
|
||
## Run Grouping | ||
|
||
AutomationConditions generally depend on the status of their parent assets. For example, `AutomationCondition.eager()` executes after a parent updates, and `AutomationCondition.on_cron()` only executes after all parents have updated since a given cron schedule tick. | ||
|
||
However, when you have multiple assets in a sequence, all with conditions which depend on the state of their parents, it would be inconvenient for each asset in that sequence to be executed in its own independent run. Ideally, if you have multiple eager assets in a chain, an update to the first would create a single run that targets all downstream assets, even though the dependencies of those assets haven't technically updated yet. The intuition here is that if we know we plan to update an asset on this evaluation, then downstream assets can treat that the same as if the asset already did update. | ||
|
||
This handling is included automatically in the composite conditions `AutomationCondition.any_deps_updated()` and `AutomationCondition.any_deps_missing()`, which both rely on `AutomationCondition.will_be_requested()` to find asset partitions that will be executed on this tick, and can be grouped into the same run as the currently-evaluated asset. | ||
|
||
--- |
Oops, something went wrong.
12b54a1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deploy preview for dagster-docs ready!
✅ Preview
https://dagster-docs-6tzsv2mjt-elementl.vercel.app
https://master.dagster.dagster-docs.io
Built with commit 12b54a1.
This pull request is being automatically deployed with vercel-action