Skip to content

Commit

Permalink
[DA] Update docs (#26262)
Browse files Browse the repository at this point in the history
## Summary & Motivation

This splits up the existing customizing-automation-conditions page into
two. The first is a slimmed-down version of the original page that stays
much more focused on specific use cases. I also took the liberty of
backing out of the original sin of directly embedding python into the
mdx (now it's broken out into separate python files, at least for the
new stuff).

The second page is a more advanced conceptual guide for the ins and outs
of AutomationConditions. This is not necessary information for the
majority of users, but I think it is valuable to have in some place to
give interested users a more detailed conceptual understanding of how
the system fits together, especially if they have highly-specific use
cases that require deep manipulation of the APIs.

Also, some of the existing code examples in the
cusotmizing-automation-conditions page are a bit complex, and will serve
as good motivating examples for some lightweight chagnes to the core
APIs to make those examples simpler


## How I Tested These Changes

## Changelog

NOCHANGELOG
  • Loading branch information
OwenKephart authored Dec 5, 2024
1 parent c60f5d4 commit 12b54a1
Show file tree
Hide file tree
Showing 10 changed files with 325 additions and 163 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
---
title: "Advanced: Automation Conditions Explained | Dagster Docs"
description: "Learn the details of how Automation Conditions are evaluated"
---

# Automation conditions explained

[Declarative Automation](/concepts/automation/declarative-automation) includes pre-built conditions to handle common use cases, such as executing on a periodic schedule or whenever an upstream dependency updates, but the core system is extremely flexible and can be tailored to your specific needs.

By the end of this guide, you'll understand how <PyObject object="AutomationCondition" pluralize /> work and how to create your own custom conditions.

---

## Prerequisites

Before continuing, you should be familiar with:

- [Asset definitions](/concepts/assets/software-defined-assets)
- [Declarative Automation](/concepts/automation/declarative-automation)
- [Customizing Automation Conditions](/concepts/automation/declarative-automation/customizing-automation-conditions)

---

## How it works

Each <PyObject object="AutomationCondition" /> consists of a set of **operands** and various **operators**. To create conditions that suit your needs, you can combine the operators and operands listed below. For example:

```python
from dagster import AutomationCondition

in_progress_or_failed_parents = AutomationCondition.any_deps_match(
AutomationCondition.in_progress() | AutomationCondition.failed()
)
```

This condition translates to **Any upstream dependencies (parents) are part of an in-progress run or failed during the latest run**.

### Operands

Operands are base conditions which can be true or false about a given target. For partitioned assets, the target will be a given partition of the asset.

| Operand | Description |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `AutomationCondition.missing` | Target has not been executed |
| `AutomationCondition.in_progress` | Target is part of an in-progress run |
| `AutomationCondition.execution_failed` | Target failed to be executed in its latest run |
| `AutomationCondition.newly_updated` | Target was updated since the previous evaluation |
| `AutomationCondition.newly_requested` | Target was requested on the previous evaluation |
| `AutomationCondition.code_version_changed` | Target has a new code version since the previous evaluation |
| `AutomationCondition.cron_tick_passed` | A new tick of the provided cron schedule occurred since the previous evaluation |
| `AutomationCondition.in_latest_time_window` | Target falls within the latest time window of the asset’s <PyObject object="PartitionsDefinition" />, if applicable. |
| `AutomationCondition.will_be_requested` | Target will be requested in this tick |
| `AutomationCondition.initial_evaluation` | This is the first evaluation of this condition |

### Operators

The above conditions can be built into more complex expressions using the following operators:

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "40%",
}}
>
Operator
</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<code>~</code> (tilde)
</td>
<td>
NOT; condition is not true; ex: <code>~A</code>
</td>
</tr>
<tr>
<td>
<code>|</code> (pipe)
</td>
<td>
OR; either condition is true; ex: <code>A | B</code>
</td>
</tr>
<tr>
<td>
<code>&</code> (ampersand)
</td>
<td>
AND; both conditions are true; ex: <code>A & B</code>
</td>
</tr>
<tr>
<td>
<code>A.newly_true()</code>
</td>
<td>Condition A was false on the previous evaluation and is now true.</td>
</tr>
<tr>
<td>
<code>A.since(B)</code>
</td>
<td>Condition A became true more recently than Condition B.</td>
</tr>
<tr>
<td>
<code>AutomationCondition.any_deps_match(A)</code>
</td>
<td>
Condition A is true for any upstream partition. Can be used with{" "}
<code>.allow()</code> and <code>.ignore()</code> to target specific
upstream assets. Refer to the{" "}
<a href="#targeting-dependencies">Targeting dependencies</a> section for
an example.
</td>
</tr>
<tr>
<td>
<code>AutomationCondition.all_deps_match(A)</code>
</td>
<td>
Condition A is true for at least one partition of each upstream asset.
Can be used with <code>.allow()</code> and <code>.ignore()</code> to
target specific upstream assets. Refer to the{" "}
<a href="#targeting-dependencies">Targeting dependencies</a> section for
an example.
</td>
</tr>
<tr>
<td>
<code>AutomationCondition.any_downstream_condition()</code>
</td>
<td>
Any <PyObject object="AutomationCondition" /> on a downstream asset
evaluates to true
</td>
</tr>
</tbody>
</table>

### Composite conditions

Finally, there are a set of pre-built conditions which make it easier to construct common combinations of the above conditions.

| Condition | Description |
| ------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `AutomationCondition.any_deps_updated` | Any dependencies have been updated since the previous evaluation |
| `AutomationCondition.any_deps_missing` | Any dependencies have never been materialized or observed |
| `AutomationCondition.any_deps_in_progress` | Any dependencies are part of an in-progress run |
| `AutomationCondition.all_deps_updated_since_cron` | All dependencies have been updated since the latest tick of the provided cron schedule |

---

## Evaluations

Evaluation of each automation condition is handled by an <PyObject object="AutomationConditionSensorDefinition" />. By default, a sensor with the name `default_automation_condition_sensor` will be available in all code locations that have at least one asset with an `AutomationCondition`. This sensor will evaluate all available conditions every 30 seconds, and launch runs for any conditions that evaluate to true at that time.

Because evaluations happen at discrete times, and not continuously, this means that many of the above conditions are defined in relation to these evaluation ticks. For example, `AutomationCondition.cron_tick_passed()` becomes true on the first evaluation after a cron schedule tick is passed.

## Statuses and events

There are two general categories of AutomationConditions:

- **Statuses** are persistent states that are and will be true for some period of time. For example, the `AutomationCondition.missing()` condition will be true only if an asset partition has never been materialized or observed.

- **Events** are transient and reflect something that may only be true for an instant. For example, the `AutomationCondition.newly_updated()` condition will be true only if an asset partition was materialized since the previous evaluation.

Using the `<A>.since(<B>)` operator, you can create conditions that detect if one event has happened more recently than another. Think of this as converting two events to a status - in this case, `A has occurred more recently than B` - as this will stay true for some period of time. This operator becomes true whenever `<A>` is true, and will remain true until `<B>` is also true. Conversely, it can also be useful to convert statuses to events. For example, the default `eager()` condition ensures that Dagster only tries to materialize a missing asset partition once using the following sub-condition:

```python
import dagster as dg

dg.AutomationCondition.missing().newly_true().since_last_handled()
```

By using the `<A>.newly_true()` operator, you can turn the status of _"being missing"_ into a single event, specifically the point in time where an asset partition entered the _missing_ state. This is done because an asset partition will generally remain missing for several evaluations after a run is initially requested, as that run spins up and does the work required to materialize the asset. To avoid continually requesting partitions during this time, this condition is designed to only be true from the point in time that the partition becomes missing to the point in time that we request a run to materialize it. After that point in time, the event is considered to be "handled", and the subcondition will remain false.

## Run Grouping

AutomationConditions generally depend on the status of their parent assets. For example, `AutomationCondition.eager()` executes after a parent updates, and `AutomationCondition.on_cron()` only executes after all parents have updated since a given cron schedule tick.

However, when you have multiple assets in a sequence, all with conditions which depend on the state of their parents, it would be inconvenient for each asset in that sequence to be executed in its own independent run. Ideally, if you have multiple eager assets in a chain, an update to the first would create a single run that targets all downstream assets, even though the dependencies of those assets haven't technically updated yet. The intuition here is that if we know we plan to update an asset on this evaluation, then downstream assets can treat that the same as if the asset already did update.

This handling is included automatically in the composite conditions `AutomationCondition.any_deps_updated()` and `AutomationCondition.any_deps_missing()`, which both rely on `AutomationCondition.will_be_requested()` to find asset partitions that will be executed on this tick, and can be grouped into the same run as the currently-evaluated asset.

---
Loading

1 comment on commit 12b54a1

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-6tzsv2mjt-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 12b54a1.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.