[backfill daemon run retries 3/n] retries of runs in completed backfills should not be considered part of the backfill

[jamiedemaria]

Summary & Motivation

If a run is retried after a backfill is complete, that run is given the backfill tag, but has no affect on the backfill itself. This can cause confusion. Imagine the scenario where a single asset-partition failed in a backfill. The backfill is complete and a user retries the failed asset and the retry succeeds. That retried run will show up in the list of runs for the backfill, but the status in the overview tab for partition will still be failed since the status is locked when the backfill completes.

We should be more strict about when run retries are considered part of the backfill. We decided in https://github.com/dagster-io/internal/discussions/12460 that retries that are launched while the backfill is in progress will be part of the backfill, but that retries that are launched after the backfill is complete should not be considered part of the backfill.

To make this change we need to remove the backfill tag from retried runs if the backfill is not in progress.

How I Tested These Changes

new unit tests

manually launched a retry of a run that was launched by a backfill after the backfill was complete. no backfill tags were added

Changelog

Manual retries of runs launched by backfills are no longer considered part of the backfill if the backfill is complete when the retry is launched.

[jamiedemaria]

• [backfill daemon run retries 3/n] retries of runs in completed backfills should not be considered part of the backfill #25900 👈 (View in Graphite)
• [backfill daemon run retries 2/n] backfill daemon incorporates retries runs when launching new runs #25853
• [backfill daemon run retries 1/n] update how we determine backfill completion to account for retried runs #25771
• [auto run retry tags] 3/n - when a run is retried, add theauto_retry_run_id tag #26054
• [auto run retry tags] 2/n - Add dagster/will_retry tag to runs that fail #25932
• [auto run retry tags] 1/n - make run_retries retry_on_asset_or_op_failure setting available on instance #26046
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[gibsondan]

this applies to the old code too but maybe this should be parent_run_tags_to_include?

[gibsondan]

given that this is the sole place that TAGS_TO_OMIT_ON_RETRY and TAGS_TO_MAYBE_OMIT_ON_RETRY are used I think we could keep it as one list of tags vs. splitting them out into two lists? should it all just be TAGS_TO_MAYBE_OMIT_ON_RETRY?

[jamiedemaria]

and then just not have a condition to add back the tags that are in the TAGS_TO_OMIT_ON_RETRY list? that sounds good to me