Skip Mode #6039

wxtim · 2024-03-27T12:49:18Z

Closes #5641
closes #5820

This branch includes (marked against skip mode proposal):

A new task run mode called "skip" to sit alongside the existing "live" and "simulation" and "dummy" modes.
Skip mode settings in `[runtime][<namespace>][skip].
Task-level control of the run mode - allowing run mode of tasks to override that of the workflow using [runtime][<namespace>]run mode.
a. Broadcast can change run mode for future task job submissions.
b. Cylc Validate and lint will warn about the setting not being live.
cylc set --out skip sets outputs from skip mode.
Tests to ensure that run mode = skip respects is_held flag.
Tests to ensure that force triggering a task will not override the run mode.

Extras
7. Run Mode is available as an task attribute in the UI
8. When tasks are run in skip mode, the prerequisites which correspond to the outputs they generate should be marked as satisfied by skip mode rather than satisfied naturally for provenance reasons. For the purpose of cylc remove logic, satisfied by skip mode should be treated the same as satisfied naturally.

There are two extensions, which I haven't dealt with yet, because I want to ensure that the basic functionality works, and move to the substantial documentation PR which need follow this.

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
CHANGES.md entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
Feature: PR against master

cylc/flow/cfgspec/workflow.py

cylc/flow/prerequisite.py

cylc/flow/scheduler_cli.py

cylc/flow/task_state.py

cylc/flow/scheduler.py

cylc/flow/task_pool.py

cylc/flow/network/schema.py

cylc/flow/task_state.py

cylc/flow/unicode_rules.py

cylc/flow/task_proxy.py

cylc/flow/task_pool.py

cylc/flow/run_modes/skip.py

cylc/flow/run_modes/nonlive.py

cylc/flow/cfgspec/workflow.py

Make `cylc set --pre` & `cylc set --out skip` not override already-satisfied outputs

hjoliver

I'm part way through this behemoth. Code LGreatTM, working well.

changes.d/6039.feat.md

cylc/flow/cfgspec/workflow.py

cylc/flow/run_modes/__init__.py

cylc/flow/scheduler.py

Co-authored-by: Hilary James Oliver <[email protected]>

cylc/flow/task_events_mgr.py

Co-authored-by: Ronnie Dutta <[email protected]>

MetRonnie

🥂

hjoliver

One small suggestion. Great job @wxtim 🎉

cylc/flow/scheduler_cli.py

Co-authored-by: Ronnie Dutta <[email protected]>

Co-authored-by: Hilary James Oliver <[email protected]>

cylc/flow/cfgspec/workflow.py

changes.d/6039.feat.md

Co-authored-by: Oliver Sanders <[email protected]>

oliver-sanders · 2024-12-03T17:49:38Z

🎉

Congrats Tim!

wxtim marked this pull request as draft March 27, 2024 12:57

wxtim force-pushed the feature.skip_mode branch 2 times, most recently from b43d296 to d26c315 Compare March 28, 2024 09:11

oliver-sanders added this to the cylc-8.x milestone Apr 2, 2024

wxtim self-assigned this Apr 22, 2024

wxtim force-pushed the feature.skip_mode branch from d378e3a to e9c2e25 Compare April 22, 2024 13:57

wxtim requested review from oliver-sanders, markgrahamdawson and MetRonnie and removed request for markgrahamdawson April 23, 2024 10:00

wxtim marked this pull request as ready for review April 23, 2024 14:26

MetRonnie modified the milestones: 8.x, 8.4.0 Apr 23, 2024

MetRonnie reviewed Apr 23, 2024

View reviewed changes

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

MetRonnie added the config change Involves a change to global or workflow config label Apr 23, 2024

wxtim requested a review from MetRonnie April 24, 2024 15:10

wxtim force-pushed the feature.skip_mode branch 4 times, most recently from 3d002ee to dd0a9fc Compare April 25, 2024 12:56

wxtim force-pushed the feature.skip_mode branch from dd0a9fc to 19faa47 Compare May 2, 2024 12:46

MetRonnie reviewed May 8, 2024

View reviewed changes

wxtim force-pushed the feature.skip_mode branch 3 times, most recently from 0d42eab to c37ef72 Compare May 8, 2024 12:43

wxtim requested a review from MetRonnie May 8, 2024 13:47

wxtim force-pushed the feature.skip_mode branch 2 times, most recently from 3839a8a to d7ee653 Compare May 9, 2024 08:50

wxtim force-pushed the feature.skip_mode branch from 6677605 to 8de3073 Compare May 17, 2024 12:14

MetRonnie reviewed May 20, 2024

View reviewed changes

MetRonnie mentioned this pull request Nov 29, 2024

Make cylc set --pre & cylc set --out skip not override already-satisfied outputs wxtim/cylc#71

Merged

Merge pull request #71 from MetRonnie/prereq

3504e63

Make `cylc set --pre` & `cylc set --out skip` not override already-satisfied outputs

wxtim requested a review from MetRonnie November 29, 2024 19:52

hjoliver reviewed Dec 2, 2024

View reviewed changes

wxtim force-pushed the feature.skip_mode branch from b1aca0d to 68cfd1c Compare December 2, 2024 10:48

wxtim requested a review from hjoliver December 2, 2024 10:52

Suggestions from Hilary

f89719d

Co-authored-by: Hilary James Oliver <[email protected]>

wxtim force-pushed the feature.skip_mode branch from 68cfd1c to f89719d Compare December 2, 2024 10:59

MetRonnie reviewed Dec 2, 2024

View reviewed changes

cylc/flow/task_events_mgr.py Outdated Show resolved Hide resolved

wxtim and others added 2 commits December 2, 2024 16:20

Update cylc/flow/cfgspec/workflow.py

644a41c

Co-authored-by: Ronnie Dutta <[email protected]>

record run times for Dummy and Sim mode tasks

2304d57

wxtim requested a review from MetRonnie December 2, 2024 16:27

fix refs

5a5432a

wxtim force-pushed the feature.skip_mode branch from 13e20ed to 5a5432a Compare December 2, 2024 16:57

MetRonnie approved these changes Dec 2, 2024

View reviewed changes

hjoliver approved these changes Dec 3, 2024

View reviewed changes

cylc/flow/scheduler_cli.py Outdated Show resolved Hide resolved

wxtim and others added 2 commits December 3, 2024 07:27

Update cylc/flow/cfgspec/workflow.py

4f659de

Co-authored-by: Ronnie Dutta <[email protected]>

Update cylc/flow/scheduler_cli.py

a6c1f94

Co-authored-by: Hilary James Oliver <[email protected]>

MetRonnie requested changes Dec 3, 2024

View reviewed changes

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

wxtim commented Dec 3, 2024

View reviewed changes

cylc/flow/cfgspec/workflow.py Outdated Show resolved Hide resolved

Update cylc/flow/cfgspec/workflow.py

de13f75

wxtim requested a review from MetRonnie December 3, 2024 11:43

MetRonnie approved these changes Dec 3, 2024

View reviewed changes

oliver-sanders reviewed Dec 3, 2024

View reviewed changes

changes.d/6039.feat.md Outdated Show resolved Hide resolved

wxtim and others added 3 commits December 3, 2024 14:52

Update changes.d/6039.feat.md

30a91ae

Co-authored-by: Oliver Sanders <[email protected]>

Merge branch 'master' into feature.skip_mode

e7c3f3f

fix

414496c

oliver-sanders merged commit 5b2b180 into cylc:master Dec 3, 2024
23 of 24 checks passed

wxtim deleted the feature.skip_mode branch December 4, 2024 08:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip Mode #6039

Skip Mode #6039

wxtim commented Mar 27, 2024 •

edited by MetRonnie

Loading

hjoliver left a comment

MetRonnie left a comment

hjoliver left a comment

oliver-sanders commented Dec 3, 2024 •

edited

Loading

Skip Mode #6039

Skip Mode #6039

Conversation

wxtim commented Mar 27, 2024 • edited by MetRonnie Loading

hjoliver left a comment

Choose a reason for hiding this comment

MetRonnie left a comment

Choose a reason for hiding this comment

hjoliver left a comment

Choose a reason for hiding this comment

oliver-sanders commented Dec 3, 2024 • edited Loading

wxtim commented Mar 27, 2024 •

edited by MetRonnie

Loading

oliver-sanders commented Dec 3, 2024 •

edited

Loading