diff --git a/src/7-to-8/major-changes/cylc-install.rst b/src/7-to-8/major-changes/cylc-install.rst index 361948002d2..5ad51b6c3a3 100644 --- a/src/7-to-8/major-changes/cylc-install.rst +++ b/src/7-to-8/major-changes/cylc-install.rst @@ -125,7 +125,7 @@ See :ref:`the user guide ` for more details. cylc config -i '[install][symlink dirs][]' - The recommended workaround is to use a "dummy" task that runs on the + The recommended workaround is to use a :term:`dummy task` that runs on the particular platform before any such mirror tasks in order to setup symlink directories, but without running anything. diff --git a/src/glossary.rst b/src/glossary.rst index bd1aaa8b348..c91e0a6558c 100644 --- a/src/glossary.rst +++ b/src/glossary.rst @@ -1767,3 +1767,17 @@ Glossary :term:`incomplete outputs `, then the workflow is not complete and the scheduler will :term:`stall` pending manual intervention. + dummy task + A task which runs a trivially simple script such as ``sleep 1``, + ``exit 0`` or ``true``, or which uses alternative :ref:`task-run-modes` + or :ref:`workflow-run-modes` to run such a script or avoid running a + script at all. + + .. important:: + + All tasks in a workflow run in dummy or simulation mode are dummy + tasks, but the methods outlined above will create dummy tasks in a + live workflow. + + dummy mode + Modifies all tasks to be :ref:`workflow-run-modes.dummy` tasks. diff --git a/src/user-guide/running-workflows/index.rst b/src/user-guide/running-workflows/index.rst index b86c52a95c3..f4b3f74cf3c 100644 --- a/src/user-guide/running-workflows/index.rst +++ b/src/user-guide/running-workflows/index.rst @@ -12,7 +12,7 @@ Running Workflows tracking-task-state workflow-completion reflow - simulation-modes + workflow-run-modes scheduler-log-files dynamic-behaviour authentication-files diff --git a/src/user-guide/running-workflows/workflow-run-modes.rst b/src/user-guide/running-workflows/workflow-run-modes.rst index d7f08914ed5..3b0e7a29493 100644 --- a/src/user-guide/running-workflows/workflow-run-modes.rst +++ b/src/user-guide/running-workflows/workflow-run-modes.rst @@ -1,33 +1,60 @@ -.. _SimulationMode: +.. _workflow-run-modes: -Simulation Modes ----------------- +Workflow Run Modes +------------------ Cylc can run a workflow without running the real jobs, which may be long-running and resource-hungry. -**Dummy mode** replaces real jobs with background ``sleep`` jobs on the -scheduler host. This avoids :term:`job runner` directives that request compute -resources for real workflow tasks, and it allows any workflow configuration to -be run locally in dummy mode. +This is useful when developing workflows. Tasks submitted +using :ref:`workflow-run-modes.dummy` and +:ref:`workflow-run-modes.simulation` can be used while +developing workflows to understand how different outcomes +and run lengths will affect the workflow. + +To apply a run mode to a workflow: .. code-block:: console - $ cylc play --mode=dummy # real dummy jobs + $ cylc play --mode=dummy + $ cylc play --mode=simulation -.. note:: +.. admonition:: Limitations - All configured task ``script`` items including ``init-script`` are ignored - in dummy mode. If your ``init-script`` is required to run even local dummy - jobs, the relevant environment setup should be done elsewhere. + * Workflow run mode is recorded in the workflow run database. Cylc will not let you + *restart* a dummy mode workflow in live mode, or vice versa. Instead, + install a new instance of the workflow and run it from scratch in the new mode. + * Workflows run in Dummy and Simulation mode do not allow the run mode of + individual tasks to be overridden by :ref:`task-run-modes.skip` -**Simulation mode** does not run real jobs at all, and does not generate job -log files. +.. _workflow-run-modes.dummy: -.. code-block:: console +Dummy Mode +---------- + +**Dummy mode** replaces real jobs with background jobs on the +scheduler host which use ``sleep`` to simulate the run length according to +the settings described for simulation mode. +This avoids :term:`job runner` directives that request compute +resources for real workflow tasks, and it allows any workflow configuration to +be run locally in dummy mode. + +.. admonition:: Limitations + + * Dummy mode can only be applied to a whole workflow. + * Dummy tasks run locally, so dummy mode does not test communication with remote + job platforms. However, it is easy to write a live-mode test workflow with + simple ``sleep 10`` tasks that submit to a remote platform. - $ cylc play --mode=simulation # no real jobs + +.. _workflow-run-modes.simulation: + +Simulation Mode +--------------- + +**Simulation mode** does not run real jobs, and does not generate job +log files. Simulated Run Length @@ -38,19 +65,17 @@ changed with :cylc:conf:`[runtime][][simulation]default run length`. If :cylc:conf:`[runtime][]execution time limit` and :cylc:conf:`[runtime][][simulation]speedup factor` are both set, -run length is computed by dividing the time limit by the speedup factor. - +run length is computed by dividing the +:cylc:conf:`[runtime][]execution time limit` by +:cylc:conf:`[runtime][][simulation]` speedup factor. Simulated Failure ^^^^^^^^^^^^^^^^^ -Tasks always complete custom outputs, by default they succeed. +Tasks always complete all custom outputs, by default they succeed. -.. warning:: - - In simulation mode a succeeded output may not imply that - submitted and/or started outputs are generated - so you will not - be able to test graph pathways such as ``task:started => do_something``. + If you want to test individual workflow pathways, use + :ref:`skip mode `. You can set some or all instances of a task to fail using :cylc:conf:`[runtime][][simulation]fail cycle points`. @@ -81,21 +106,11 @@ whether your failure handling works correctly: fail try 1 only = false fail cycle points = 2022-01-01T03:00Z -Limitations -^^^^^^^^^^^ - -Dummy tasks run locally, so dummy mode does not test communication with remote -job platforms. However, it is easy to write a live-mode test workflow with -simple ``sleep 10`` tasks that submit to a remote platform. - -Alternate path branching is difficult to simulate effectively. You can -configure certain tasks to fail via -:cylc:conf:`[runtime][][simulation]`, but all branches based -on mutually exclusive custom outputs will run because all custom outputs get -artificially completed in dummy mode and in simulation mode. - -.. note:: +.. admonition:: Limitations - Run mode is recorded in the workflow run database. Cylc will not let you - *restart* a dummy mode workflow in live mode, or vice versa. Instead, - install a new instance of the workflow and run it from scratch in the new mode. + * Simulation mode can only be applied to a whole workflow. + * Alternate path branching is difficult to simulate effectively. You can + configure certain tasks to fail via + :cylc:conf:`[runtime][][simulation]`, but all branches based + on mutually exclusive custom outputs will run because all custom outputs get + artificially completed in dummy mode and in simulation mode. diff --git a/src/user-guide/task-implementation/index.rst b/src/user-guide/task-implementation/index.rst index 4e161367589..01c0afbe41e 100644 --- a/src/user-guide/task-implementation/index.rst +++ b/src/user-guide/task-implementation/index.rst @@ -16,3 +16,4 @@ job fails and the task is re-run). job-scripts job-submission ssh-job-management + task-run-modes diff --git a/src/user-guide/task-implementation/task-run-modes.rst b/src/user-guide/task-implementation/task-run-modes.rst new file mode 100644 index 00000000000..9d061eb9df1 --- /dev/null +++ b/src/user-guide/task-implementation/task-run-modes.rst @@ -0,0 +1,112 @@ +.. _task-run-modes: + +Task Run Modes +============== + +.. versionadded:: 8.4.0 + + * Before 8.4.0 run modes could only be set for the workflow, not + for individual tasks. + * Skip mode. + + +As well as the default task implementation ("live" mode) +other implementations can be selected: + +* By setting the scheduler run mode as an argument to the + ``cylc play --mode`` command (:ref:`workflow-run-modes.simulation` + and :ref:`workflow-run-modes.dummy`). +* By setting :cylc:conf:`[runtime][]run mode` for + "skip" mode. + +.. note:: + + The simulation and dummy workflow run modes ignore the + :cylc:conf:`[runtime][]run mode` setting. + + +.. _task-run-modes.skip: + +Skip Mode +--------- + +Skip mode is designed as an aid to workflow control: + +* Allows creation of :term:`dummy tasks` as part of workflow + design. +* Allows skipping of tasks in a running workflow using + ``cylc broadcast -s 'run mode = skip'``. + +Skip mode allows the user to specify which task outputs +will be emitted using :cylc:conf:`[runtime][][skip]outputs`. + +By default task event handlers are disabled by skip mode, but they +can be enabled using +:cylc:conf:`[runtime][][skip]disable task event handlers`. + +Set task to succeeded +^^^^^^^^^^^^^^^^^^^^^ + +.. admonition:: Scenario + + We don't need a task to run, but want to set it to succeed. + +Broadcast :cylc:conf:`[runtime][]run mode` setting the +value to ``skip``. + +Create a Graph Control Task +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. admonition:: Scenario + + We have a large family to large family trigger. + + .. code-block:: cylc + + [task parameters] + x = 1..N # Where N is big. + [scheduling] + [[graph]] + R1 = FAMILY:succeed-all => FAMILY2 + [runtime] + [[FAMILY, FAMILY2]] + [[task1]] + inherit = FAMILY + [[task2]] + inherit = FAMILY2 + + If we increase N, the number of dependencies being tracked + by the scheduler is N^2. + +Adding a skip mode task between the families: + +.. code-block:: cylc + + FAMILY:succeed-all => skiptask => FAMILY2 + +Will reduce the number of dependencies to 2N. + +Parameter Exclusion +^^^^^^^^^^^^^^^^^^^ + +.. admonition:: Scenario + + We want to skip a small number of tasks from a parameterized + group of tasks: + + .. code-block:: cylc + + [task parameters] + # House number 13 doesn't actually exist on this street... + house_number = 1..20 + +We can use skip mode to make sure that a parameter task always +succeeds without running anything: + +.. code-block:: cylc + + [runtime] + [[post parcel]] + script = send letter + [[post parcel]] + run mode = skip diff --git a/src/user-guide/troubleshooting.rst b/src/user-guide/troubleshooting.rst index 57897507c2f..3968880973a 100644 --- a/src/user-guide/troubleshooting.rst +++ b/src/user-guide/troubleshooting.rst @@ -309,12 +309,12 @@ logged a file in this directory: If you need to access files from a remote platform (e.g. 2-stage ``fcm_make``), ensure that a task has submitted to it before you do so. If needed you can use -a blank "dummy" task to ensure that remote installation is completed *before* +a blank "non-functional" task to ensure that remote installation is completed *before* you run any tasks which require this e.g: .. code-block:: cylc-graph - dummy => fetch_files + non_functional => fetch_files Conda / Mamba environment activation fails diff --git a/src/user-guide/writing-workflows/runtime.rst b/src/user-guide/writing-workflows/runtime.rst index 6363a7248b0..9510d4be49e 100644 --- a/src/user-guide/writing-workflows/runtime.rst +++ b/src/user-guide/writing-workflows/runtime.rst @@ -468,7 +468,7 @@ An implicit task is one that appears in the graph but is not defined under Depending on the value of :cylc:conf:`flow.cylc[scheduler]allow implicit tasks`, Cylc can automatically create default task definitions for these, to submit -local dummy jobs that just return the standard job status messages. +dummy jobs that just return the standard job status messages. Implicit tasks can be used to mock up functional workflows very quickly. A default ``script`` can be added to the root family, e.g. to slow job execution diff --git a/src/user-guide/writing-workflows/scheduling.rst b/src/user-guide/writing-workflows/scheduling.rst index 9e6c280da2a..0a5970b90fe 100644 --- a/src/user-guide/writing-workflows/scheduling.rst +++ b/src/user-guide/writing-workflows/scheduling.rst @@ -1287,9 +1287,10 @@ number of members of ``FAM1`` and ``M`` is the number of members of ``FAM2``. This can result in high memory use as the number of family members grows. You can greatly reduce the number of dependencies generated here by putting -dummy tasks in the graph to represent the state of the upstream family. For -example, if ``FAM2`` should trigger off any member of ``FAM1`` succeeding you -can use a dummy task ``FAM1_done`` like this: +:term:`dummy tasks` in the graph to represent the state of the +upstream family. For example, if ``FAM2`` should trigger off any member of +``FAM1`` succeeding you can use a :term:`dummy tasks` +``FAM1_done`` like this: .. code-block:: cylc @@ -1301,12 +1302,18 @@ can use a dummy task ``FAM1_done`` like this: [runtime] # ... [[FAM1_done]] - script = true + run mode = skip # ... This :term:`graph` generates only ``N + M`` dependencies, which takes significantly less memory and CPU to store and evaluate. +.. note:: + + By setting ``run mode = skip`` the task will be follow the + :ref:`task-run-modes.skip` submission pathway, which + is significantly more efficient than a conventional task. + .. _InterCyclePointTriggers: