diff --git a/src/glossary.rst b/src/glossary.rst index c30c562b036..c87e35bc652 100644 --- a/src/glossary.rst +++ b/src/glossary.rst @@ -9,6 +9,16 @@ Glossary The ``cylc validate`` command parses ``flow.cylc`` workflow configuration files and reports syntax errors and deprecation warnings. + retry + Tasks configured to retry on failure will return to the ``waiting`` state + with a new clock trigger to handle the retry delay. Task jobs can + get their try number from the environment, if needed. + + See also: + + * :ref:`TaskRetries` + + workflow cylc workflow A Cylc workflow is a collection of :term:`tasks ` to carry out and diff --git a/src/img/task-job-icons/job-blank.png b/src/img/task-job-icons/job-blank.png new file mode 100644 index 00000000000..b964ba6a5df Binary files /dev/null and b/src/img/task-job-icons/job-blank.png differ diff --git a/src/img/task-job-icons/job-failed.png b/src/img/task-job-icons/job-failed.png new file mode 100644 index 00000000000..d73f1057e75 Binary files /dev/null and b/src/img/task-job-icons/job-failed.png differ diff --git a/src/img/task-job-icons/job-running.png b/src/img/task-job-icons/job-running.png new file mode 100644 index 00000000000..199911143b4 Binary files /dev/null and b/src/img/task-job-icons/job-running.png differ diff --git a/src/img/task-job-icons/job-submit-failed.png b/src/img/task-job-icons/job-submit-failed.png new file mode 100644 index 00000000000..0fdd0fcc0e8 Binary files /dev/null and b/src/img/task-job-icons/job-submit-failed.png differ diff --git a/src/img/task-job-icons/job-submitted.png b/src/img/task-job-icons/job-submitted.png new file mode 100644 index 00000000000..003c55c497b Binary files /dev/null and b/src/img/task-job-icons/job-submitted.png differ diff --git a/src/img/task-job-icons/job-succeeded.png b/src/img/task-job-icons/job-succeeded.png new file mode 100644 index 00000000000..de7300a7fb0 Binary files /dev/null and b/src/img/task-job-icons/job-succeeded.png differ diff --git a/src/img/task-job-icons/job-waiting.png b/src/img/task-job-icons/job-waiting.png new file mode 100644 index 00000000000..15078b46933 Binary files /dev/null and b/src/img/task-job-icons/job-waiting.png differ diff --git a/src/img/task-job-icons/task-expired.png b/src/img/task-job-icons/task-expired.png new file mode 100644 index 00000000000..0028fcfdd3e Binary files /dev/null and b/src/img/task-job-icons/task-expired.png differ diff --git a/src/img/task-job-icons/task-failed.png b/src/img/task-job-icons/task-failed.png new file mode 100644 index 00000000000..3ce9774ebc5 Binary files /dev/null and b/src/img/task-job-icons/task-failed.png differ diff --git a/src/img/task-job-icons/task-preparing.png b/src/img/task-job-icons/task-preparing.png new file mode 100644 index 00000000000..59e481708ac Binary files /dev/null and b/src/img/task-job-icons/task-preparing.png differ diff --git a/src/img/task-job-icons/task-running.png b/src/img/task-job-icons/task-running.png new file mode 100644 index 00000000000..7f60c7f9bae Binary files /dev/null and b/src/img/task-job-icons/task-running.png differ diff --git a/src/img/task-job-icons/task-submit-failed.png b/src/img/task-job-icons/task-submit-failed.png new file mode 100644 index 00000000000..34145502f80 Binary files /dev/null and b/src/img/task-job-icons/task-submit-failed.png differ diff --git a/src/img/task-job-icons/task-submitted.png b/src/img/task-job-icons/task-submitted.png new file mode 100644 index 00000000000..17ec00490df Binary files /dev/null and b/src/img/task-job-icons/task-submitted.png differ diff --git a/src/img/task-job-icons/task-succeeded.png b/src/img/task-job-icons/task-succeeded.png new file mode 100644 index 00000000000..13818b2c5a4 Binary files /dev/null and b/src/img/task-job-icons/task-succeeded.png differ diff --git a/src/img/task-job-icons/task-waiting.png b/src/img/task-job-icons/task-waiting.png new file mode 100644 index 00000000000..66a25c1c9e1 Binary files /dev/null and b/src/img/task-job-icons/task-waiting.png differ diff --git a/src/user-guide/running-workflows/task-job-states.rst b/src/user-guide/running-workflows/task-job-states.rst index 5e7e16c6012..6ac974e2774 100644 --- a/src/user-guide/running-workflows/task-job-states.rst +++ b/src/user-guide/running-workflows/task-job-states.rst @@ -1,81 +1,132 @@ +.. |task-waiting| image:: ../../img/task-job-icons/task-waiting.png + :scale: 100% + :align: middle + +.. |task-expired| image:: ../../img/task-job-icons/task-expired.png + :scale: 100% + :align: middle + +.. |task-preparing| image:: ../../img/task-job-icons/task-preparing.png + :scale: 100% + :align: middle + +.. |task-submitted| image:: ../../img/task-job-icons/task-submitted.png + :scale: 100% + :align: middle + +.. |task-submit-failed| image:: ../../img/task-job-icons/task-submit-failed.png + :scale: 100% + :align: middle + +.. |task-running| image:: ../../img/task-job-icons/task-running.png + :scale: 100% + :align: middle + +.. |task-succeeded| image:: ../../img/task-job-icons/task-succeeded.png + :scale: 100% + :align: middle + +.. |task-failed| image:: ../../img/task-job-icons/task-failed.png + :scale: 100% + :align: middle + + +.. |job-blank| image:: ../../img/task-job-icons/job-blank.png + :scale: 100% + :align: middle + +.. |job-submitted| image:: ../../img/task-job-icons/job-submitted.png + :scale: 100% + :align: middle + +.. |job-submit-failed| image:: ../../img/task-job-icons/job-submit-failed.png + :scale: 100% + :align: middle + +.. |job-running| image:: ../../img/task-job-icons/job-running.png + :scale: 100% + :align: middle + +.. |job-succeeded| image:: ../../img/task-job-icons/job-succeeded.png + :scale: 100% + :align: middle + +.. |job-failed| image:: ../../img/task-job-icons/job-failed.png + :scale: 100% + :align: middle + + Task and Job States =================== **Tasks** are a workflow abstraction; they represent future and past jobs as -well as current active ones. +well as current active ones. Task states are represented by monochromatic icons +like |task-running|. **Jobs** are less of an abstraction; they represent real job scripts submitted to run as a process on a computer somewhere, or the final status of those -real processes. - -A single task can submit multiple jobs, by automatic retry or manual triggering. - -In the Cylc UI, task states are represented by monochromatic task icons, and -job states by coloured job icons. - +real processes. Job states are represented by colored icons like |job-running|. +Several color themes are provided. -.. TODO include task and job state images +A single task can have multiple jobs, by automatic retry or manual triggering. .. table:: - ================ =========== - Task-only states Description - ================ =========== - waiting waiting on prerequisites, triggers, internal queues, or runahead limit - expired will not submit job - too far behind the clock - preparing job being prepared for submission by the scheduler - ================ =========== + ======================================================= =========== + Task & Job States Description + ======================================================= =========== + |task-waiting| |job-blank| waiting waiting on prerequisites + |task-expired| |job-blank| expired will not submit job (too far behind) + |task-preparing| |job-blank| preparing job being prepared + |task-submitted| |job-submitted| submitted job submitted + |task-submit-failed| |job-submit-failed| submit-failed job submission failed + |task-running| |job-running| running job running + |task-succeeded| |job-succeeded| succeeded job succeeded + |task-failed| |job-failed| failed job failed + ======================================================= =========== -.. table:: +Windows on the Workflow +======================= - =============== =========== - Task-job states Description - =============== =========== - submitted job submitted to the job runner on the job platform - submit-failed job submission failed - running job running (*started* message received) - succeeded job succeeded (*succeeded* message received) - failed job failed (*failed* message received) - =============== =========== +Cylc graphs can be very large or even infinite in extent, so the UI often can't +display all the tasks at once. Instead, it displays a moving *window* on the +graph, out to ``n`` graph edges from current active tasks. +The ``n = 0`` window includes only active tasks, where "active" means: -The Active Task Window -====================== +- ``preparing`` tasks +- ``submitted`` and ``running`` tasks - i.e. those with active jobs +- ``waiting`` tasks, that are waiting on: + + - :ref:`clock triggers ` + - :ref:`external triggers
` + - :ref:`internal queues ` + - :ref:`runahead limit ` + +- finished tasks retained as *incomplete* by the scheduler: -Cylc graphs can be very large, or even infinite in cycling workflows, but the -scheduler does not need to be aware of every single task in the entire run. It -just keeps track of current active tasks and what comes after them in the -graph. The meaning of *active tasks* here is: + - ``submit-failed`` tasks if successful submission was not *optional* + - ``succeeded`` or ``failed`` tasks that did not complete *expected outputs* -- Tasks in the ``preparing`` state -- Tasks with active jobs: ``submitted``, ``running`` -- Any ``succeeded`` or ``failed`` tasks being retained as incomplete (and any - ``submit-failed`` tasks) -- Any ``waiting`` tasks held back by unsatisfied external triggers +.. TODO add links above to optional and expected outputs, once documented. -We also call this the ``n=0`` task window. The UI can display more than this, -to a requested number of graph edges around the active tasks. The default -window size is ``n=1``, which is active tasks plus tasks one graph edge away -from them. -.. note:: - Tasks ahead of the n=0 window are displayed by the UI as ``waiting``, but - unlike the "active" waiting tasks they don't actually exist yet as far as - the scheduler is concerned. +The default window extent is ``n = 1``, i.e. tasks out to one graph edge from +current active tasks. +Tasks ahead of the ``n=0`` window are displayed by the UI as ``waiting`` but +the scheduler is not actively managing them yet. -Waiting Tasks in the Active Window ----------------------------------- -Waiting tasks in the ``n=0`` window depend on unsatisfied non-task prerequisites: +Retrying Tasks +============== -- :ref:`InternalQueues` -- :ref:`RunaheadLimit` -- external triggers -- clock triggers - +In Cylc 8, a task that fails but is configured to :term:`retry` will return to +the ``waiting`` state, with a new clock trigger to handle the retry delay. + +.. note:: -A ``waiting`` task that has one or more jobs associated with it must be going -to retry. + A task that is waiting on a retry will already have one or more failed jobs + associated with it. diff --git a/src/user-guide/writing-workflows/runtime.rst b/src/user-guide/writing-workflows/runtime.rst index cc051b22b0d..4e89d227922 100644 --- a/src/user-guide/writing-workflows/runtime.rst +++ b/src/user-guide/writing-workflows/runtime.rst @@ -490,8 +490,8 @@ adding empty runtime placeholders instead of allowing implicit tasks: .. _TaskRetries: -Automatic Task Retry On Failure -------------------------------- +Task Retry On Failure +--------------------- .. seealso::