Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Xtrig arg validate #38

Closed
wants to merge 39 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
f4e2853
Fix bugs in `cylc broadcast` (#5933)
MetRonnie Jan 24, 2024
08d9c53
add example workflows (#5773)
oliver-sanders Jan 29, 2024
94fd7ac
docs: document exceptions (#5940)
oliver-sanders Jan 29, 2024
82af1f3
docs: fix sort order of the first few examples (#5948)
oliver-sanders Jan 30, 2024
870fb47
stop after cycle point: support offsets
oliver-sanders Jan 29, 2024
a83df17
doc: correct spelling
oliver-sanders Jan 30, 2024
8a676a8
Merge pull request #5936 from cylc/8.2.x-sync
MetRonnie Jan 30, 2024
f6752d8
Merge pull request #5951 from oliver-sanders/spellcheck
hjoliver Jan 30, 2024
6fea50f
Add integration tests for xtrigger validation, which provides simple …
wxtim Jan 3, 2024
2bcad6b
Apply suggestions from code review
oliver-sanders Jan 31, 2024
ea50b53
loosen xargs validation checks
wxtim Jan 31, 2024
a8988df
Tidy (#5886)
MetRonnie Jan 31, 2024
b543af7
avoid use of assert
wxtim Jan 31, 2024
85492ae
build(deps): bump codecov/codecov-action from 3 to 4
dependabot[bot] Feb 1, 2024
38a4e15
cylc lint: deprecate `pyproject.toml` section `cylc-lint` in favour o…
MetRonnie Feb 1, 2024
26bc684
make docstrings play nicely with Napoleon
wxtim Feb 1, 2024
3013d21
fix doctest
wxtim Feb 1, 2024
690fd5d
spelling fixes
wxtim Feb 2, 2024
810f3be
Apply suggestions from code review
oliver-sanders Feb 2, 2024
2887819
Update tests/unit/test_config.py
oliver-sanders Feb 2, 2024
72615f2
Automatically validate xtrigger function signature (#59)
MetRonnie Feb 5, 2024
8b2d9d4
Apply suggestions from code review
wxtim Feb 5, 2024
57fe4ae
Don't record satisfied wall clock triggers in the DB (#5923)
wxtim Feb 5, 2024
28e68fe
Tidy
MetRonnie Feb 1, 2024
f301d54
Changelog
MetRonnie Feb 1, 2024
3df8835
Merge pull request #5953 from cylc/dependabot/github_actions/codecov/…
oliver-sanders Feb 5, 2024
124666b
fix flake8
wxtim Feb 6, 2024
6cfbe7f
Merge branch 'master' into 8.2.x-sync
wxtim Feb 6, 2024
b1822a6
added suite state xtrigger for back-compat (#5864)
markgrahamdawson Feb 8, 2024
b1b189b
Merge pull request #5957 from cylc/8.2.x-sync
wxtim Feb 8, 2024
381c691
fixed flake8 (#5962)
markgrahamdawson Feb 8, 2024
7bf0b7a
Merge pull request #5956 from MetRonnie/cylc-lint
wxtim Feb 9, 2024
14711c9
Merge pull request #5943 from oliver-sanders/5939
wxtim Feb 9, 2024
8ab57ad
fix tui tests broken by urwid update (#5967)
wxtim Feb 13, 2024
4eb7d82
remote_init: remove trailing space from symlink paths (#5969)
oliver-sanders Feb 15, 2024
4515ace
Merge pull request #5972 from cylc/8.2.x-sync
oliver-sanders Feb 15, 2024
ca6c64d
doc: link workflow event configs to user guide section (#5954)
oliver-sanders Feb 15, 2024
68dbfd2
Merge branch 'master' into xtrig-arg-validate
MetRonnie Feb 15, 2024
933b273
Improve xtrigger validation (#60)
MetRonnie Feb 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test_fast.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ jobs:
uses: actions/download-artifact@v4

- name: Codecov upload
uses: codecov/codecov-action@v3
uses: codecov/codecov-action@v4
with:
name: ${{ github.workflow }}
flags: fast-tests
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test_functional.yml
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ jobs:
uses: actions/download-artifact@v4

- name: Codecov upload
uses: codecov/codecov-action@v3
uses: codecov/codecov-action@v4
with:
name: ${{ github.workflow }}
flags: functional-tests
Expand Down
22 changes: 4 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,28 +29,14 @@ domains.
# install cylc
conda install cylc-flow

# write your first workflow
mkdir -p ~/cylc-src/example
cat > ~/cylc-src/example/flow.cylc <<__CONFIG__
[scheduling]
initial cycle point = 1
cycling mode = integer
[[graph]]
P1 = """
a => b => c & d
b[-P1] => b
"""
[runtime]
[[a, b, c, d]]
script = echo "Hello $CYLC_TASK_NAME"
__CONFIG__
# extract an example to run
cylc get-resources examples/integer-cycling

# install and run it
cylc install example
cylc play example
cylc vip integer-cycling # vip = validate, install and play

# watch it run
cylc tui example
cylc tui integer-cycling
```

### The Cylc Ecosystem
Expand Down
1 change: 1 addition & 0 deletions changes.d/5864.feat.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Reimplemented the `suite-state` xtrigger for interoperability with Cylc 7.
1 change: 1 addition & 0 deletions changes.d/5933.fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed bug in `cylc broadcast` (and the GUI Edit Runtime command) where everything after a `#` character in a setting would be stripped out.
1 change: 1 addition & 0 deletions changes.d/5943.feat.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The `stop after cycle point` can now be specified as an offset from the inital cycle point.
File renamed without changes.
1 change: 1 addition & 0 deletions changes.d/5956.break.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`cylc lint`: deprecated `[cylc-lint]` section in favour of `[tool.cylc.lint]` in `pyproject.toml`
64 changes: 64 additions & 0 deletions cylc/flow/cfgspec/globalcfg.py
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,17 @@

Template variables can be used to configure handlers. For a full list
of supported variables see :ref:`workflow_event_template_variables`.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`
''',
'handler events': '''
Specify the events for which workflow event handlers should be invoked.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`
''',
'mail events': '''
Specify the workflow events for which notification emails should
Expand All @@ -176,13 +184,21 @@
'startup handlers': f'''
Handlers to run at scheduler startup.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``startup handler``.
''',
'shutdown handlers': f'''
Handlers to run at scheduler shutdown.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``shutdown handler``.
Expand All @@ -191,6 +207,10 @@
Handlers to run if the scheduler shuts down with error status due to
a configured timeout or a fatal error condition.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``aborted handler``.
Expand All @@ -199,36 +219,60 @@
Workflow timeout interval. The timer starts counting down at scheduler
startup. It resets on workflow restart.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionadded:: 8.0.0
''',
'workflow timeout handlers': '''
Handlers to run if the workflow timer times out.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionadded:: 8.0.0
''',
'abort on workflow timeout': '''
Whether the scheduler should shut down immediately with error status if
the workflow timer times out.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionadded:: 8.0.0
''',
'stall handlers': f'''
Handlers to run if the scheduler stalls.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``stalled handler``.
''',
'stall timeout': f'''
The length of a timer which starts if the scheduler stalls.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``timeout``.
''',
'stall timeout handlers': f'''
Handlers to run if the stall timer times out.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``timeout handler``.
Expand All @@ -237,6 +281,10 @@
Whether the scheduler should shut down immediately with error status if
the stall timer times out.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``abort on timeout``.
Expand All @@ -245,13 +293,21 @@
Scheduler inactivity timeout interval. The timer resets when any
workflow activity occurs.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES} ``inactivity``.
''',
'inactivity timeout handlers': f'''
Handlers to run if the inactivity timer times out.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``inactivity handler``.
Expand All @@ -260,6 +316,10 @@
Whether the scheduler should shut down immediately with error status if
the inactivity timer times out.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionchanged:: 8.0.0

{REPLACES}``abort on inactivity``.
Expand All @@ -268,6 +328,10 @@
How long to wait for intervention on restarting a completed workflow.
The timer stops if any task is triggered.

.. seealso::

:ref:`user_guide.scheduler.workflow_events`

.. versionadded:: 8.2.0

'''
Expand Down
21 changes: 19 additions & 2 deletions cylc/flow/cfgspec/workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -539,14 +539,20 @@ def get_script_common_text(this: str, example: Optional[str] = None):
''')
# NOTE: final cycle point is not a V_CYCLE_POINT to allow expressions
# such as '+P1Y' (relative to initial cycle point)
Conf('final cycle point', VDR.V_STRING, desc='''
Conf('final cycle point', VDR.V_CYCLE_POINT_WITH_OFFSETS, desc='''
The (optional) last cycle point at which tasks are run.

Once all tasks have reached this cycle point, the
workflow will shut down.

This item can be overridden on the command line using
``cylc play --final-cycle-point`` or ``--fcp``.

Examples:

- ``2000`` - Shorthand for ``2000-01-01T00:00``.
- ``+P1D`` - The initial cycle point plus one day.
- ``2000 +P1D +P1Y`` - The year ``2000`` plus one day and one year.
''')
Conf('initial cycle point constraints', VDR.V_STRING_LIST, desc='''
Rules to allow only some initial datetime cycle points.
Expand Down Expand Up @@ -599,7 +605,7 @@ def get_script_common_text(this: str, example: Optional[str] = None):

{REPLACES}``[scheduling]hold after point``.
''')
Conf('stop after cycle point', VDR.V_CYCLE_POINT, desc='''
Conf('stop after cycle point', VDR.V_CYCLE_POINT_WITH_OFFSETS, desc='''
Shut down the workflow after all tasks pass this cycle point.

The stop cycle point can be overridden on the command line using
Expand All @@ -612,7 +618,18 @@ def get_script_common_text(this: str, example: Optional[str] = None):
choosing not to run that part of the graph. You can play
the workflow and continue.

Examples:

- ``2000`` - Shorthand for ``2000-01-01T00:00``.
- ``+P1D`` - The initial cycle point plus one day.
- ``2000 +P1D +P1Y`` - The year ``2000`` plus one day and one year.

.. versionadded:: 8.0.0

.. versionchanged:: 8.3.0

This now supports offsets (e.g. ``+P1D``) in the same way the
:cylc:conf:`[..]final cycle point` does.
''')
Conf('cycling mode', VDR.V_STRING, Calendar.MODE_GREGORIAN,
options=list(Calendar.MODES) + ['integer'], desc='''
Expand Down
26 changes: 12 additions & 14 deletions cylc/flow/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,6 @@
from cylc.flow.print_tree import print_tree
from cylc.flow.simulation import configure_sim_modes
from cylc.flow.subprocctx import SubFuncContext
from cylc.flow.subprocpool import get_xtrig_func
from cylc.flow.task_events_mgr import (
EventData,
get_event_handler_data
Expand Down Expand Up @@ -853,9 +852,16 @@ def process_stop_cycle_point(self) -> None:
if stopcp_str is None:
stopcp_str = self.cfg['scheduling']['stop after cycle point']

if stopcp_str is not None:
self.stop_point = get_point(stopcp_str).standardise()
if self.final_point and (self.stop_point > self.final_point):
if stopcp_str:
self.stop_point = get_point_relative(
stopcp_str,
self.initial_point,
).standardise()
if (
self.final_point is not None
and self.stop_point is not None
and self.stop_point > self.final_point
):
LOG.warning(
f"Stop cycle point '{self.stop_point}' will have no "
"effect as it is after the final cycle "
Expand Down Expand Up @@ -1728,14 +1734,6 @@ def generate_triggers(self, lexpression, left_nodes, right, seq,
# Generic xtrigger validation.
XtriggerManager.check_xtrigger(label, xtrig, self.fdir)

# Specific xtrigger.validate(), if available.
with suppress(AttributeError, ImportError):
get_xtrig_func(xtrig.func_name, "validate", self.fdir)(
xtrig.func_args,
xtrig.func_kwargs,
xtrig.get_signature()
)

if self.xtrigger_mgr:
# (not available during validation)
self.xtrigger_mgr.add_trig(label, xtrig, self.fdir)
Expand Down Expand Up @@ -2427,8 +2425,8 @@ def upgrade_clock_triggers(self):
# Derive an xtrigger label.
label = '_'.join(('_cylc', 'wall_clock', task_name))
# Define the xtrigger function.
xtrig = SubFuncContext(label, 'wall_clock', [], {})
xtrig.func_kwargs["offset"] = offset
args = [] if offset is None else [offset]
xtrig = SubFuncContext(label, 'wall_clock', args, {})
if self.xtrigger_mgr is None:
XtriggerManager.check_xtrigger(label, xtrig, self.fdir)
else:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/bin/bash
# THIS FILE IS PART OF THE CYLC WORKFLOW ENGINE.
# Copyright (C) NIWA & British Crown (Met Office) & Contributors.
#
Expand All @@ -14,27 +14,10 @@
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#-------------------------------------------------------------------------------
# Test validation of special tasks names with non-word characters
. "$(dirname "$0")/test_header"
set_test_number 1
cat >'flow.cylc' <<'__FLOW_CONFIG__'
[scheduling]
initial cycle point = 20200202
final cycle point = 20300303
[[special tasks]]
clock-trigger = t-1, t+1, t%1, t@1
[[graph]]
P1D = """
t-1
t+1
t%1
t@1
"""

[runtime]
[[t-1, t+1, t%1, t@1]]
script = true
__FLOW_CONFIG__
run_ok "${TEST_NAME_BASE}" cylc validate "${PWD}"
exit
set -eux

ID="$(< /dev/urandom tr -dc A-Za-z | head -c6)"
cylc vip --check-circular --no-detach --no-run-name --workflow-name "$ID"
cylc lint "$ID"
cylc clean "$ID"
13 changes: 13 additions & 0 deletions cylc/flow/etc/examples/1-hello-world/flow.cylc
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[meta]
title = Hello World
description = """
A simple workflow which runs a single task (hello_world) once.
"""

[scheduling]
[[graph]]
R1 = hello_world

[runtime]
[[hello_world]]
script = echo "Hello World!"
Loading