From 1ef82de6d5909a7f93f7d92f038b1b8fef03da17 Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Thu, 14 Sep 2023 15:08:18 +1000 Subject: [PATCH 1/8] Document ssh forward environment variables Documents cylc/cylc-flow#5709 --- CONTRIBUTING.md | 1 + .../config/writing-platform-configs.rst | 62 ++++++++++++++++++- 2 files changed, 61 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5c33efc356..cf1d3250ee 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -54,6 +54,7 @@ requests_). - Yinjing Lin - David Sutherland - Thomas Coleman + - Scott Wales (All contributors are identifiable with email addresses in the git version diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index b38d89bec5..2222f57cd2 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -327,8 +327,8 @@ Symlinks for remote install targets are created during :ref:`RemoteInit` followi ``cylc play``. -Advanced Platform Example -------------------------- +Advanced Platform Examples +-------------------------- Platform with no ``$HOME`` directory ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -370,3 +370,61 @@ In this example Cylc will install workflows into If you are running :term:`schedulers ` directly on the login node and submitting jobs locally then the platform name and install target should be ``localhost``. + +Sharing environment variables with the Cylc server +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. admonition:: Scenario + + An environment variable on the computer where you're running `cylc play` + needs to be shared with the Cylc server (e.g. to set up storage paths) or + with Cylc task hosts (e.g. to set PBS account defaults) + +Normally if a Cylc server or task is running on a remote host it will not keep +environment variables that are present when you run ``cylc play``. You can +whitelist variables to be sent to remote hosts using +:cylc:conf:`global.cylc[scheduler][run hosts]ssh forward environment variables`. + +These variables are forwared to the Cylc server and then from there to task +hosts. They can then be used to affect the global configuration or the task +configuration (for instance ``qsub`` may use the value of ``$PROJECT`` to set a +default account to charge usage to). + + .. code-block:: cylc + :caption: part of a ``global.cylc`` config file + + [scheduler] + [[run hosts]] + ssh forward environment variables = LUSTRE_DISK, PROJECT + + [install] + [[symlink dirs]] + [[[hpc]]] + run = {{ environ['LUSTRE_DISK'] }} + + [platforms] + [[hpc]] + submit method = pbs + [[[directives]]] + -P = {{ environ['PROJECT'] }} + +In this example Cylc will install workflows into the directory specified by +``$LUSTRE_DISK`` and use the project specified by ``$PROJECT`` on the server +where you run ``cylc play``, e.g. + + .. code-block:: sh + + export LUSTRE_DISK=/g/data/foo + export PROJECT=bar + cylc play + +will store the workflow under ``/g/data/foo`` and submit jobs under project +``bar``. + +It's better to be explicit in what's happening when setting up the +configuration - even though ``qsub`` will pick up ``$PROJECT`` from the +environment we're also explicitly setting the account to use with ``-P`` in +case the task is run manually. + +This setting only affects task submission - to set a variable once the task has +started see :cylc:conf:`global.cylc[platforms][]copyable environment variables`. From 32acf9b07bff76bcb44173ba5eb92decf5f9767b Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Wed, 27 Sep 2023 14:24:44 +1000 Subject: [PATCH 2/8] Don't forward to run hosts --- .../config/writing-platform-configs.rst | 21 +++++++------------ 1 file changed, 7 insertions(+), 14 deletions(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index 2222f57cd2..c22e7e5b27 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -380,15 +380,13 @@ Sharing environment variables with the Cylc server needs to be shared with the Cylc server (e.g. to set up storage paths) or with Cylc task hosts (e.g. to set PBS account defaults) -Normally if a Cylc server or task is running on a remote host it will not keep +Normally if a Cylc server is running on a remote host it will not keep environment variables that are present when you run ``cylc play``. You can -whitelist variables to be sent to remote hosts using +whitelist variables to be sent to remote servers using :cylc:conf:`global.cylc[scheduler][run hosts]ssh forward environment variables`. -These variables are forwared to the Cylc server and then from there to task -hosts. They can then be used to affect the global configuration or the task -configuration (for instance ``qsub`` may use the value of ``$PROJECT`` to set a -default account to charge usage to). +These variables are forwared to the Cylc server and may be used in the +`global.cylc` file templating. .. code-block:: cylc :caption: part of a ``global.cylc`` config file @@ -409,8 +407,8 @@ default account to charge usage to). -P = {{ environ['PROJECT'] }} In this example Cylc will install workflows into the directory specified by -``$LUSTRE_DISK`` and use the project specified by ``$PROJECT`` on the server -where you run ``cylc play``, e.g. +``$LUSTRE_DISK`` and use the project specified by ``$PROJECT`` in the +environment where you run ``cylc play``, e.g. .. code-block:: sh @@ -421,10 +419,5 @@ where you run ``cylc play``, e.g. will store the workflow under ``/g/data/foo`` and submit jobs under project ``bar``. -It's better to be explicit in what's happening when setting up the -configuration - even though ``qsub`` will pick up ``$PROJECT`` from the -environment we're also explicitly setting the account to use with ``-P`` in -case the task is run manually. - -This setting only affects task submission - to set a variable once the task has +This setting only affects the server - to set a variable once the task has started see :cylc:conf:`global.cylc[platforms][]copyable environment variables`. From 495059b1ead42df61485ff4c532e52b5a089f1ad Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Wed, 27 Sep 2023 14:25:17 +1000 Subject: [PATCH 3/8] Don't forward to run hosts --- src/reference/config/writing-platform-configs.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index c22e7e5b27..87fbdf7034 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -377,8 +377,7 @@ Sharing environment variables with the Cylc server .. admonition:: Scenario An environment variable on the computer where you're running `cylc play` - needs to be shared with the Cylc server (e.g. to set up storage paths) or - with Cylc task hosts (e.g. to set PBS account defaults) + needs to be shared with the Cylc server (e.g. to set up storage paths) Normally if a Cylc server is running on a remote host it will not keep environment variables that are present when you run ``cylc play``. You can From 3b0bc975888f9eb3df35ff344faa86d968a54e9b Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Fri, 27 Oct 2023 14:33:34 +1100 Subject: [PATCH 4/8] Update the documentation with platform based forwarding --- .../config/writing-platform-configs.rst | 32 +++++++++++++++---- 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index 87fbdf7034..471698e8ac 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -390,16 +390,17 @@ These variables are forwared to the Cylc server and may be used in the .. code-block:: cylc :caption: part of a ``global.cylc`` config file - [scheduler] - [[run hosts]] - ssh forward environment variables = LUSTRE_DISK, PROJECT - [install] [[symlink dirs]] [[[hpc]]] + # Here the environment variable on the server is used run = {{ environ['LUSTRE_DISK'] }} [platforms] + # 'localhost' platform is used when communicating with the server + [[localhost]] + ssh forward environment variables = LUSTRE_DISK, PROJECT + [[hpc]] submit method = pbs [[[directives]]] @@ -418,5 +419,24 @@ environment where you run ``cylc play``, e.g. will store the workflow under ``/g/data/foo`` and submit jobs under project ``bar``. -This setting only affects the server - to set a variable once the task has -started see :cylc:conf:`global.cylc[platforms][]copyable environment variables`. +You can also forward variables from the server to other platforms. You should +first ensure the variable is available on the server, e.g. by also forwarding +the variable to ``[[localhost]]``. + +This setting only affects the task submission (e.g. ``qsub``) which may use +environment variables to set default direvtives. To set a variable once the +task has started see +:cylc:conf:`global.cylc[platforms][]copyable environment variables`. + + .. code-block:: cylc + :caption: part of a ``global.cylc`` config file + + [platforms] + [[localhost]] + ssh forward environment variables = PROJECT + + [[hpc]] + # Here qsub has been configured to read from $PROJECT + ssh forward environment variables = PROJECT + submit method = pbs + From 1affddc06a1b932797ba2ede9b5a247acea4bc18 Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Fri, 27 Oct 2023 14:34:26 +1100 Subject: [PATCH 5/8] Typo --- src/reference/config/writing-platform-configs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index 471698e8ac..d964cada9b 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -424,7 +424,7 @@ first ensure the variable is available on the server, e.g. by also forwarding the variable to ``[[localhost]]``. This setting only affects the task submission (e.g. ``qsub``) which may use -environment variables to set default direvtives. To set a variable once the +environment variables to set default directives. To set a variable once the task has started see :cylc:conf:`global.cylc[platforms][]copyable environment variables`. From f326a6ca76ef2e64fec9611636840304137091bf Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Tue, 9 Jan 2024 10:57:51 +1100 Subject: [PATCH 6/8] Update src/reference/config/writing-platform-configs.rst Co-authored-by: Oliver Sanders --- src/reference/config/writing-platform-configs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index d964cada9b..d72e5fffda 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -376,7 +376,7 @@ Sharing environment variables with the Cylc server .. admonition:: Scenario - An environment variable on the computer where you're running `cylc play` + An environment variable on the computer where you're running ``cylc play`` needs to be shared with the Cylc server (e.g. to set up storage paths) Normally if a Cylc server is running on a remote host it will not keep From 233f4f084f3cbf26b0d16739e453f1494c20dba1 Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Tue, 9 Jan 2024 11:03:04 +1100 Subject: [PATCH 7/8] Update link to point to platforms --- src/reference/config/writing-platform-configs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index d72e5fffda..347f24cf3c 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -382,7 +382,7 @@ Sharing environment variables with the Cylc server Normally if a Cylc server is running on a remote host it will not keep environment variables that are present when you run ``cylc play``. You can whitelist variables to be sent to remote servers using -:cylc:conf:`global.cylc[scheduler][run hosts]ssh forward environment variables`. +:cylc:conf:`global.cylc[platforms][]ssh forward environment variables`. These variables are forwared to the Cylc server and may be used in the `global.cylc` file templating. From 39acdeca8f33cfe8dbf037d315130eb4630452d9 Mon Sep 17 00:00:00 2001 From: Scott Wales Date: Tue, 9 Jan 2024 22:45:59 +1100 Subject: [PATCH 8/8] Update src/reference/config/writing-platform-configs.rst Co-authored-by: Oliver Sanders --- src/reference/config/writing-platform-configs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/reference/config/writing-platform-configs.rst b/src/reference/config/writing-platform-configs.rst index 347f24cf3c..7f5eb646b5 100644 --- a/src/reference/config/writing-platform-configs.rst +++ b/src/reference/config/writing-platform-configs.rst @@ -384,7 +384,7 @@ environment variables that are present when you run ``cylc play``. You can whitelist variables to be sent to remote servers using :cylc:conf:`global.cylc[platforms][]ssh forward environment variables`. -These variables are forwared to the Cylc server and may be used in the +These variables are forwarded to the Cylc server and may be used in the `global.cylc` file templating. .. code-block:: cylc