From 87bf4abd6730056121baac2a4d03cc11d183cddf Mon Sep 17 00:00:00 2001 From: satyaog Date: Tue, 17 Dec 2024 13:05:36 -0500 Subject: [PATCH 1/4] Add docs on globus personal utilities --- docs/Information_data_transmission.rst | 2 +- docs/Userguide_data_transfer.rst | 51 +++++++++++++++++++++++--- 2 files changed, 47 insertions(+), 6 deletions(-) diff --git a/docs/Information_data_transmission.rst b/docs/Information_data_transmission.rst index b683eacc..addcfbbc 100644 --- a/docs/Information_data_transmission.rst +++ b/docs/Information_data_transmission.rst @@ -1,10 +1,10 @@ Data Transmission ================= - Multiple methods can be used to transfer data to/from the cluster: * ``rsync --bwlimit=10mb``; this is the favored method since the bandwidth can be limited to prevent impacting the usage of the cluster: `rsync `_ * Digital Research Alliance of Canada: `Globus `_ + * Setup a Globus personal endpoint: :ref:`globus_connect_personal` diff --git a/docs/Userguide_data_transfer.rst b/docs/Userguide_data_transfer.rst index 79b2ec10..bd9affb0 100644 --- a/docs/Userguide_data_transfer.rst +++ b/docs/Userguide_data_transfer.rst @@ -1,12 +1,53 @@ +.. _globus_connect_personal: + Data Transmission using Globus Connect Personal =============================================== - Mila doesn't own a Globus license but if the source or destination provides a Globus account, like Digital Research Alliance of Canada for example, it's possible to setup Globus Connect Personal to create a personal endpoint on the -Mila cluster by following the Globus guide to `Install, Configure, and -Uninstall Globus Connect Personal for Linux -`_. +Mila cluster and then transfer data to and from the Mila cluster. + +Some utilities have been made available to help setup a Globus personal +endpoint: + + +Login and add a Globus personal endpoint +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: sh + + # login to your globus account + /network/datasets/scripts/globus_utils.sh globus whoami + + # add a globus personal endpoint + /network/datasets/scripts/globus_utils.sh add_endpoint --name "mila-cluster" + + +Start a Globus personal endpoint +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: sh + + /network/datasets/scripts/globus_utils.sh start_endpoint --dir PATH/TO/DATA + +.. note:: + The endpoint used will be the one precedently added. The utility does not + allow multiple endpoints per user + +.. note:: + It's best to use a cpu allocation to handle the Globus personal endpoint. + Once the personal endpoint is setup, execute the following command to run in + slurm: + + .. code-block:: sh + + sbatch --ntasks=1 --cpus-per-task=4 --mem=8G /network/datasets/scripts/globus_utils.sh start_endpoint --name "mila-cluster" + + +Do more with Globus Personal +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This endpoint can then be used to transfer data to and from the Mila cluster. +To understand better how Globus Personal works and do more with Globus, follow +the Globus guide to `Install, Configure, and Uninstall Globus Connect Personal +for Linux `_. From 375509d055da0d5b4c0fd06f6ced89f9eacf70dc Mon Sep 17 00:00:00 2001 From: satyaog Date: Tue, 17 Dec 2024 13:09:13 -0500 Subject: [PATCH 2/4] Fix rst --- docs/Userguide_data_transfer.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Userguide_data_transfer.rst b/docs/Userguide_data_transfer.rst index bd9affb0..6024b0fa 100644 --- a/docs/Userguide_data_transfer.rst +++ b/docs/Userguide_data_transfer.rst @@ -13,7 +13,7 @@ endpoint: Login and add a Globus personal endpoint -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------------------- .. code-block:: sh @@ -25,7 +25,7 @@ Login and add a Globus personal endpoint Start a Globus personal endpoint -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------- .. code-block:: sh @@ -46,7 +46,7 @@ Start a Globus personal endpoint Do more with Globus Personal -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------- To understand better how Globus Personal works and do more with Globus, follow the Globus guide to `Install, Configure, and Uninstall Globus Connect Personal From 88814f146329f57c687fe801ac855bfbe6fe6538 Mon Sep 17 00:00:00 2001 From: satyaog Date: Tue, 17 Dec 2024 13:12:48 -0500 Subject: [PATCH 3/4] Fix rst --- docs/Information_data_transmission.rst | 1 + docs/Userguide_data_transfer.rst | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Information_data_transmission.rst b/docs/Information_data_transmission.rst index addcfbbc..277cdec6 100644 --- a/docs/Information_data_transmission.rst +++ b/docs/Information_data_transmission.rst @@ -7,4 +7,5 @@ Multiple methods can be used to transfer data to/from the cluster: be limited to prevent impacting the usage of the cluster: `rsync `_ * Digital Research Alliance of Canada: `Globus `_ + * Setup a Globus personal endpoint: :ref:`globus_connect_personal` diff --git a/docs/Userguide_data_transfer.rst b/docs/Userguide_data_transfer.rst index 6024b0fa..b6512898 100644 --- a/docs/Userguide_data_transfer.rst +++ b/docs/Userguide_data_transfer.rst @@ -45,8 +45,8 @@ Start a Globus personal endpoint sbatch --ntasks=1 --cpus-per-task=4 --mem=8G /network/datasets/scripts/globus_utils.sh start_endpoint --name "mila-cluster" -Do more with Globus Personal ----------------------------- +Do more with Globus Connect Personal +------------------------------------ To understand better how Globus Personal works and do more with Globus, follow the Globus guide to `Install, Configure, and Uninstall Globus Connect Personal From 036fc9f9d0dd6157fe75790aa9d6942802be6048 Mon Sep 17 00:00:00 2001 From: satyaog Date: Tue, 17 Dec 2024 13:14:10 -0500 Subject: [PATCH 4/4] Fix rst --- docs/Information_data_transmission.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Information_data_transmission.rst b/docs/Information_data_transmission.rst index 277cdec6..f9c57445 100644 --- a/docs/Information_data_transmission.rst +++ b/docs/Information_data_transmission.rst @@ -8,4 +8,4 @@ Multiple methods can be used to transfer data to/from the cluster: `_ * Digital Research Alliance of Canada: `Globus `_ - * Setup a Globus personal endpoint: :ref:`globus_connect_personal` + * Setup Globus Connect Personal: :ref:`globus_connect_personal`