From a90f0cf977c57300e1198637de0944043bedecb4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:16:19 +0100 Subject: [PATCH 01/10] Extend the 'Your first script' section. * Bit more verbose description of what the script is actually doing * Mini subsection switching the (often unfamiliar) bash commands with some Python --- docs/basic_training/intro.md | 20 +++++++++++++++++- nf-training/hello_py.nf | 41 ++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 1 deletion(-) create mode 100644 nf-training/hello_py.nf diff --git a/docs/basic_training/intro.md b/docs/basic_training/intro.md index 42322ee2..e0841d1e 100644 --- a/docs/basic_training/intro.md +++ b/docs/basic_training/intro.md @@ -1,5 +1,6 @@ --- description: Getting started with Nextflow +toc_depth: 1 --- # Introduction @@ -142,7 +143,24 @@ workflow { // (18)! 21. The final output (in the `results_ch` channel) is printed to screen using the `view` operator (appended onto the channel name). 22. End of the workflow scope. -The use of the operator `.flatten()` here is to split the two files into two separate items to be put through the next process (else they would be treated as a single element). +This pipeline takes `params.greeting`, which defaults to the string `Hello world!`, and splits it into indiviudual words in the `SPLITLETTERS` process. Each word is written to a separate file, named `chunk_1`, `chunk_2` and so on. These files are picked up as the process `output`. + +The second process `CONVERTTOUPPER` takes the output channel from the first process as its input. +The use of the operator `.flatten()` here is to split the two files into two separate items to be put through the this process, else they would be treated as a single element. +The `CONVERTTOUPPER` process thus launches two tasks, one for each element. The bash script uses `cat` to print the file contents and `tr` to convert to upper-case. It takes the resulting standard-out as the process output channel. + +#### Python instead of bash + +If you're not completely comfortable with the bash code used in the example, don't worry! You can use whatever programming language you like within Nextflow `script` blocks. For example, here is the same example using Python code: + +```python title="nf-training/hello_py.nf" linenums="14" +--8<-- "nf-training/hello_py.nf:14:18" +``` +```python title="nf-training/hello_py.nf" linenums="30" +--8<-- "nf-training/hello_py.nf:30:32" +``` + +Note that the `$x` and `$y` variables are interpolated by Nextflow, so the resulting Python scripts will have fixed strings here (`#!python x="Hello world!"`). ### In practice diff --git a/nf-training/hello_py.nf b/nf-training/hello_py.nf new file mode 100644 index 00000000..b48fec0e --- /dev/null +++ b/nf-training/hello_py.nf @@ -0,0 +1,41 @@ +#!/usr/bin/env nextflow + +params.greeting = 'Hello world!' +greeting_ch = Channel.of(params.greeting) + +process SPLITLETTERS { + input: + val x + + output: + path 'chunk_*' + + """ + #!/usr/bin/env python + x="$x" + for i, word in enumerate(x.split()): + with open(f"chunk_{i}", "w") as f: + f.write(word) + """ +} + +process CONVERTTOUPPER { + input: + path y + + output: + stdout + + """ + #!/usr/bin/env python + with open("$y") as f: + print(f.read().upper()) + """ +} + +workflow { + letters_ch = SPLITLETTERS(greeting_ch) + results_ch = CONVERTTOUPPER(letters_ch.flatten()) + results_ch.view{ it } +} + From 2065946d18c96720d004a64c2d80d7bf37be0410 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:32:07 +0100 Subject: [PATCH 02/10] Fix prose `chunk_` filenames Co-authored-by: Christopher Hakkaart --- docs/basic_training/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basic_training/intro.md b/docs/basic_training/intro.md index e0841d1e..8bab4d0f 100644 --- a/docs/basic_training/intro.md +++ b/docs/basic_training/intro.md @@ -143,7 +143,7 @@ workflow { // (18)! 21. The final output (in the `results_ch` channel) is printed to screen using the `view` operator (appended onto the channel name). 22. End of the workflow scope. -This pipeline takes `params.greeting`, which defaults to the string `Hello world!`, and splits it into indiviudual words in the `SPLITLETTERS` process. Each word is written to a separate file, named `chunk_1`, `chunk_2` and so on. These files are picked up as the process `output`. +This pipeline takes `params.greeting`, which defaults to the string `Hello world!`, and splits it into indiviudual words in the `SPLITLETTERS` process. Each word is written to a separate file, named `chunk_aa`, `chunk_ab`, `chunk_ac`and so on. These files are picked up as the process `output`. The second process `CONVERTTOUPPER` takes the output channel from the first process as its input. The use of the operator `.flatten()` here is to split the two files into two separate items to be put through the this process, else they would be treated as a single element. From 63aafc643c032c4b4a737fe3ee36ec0d229c553a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:32:37 +0100 Subject: [PATCH 03/10] Remove non-functional toc header --- docs/basic_training/intro.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/basic_training/intro.md b/docs/basic_training/intro.md index 8bab4d0f..0cc07299 100644 --- a/docs/basic_training/intro.md +++ b/docs/basic_training/intro.md @@ -1,6 +1,5 @@ --- description: Getting started with Nextflow -toc_depth: 1 --- # Introduction From c8279dbf214c5046a757d96167aefcec6f0604ef Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:34:34 +0100 Subject: [PATCH 04/10] Fix prettier linting --- docs/basic_training/intro.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/basic_training/intro.md b/docs/basic_training/intro.md index 0cc07299..51824011 100644 --- a/docs/basic_training/intro.md +++ b/docs/basic_training/intro.md @@ -155,6 +155,7 @@ If you're not completely comfortable with the bash code used in the example, don ```python title="nf-training/hello_py.nf" linenums="14" --8<-- "nf-training/hello_py.nf:14:18" ``` + ```python title="nf-training/hello_py.nf" linenums="30" --8<-- "nf-training/hello_py.nf:30:32" ``` From dbefe4121f8791feac4c4bc58dea7bba8ba833e6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:38:53 +0100 Subject: [PATCH 05/10] Run prettier with npx, different output? --- docs/basic_training/intro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basic_training/intro.md b/docs/basic_training/intro.md index 51824011..4a128446 100644 --- a/docs/basic_training/intro.md +++ b/docs/basic_training/intro.md @@ -142,7 +142,7 @@ workflow { // (18)! 21. The final output (in the `results_ch` channel) is printed to screen using the `view` operator (appended onto the channel name). 22. End of the workflow scope. -This pipeline takes `params.greeting`, which defaults to the string `Hello world!`, and splits it into indiviudual words in the `SPLITLETTERS` process. Each word is written to a separate file, named `chunk_aa`, `chunk_ab`, `chunk_ac`and so on. These files are picked up as the process `output`. +This pipeline takes `params.greeting`, which defaults to the string `Hello world!`, and splits it into indiviudual words in the `SPLITLETTERS` process. Each word is written to a separate file, named `chunk_aa`, `chunk_ab`, `chunk_ac`and so on. These files are picked up as the process `output`. The second process `CONVERTTOUPPER` takes the output channel from the first process as its input. The use of the operator `.flatten()` here is to split the two files into two separate items to be put through the this process, else they would be treated as a single element. From 402b03cc0f294ff9df4b70d38338ff7c61b467c5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:42:23 +0100 Subject: [PATCH 06/10] Update conda before installing --- .github/gitpod.Dockerfile | 1 + .github/workflows/docker.yml | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/gitpod.Dockerfile b/.github/gitpod.Dockerfile index b2a63940..663bf4bc 100644 --- a/.github/gitpod.Dockerfile +++ b/.github/gitpod.Dockerfile @@ -39,6 +39,7 @@ RUN conda config --add channels defaults && \ conda config --add channels bioconda && \ conda config --add channels conda-forge && \ conda config --set channel_priority strict && \ + conda update --quiet --yes --all && \ conda install --quiet --yes --name base mamba && \ mamba install --quiet --yes --name base \ nextflow \ diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 146ccf14..f551a47e 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -2,7 +2,7 @@ # - Only pushes if push or release # - Builds without push for PRs to check the Dockerfile -name: Build Docker image +name: Build GitPod Docker image on: pull_request: @@ -14,7 +14,7 @@ on: jobs: push_to_registry: if: github.repository == 'nextflow-io/training' - name: Build + Push Docker image + name: Build + Push Gitpod Docker image runs-on: ubuntu-latest steps: - name: Check out the repo From 9bb7e818abd2662089e2037ceb99e915b10cb498 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 10:57:42 +0100 Subject: [PATCH 07/10] Set python version in base. Update actions --- .github/gitpod.Dockerfile | 5 +++-- .github/workflows/docker.yml | 8 ++++---- .github/workflows/mkdocs-docker.yml | 6 +++--- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.github/gitpod.Dockerfile b/.github/gitpod.Dockerfile index 663bf4bc..79927f58 100644 --- a/.github/gitpod.Dockerfile +++ b/.github/gitpod.Dockerfile @@ -40,8 +40,9 @@ RUN conda config --add channels defaults && \ conda config --add channels conda-forge && \ conda config --set channel_priority strict && \ conda update --quiet --yes --all && \ - conda install --quiet --yes --name base mamba && \ - mamba install --quiet --yes --name base \ + conda install --quiet --yes --name base python==3.11 mamba + +RUN mamba install --quiet --yes --name base \ nextflow \ nf-core \ pytest-workflow && \ diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index f551a47e..6e41a7fe 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -18,17 +18,17 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out the repo - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Login to GitHub Container Registry - uses: docker/login-action@v2 + uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.repository_owner }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build / push latest image - uses: docker/build-push-action@v3 + uses: docker/build-push-action@v5 if: github.event_name != 'release' with: file: .github/gitpod.Dockerfile @@ -36,7 +36,7 @@ jobs: tags: ghcr.io/nextflow-io/training:latest - name: Push release image - uses: docker/build-push-action@v3 + uses: docker/build-push-action@v5 if: github.event_name == 'release' with: file: .github/gitpod.Dockerfile diff --git a/.github/workflows/mkdocs-docker.yml b/.github/workflows/mkdocs-docker.yml index b5aa6df0..5aa28594 100644 --- a/.github/workflows/mkdocs-docker.yml +++ b/.github/workflows/mkdocs-docker.yml @@ -14,17 +14,17 @@ jobs: runs-on: ubuntu-latest steps: - name: Check out the repo - uses: actions/checkout@v3 + uses: actions/checkout@v4 - name: Login to GitHub Container Registry - uses: docker/login-action@v2 + uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.repository_owner }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build / push latest image - uses: docker/build-push-action@v3 + uses: docker/build-push-action@v5 with: file: .github/mkdocs.Dockerfile push: ${{ github.event_name == 'push' }} From 8e18d53ada0b4f0a0aba40099b190f771add634c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 12:22:43 +0100 Subject: [PATCH 08/10] Don't use mamba for docker installation Also add some other new things from the nf-core/tools gitpod image --- .github/gitpod.Dockerfile | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/.github/gitpod.Dockerfile b/.github/gitpod.Dockerfile index 79927f58..c159e550 100644 --- a/.github/gitpod.Dockerfile +++ b/.github/gitpod.Dockerfile @@ -3,6 +3,7 @@ FROM gitpod/workspace-base USER root # Install util tools. +# software-properties-common is needed to add ppa support for Apptainer installation RUN apt-get update --quiet && \ apt-get install --quiet --yes \ apt-transport-https \ @@ -13,7 +14,8 @@ RUN apt-get update --quiet && \ wget \ curl \ tree \ - graphviz + graphviz \ + software-properties-common # Install Conda RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \ @@ -24,12 +26,11 @@ ENV PATH="/opt/conda/bin:$PATH" # User permissions RUN mkdir -p /workspace/data \ - && chown -R gitpod:gitpod /workspace/data \ - && chown -R gitpod:gitpod /opt/conda + && chown -R gitpod:gitpod /opt/conda /workspace/data +# Change user to gitpod USER gitpod - # Uncomment if we need to pin the Nextflow version # ENV NXF_EDGE=1 # ENV NXF_VER=22.09.7-edge @@ -40,13 +41,19 @@ RUN conda config --add channels defaults && \ conda config --add channels conda-forge && \ conda config --set channel_priority strict && \ conda update --quiet --yes --all && \ - conda install --quiet --yes --name base python==3.11 mamba - -RUN mamba install --quiet --yes --name base \ + conda install --quiet --yes --name base \ + mamba \ nextflow \ nf-core \ + nf-test \ + black \ + prettier \ + pre-commit \ pytest-workflow && \ - mamba clean --all -f -y + conda clean --all --force-pkgs-dirs --yes + +# Update Nextflow +RUN nextflow self-update && nextflow -version RUN unset JAVA_TOOL_OPTIONS From b097d951e3ebb4ad45e34031d47051dd19bfd615 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 12:29:55 +0100 Subject: [PATCH 09/10] Update section about setting NXF_VER to instead set 23.10.0 --- docs/advanced/setup.md | 4 ++-- docs/basic_training/setup.fr.md | 4 ++-- docs/basic_training/setup.md | 4 ++-- docs/basic_training/setup.pt.md | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/advanced/setup.md b/docs/advanced/setup.md index 4f60f222..4e4f07fe 100644 --- a/docs/advanced/setup.md +++ b/docs/advanced/setup.md @@ -158,11 +158,11 @@ The latest releases can be viewed on GitHub [here](https://github.com/nextflow-i If you want to use a specific version of Nextflow, you can set the `NXF_VER` variable as shown below: ```bash -export NXF_VER=23.04.1 +export NXF_VER=23.10.0 ``` !!! Note - This tutorial workshop requires `NXF_VER=23.04.1`, or later. This version will use DSL2 as default. + This tutorial workshop requires `NXF_VER=23.10.0`, or later. Run `nextflow -version` again to confirm that the change has taken effect. diff --git a/docs/basic_training/setup.fr.md b/docs/basic_training/setup.fr.md index 7257fc2d..ecf500a1 100644 --- a/docs/basic_training/setup.fr.md +++ b/docs/basic_training/setup.fr.md @@ -162,11 +162,11 @@ Les dernières versions peuvent être consultées sur GitHub [ici](https://githu Si vous souhaitez utiliser une version spécifique de Nextflow, vous pouvez définir la variable `NXF_VER` comme indiqué ci-dessous : ```bash -export NXF_VER=23.04.1 +export NXF_VER=23.10.0 ``` !!! Remarque - Cet atelier tutoriel nécessite `NXF_VER=23.04.1`, ou une version plus récente. Cette version utilise DSL2 par défaut. + Cet atelier tutoriel nécessite `NXF_VER=23.10.0`, ou une version plus récente. Exécutez `nextflow -version` à nouveau pour confirmer que le changement a pris effet. diff --git a/docs/basic_training/setup.md b/docs/basic_training/setup.md index 582fdbd4..67b88443 100644 --- a/docs/basic_training/setup.md +++ b/docs/basic_training/setup.md @@ -162,11 +162,11 @@ The latest releases can be viewed on GitHub [here](https://github.com/nextflow-i If you want to use a specific version of Nextflow, you can set the `NXF_VER` variable as shown below: ```bash -export NXF_VER=23.04.1 +export NXF_VER=23.23.10.0 ``` !!! Note - This tutorial workshop requires `NXF_VER=23.04.1`, or later. This version will use DSL2 as default. + This tutorial workshop requires `NXF_VER=23.23.10.0`, or later. Run `nextflow -version` again to confirm that the change has taken effect. diff --git a/docs/basic_training/setup.pt.md b/docs/basic_training/setup.pt.md index 1d3f58a1..51ad87f7 100644 --- a/docs/basic_training/setup.pt.md +++ b/docs/basic_training/setup.pt.md @@ -162,11 +162,11 @@ Os últimos lançamentos podem ser vistos no GitHub [aqui](https://github.com/ne Se você deseja usar uma versão específica do Nextflow, pode definir a variável `NXF_VER` conforme mostrado abaixo: ```bash -export NXF_VER=23.04.1 +export NXF_VER=23.10.0 ``` !!! Note - Este treinamento requer `NXF_VER=23.04.1`, ou posterior. Esta versão usará a DSL2 como padrão. + Este treinamento requer `NXF_VER=23.10.0`, ou posterior. Esta versão usará a DSL2 como padrão. Execute `nextflow -version` novamente para confirmar que a alteração entrou em vigor. From cace0e4d4204f4a09963729cf4ccacee2e5f7332 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 11 Dec 2023 12:31:20 +0100 Subject: [PATCH 10/10] Fix typo --- docs/basic_training/setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/basic_training/setup.md b/docs/basic_training/setup.md index 67b88443..6b7302f3 100644 --- a/docs/basic_training/setup.md +++ b/docs/basic_training/setup.md @@ -162,11 +162,11 @@ The latest releases can be viewed on GitHub [here](https://github.com/nextflow-i If you want to use a specific version of Nextflow, you can set the `NXF_VER` variable as shown below: ```bash -export NXF_VER=23.23.10.0 +export NXF_VER=23.10.0 ``` !!! Note - This tutorial workshop requires `NXF_VER=23.23.10.0`, or later. + This tutorial workshop requires `NXF_VER=23.10.0`, or later. Run `nextflow -version` again to confirm that the change has taken effect.