Skip to content

Commit

Permalink
🚧 wip(hello_seqera): first pass at interactive compute envs
Browse files Browse the repository at this point in the history
  • Loading branch information
kenibrewer committed Oct 27, 2024
1 parent 77c234f commit 78ab515
Show file tree
Hide file tree
Showing 5 changed files with 93 additions and 21 deletions.
6 changes: 4 additions & 2 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
{
"name": "nfcore",
"image": "ghcr.io/nextflow-io/training:latest",
"build": {
"dockerfile": "../.github/gitpod.Dockerfile"
},
"remoteUser": "gitpod",
"features": {
"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {}
Expand All @@ -23,7 +25,7 @@
"ms-python.vscode-pylance",
"nf-core.nf-core-extensionpack",
"nextflow.nextflow",
"codezombiech.gitignore",
"codezombiech.gitignore"
]
}
},
Expand Down
4 changes: 1 addition & 3 deletions docs/hello_nextflow/10_hello_seqera.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,7 @@ In this section, we'll introduce you to Seqera Platform, a powerful cloud-based

[Sign up](https://cloud.seqera.io/) to try Seqera for free or request a [demo](https://seqera.io/demo/) for deployments in your own on-premise or cloud environment.

You'll learn how to use Seqera Platform within Nextflow CLI, the Seqera Platform GUI, and the API.

You can use Seqera Platform via either the **CLI**, through the **online GUI** or through the **API**.
You'll learn how to use Seqera Platform within Nextflow CLI command-line interface, the Seqera Platform GUI (Graphical User Interface), and the Seqera API (Application Program Interface).

--8<-- "docs/hello_nextflow/seqera/01_run_with_cli.md"

Expand Down
31 changes: 16 additions & 15 deletions docs/hello_nextflow/seqera/01_run_with_cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Follow these steps to set up your token:
Open a terminal and type:

```bash
export TOWER_ACCESS_TOKEN=eyxxxxxxxxxxxxxxxQ1ZTE=
export TOWER_ACCESS_TOKEN=eyxxxxxxxxxxxxxxxQ1ZTE=
```

Where `eyxxxxxxxxxxxxxxxQ1ZTE=` is the token you have just created.
Expand Down Expand Up @@ -84,30 +84,31 @@ Before we set the configuration, we need to permanently store the token in Nextf
nextflow secrets set tower_access_token "eyxxxxxxxxxxxxxxxQ1ZTE="
```
The following block of configuration will enable Seqera Platform logging by default:
We want to configure Nextflow to use Seqera Platform by default across all our pipelines, so we will open the global Nextflow configuration file (`$HOME/.nextflow/config`) for editing:
```groovy title="nextflow.config"
tower {
enabled = true
endpoint = "https://api.cloud.seqera.io"
accessToken = secrets.tower_access_token
}
```bash
code $HOME/.nextflow/config
```
However, instead of enabling Seqera Platform for an individual pipeline, we want to enable it for ourselves globally.
Run the following command to put the config block in your user configuration file located at `$HOME/.nextflow/config`.
Add the following configuration to the file:
```bash
cat <<EOF >> $HOME/.nextflow/config
```groovy title="$HOME/.nextflow/config"
tower {
enabled = true
endpoint = "https://api.cloud.seqera.io"
accessToken = secrets.tower_access_token
workspaceId = secrets.tower_workspace_id
endpoint = "https://api.cloud.seqera.io"
}
EOF
```
!!! hint "Workspace ID and Endpoint`
We haven't set `secrets.tower_workspace_id` yet, and so Nextflow will fill in an empty string for this value.
This will default to the user's workspace in Seqera Platform which is what we want for now.
The `endpoint` is the URL of the Seqera Platform API.
If your institution is running a private instance of Seqera Platform, you will want to change this to the appropriate URL.
Run your Nextflow workflows as usual:
```bash
Expand Down
71 changes: 70 additions & 1 deletion docs/hello_nextflow/seqera/03_add_compute_environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,73 @@ To practice this process, we will simulate setting up a new slurm compute enviro

### 3.1. Setup Tower Agent

### 3.2. Add a new Slurm compute environment
Most Seqera compute environments require provisioning a [credential](https://docs.seqera.io/platform/latest/credentials/overview) that grants access to those compute resources.
In this case, we will use [Tower Agent ](https://docs.seqera.io/platform/24.1/supported_software/agent), a lightweight program that can be installed on any machine to enable Seqera Platform to run Nextflow workflows on your behalf.

Follow these instructions to configure your Tower Agent.

1. Create a new Token named "GitpodAgentToken" on the tokens page, following previous instructions.
1. Export your token into your current terminal:

```bash
export TOWER_ACCESS_TOKEN=<your-token>
```

1. Within Seqera Platform, click on your workspace name on the top left, and change back to your personal user workspace.
1. Click on the "Credentials" tab
1. Add the name "gitpodTowerAgent" to the name field
1. Select "Tower Agent" from the list of providers.
1. Copy the "Agent Connection ID" shown in the dropdown, it should look similar to `75d74f5f-9454-48b6-8967-cf20b74f6c78`
1. In your terminal execute the command below replacing with your connection ID:

```bash
tw-agent 75d74f5f-9454-48b6-8967-cf20b74f6c78 --work-dir=./work
```

1. Return to Seqera Platform and click on the "Add" button.

If you completed this successfully, you'll see "gitpodTowerAgent" in the list of credentials in Seqera Platform, and console output in your terminal similar to what's below:

```console title="tw-agent logs"
21:47:33.531 INFO - Established active environments: [cli]
21:47:33.662 INFO - TOWER AGENT v0.5.0
21:47:33.662 INFO - Compatible with TOWER API v1.8
21:47:33.662 INFO - Connecting as user 'gitpod' with default work directory '/workspaces/training/work'
21:47:34.565 INFO - Connecting to Tower
21:47:34.801 INFO - Connection to Tower established
21:48:18.674 INFO - Sending heartbeat
21:48:18.755 INFO - Received heartbeat
21:49:03.680 INFO - Sending heartbeat
21:49:03.741 INFO - Received heartbeat
```

### 3.2. Add a _simulated_ Grid Engine compute environment to Seqera Plaform.

Now that we have set up a credential granting us secure access to our computational resources, we will set up a Compute Environment in Seqera which is a set of configuration that allows us to launch nextflow pipelines.
Follow the steps below:

1. Navigate to the Compute Environments tab of Seqera Platform.
1. Click "Add compute environment"
1. Name your compute environment `gitpodGridEngine`.
1. Select "Grid Engine" as the target execution platform.
1. Make sure "gitpodTowerAgent" is selected from the list of credentials.
1. Click "Add"

### 3.3. Add a pipeline to the launchpad for execution on our "Compute Environment"

Now we need to set up a pipeline in Seqera Platform to run on our simulated Grid Engine compute environment.

Follow the steps below:

1. Click on the "Launchpad" tab of Seqera Platform.
1. On the top right click on "Add Pipeline".
1. For name enter `hello-grid-engine`
1. For the Compute Environment select `gitpodGridEngine`
1. For the "Pipeline to launch" enter `nextflow-io/hello`.
1. Finally click "Add"

You now should see "hello-grid-engine` in the list of pipelines.

### 3.4. Launch your Nextflow pipeline

To create that:
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,8 @@ markdown_extensions:
preserve_tabs: true
- pymdownx.tabbed:
alternate_style: true
- pymdownx.tasklist:
custom_checkbox: true
- tables
- toc:
title: On this page
Expand Down

0 comments on commit 78ab515

Please sign in to comment.