Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation on using the Aspera CLI #1085

Merged
merged 5 commits into from
Nov 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _scicomputing/store_aspera.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Anyone with a HutchNetID can login to Aspera Faspex.

### Support

For assistance with Aspera, email helpdesk.
If you are unable to log in, contact HelpDesk. If you are able to log in but have other questions, contact SciComp.

## Sending Files to External Collaborators

Expand Down
49 changes: 49 additions & 0 deletions _scicomputing/store_aspera_shares.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
title: Aspera Shares Storage
primary_reviewers: chrisequalsdev
---

# Aspera Shares CLI

The Aspera CLI can be used to efficiently upload and download files from externally hosted Aspera servers. Note that the Aspera server hosted by Fred Hutch is restricted to web access only and does not support CLI use. This page documents how to use the CLI when logged into `rhino`. You can also use [grabnode](/compdemos/grabnode) to do your transfer on a cluster node. Additional CLI usage documentation can be found [here](https://download.asperasoft.com/download/docs/cli/3.7.2/user_osx/webhelp/dita/cli_aspera_commands_shares.html)

## Setup

> For longer running transfers, using [tmux or screen](/scicomputing/access_methods/#screen-and-tmux) is recommended.

- Load the Aspera CLI module with `ml Aspera-CLI`
- Store the password provided by the external organization as an environment variable. `export ASPERA_PASSWORD='your-password-here'`
- Store the Aspera hostname in an environment variable: `export ASPERA_HOST='your-host-here'`. You can get the hostname from the url provided by your collaborator. For example if the url you were given is https://aspera.your-collaborator.org, you would do `export ASPERA_HOST='aspera.your-collaborator.org'`

### Notes when using grabnode

- If you are using [grabnode](/compdemos/grabnode) request 1 CPU, 2GB Ram and no GPU as file transfers don't require a lot of system resources.
- If you are also using [tmux or screen](/scicomputing/access_methods/#screen-and-tmux) start your tmux or screen session first then run `grabnode` inside your session.

### Browse

First test your credentials and access by browsing:
```
aspera shares browse [email protected] --password=$ASPERA_PASSWORD --host=$ASPERA_HOST
```

You can also specify a specific path. Use quotes whenever you specify a local or remote path.
```
aspera shares browse [email protected] --password=$ASPERA_PASSWORD --host=$ASPERA_HOST --path="Parent_Folder/Child_Folder"
```

### Upload

This command will upload local file `test_file.txt` to `Parent_Folder/test_file.txt` on the Aspera server:

```
aspera shares upload [email protected] --password=$ASPERA_PASSWORD --host=$ASPERA_HOST --source="test_file.txt" --destination="Parent_Folder"
```

### Download

This command will download `Parent_Folder/test_file.txt` from the Aspera server to `/fh/fast/path/to/folder/test_file.txt`. You can also omit `--destination` to download to the current working directory.

```
aspera shares download [email protected] --password=$ASPERA_PASSWORD --host=$ASPERA_HOST --source="Parent_Folder/test_file.txt" --destination="/fh/fast/path/to/folder"
```
10 changes: 8 additions & 2 deletions _scicomputing/store_collaboration.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,17 @@ In collaborative settings, often data transfer and sharing can pose significant

### Aspera

Aspera is a data transfer tool that enables fast transfer of large data between it and a host using the Aspera client. The primary method of operation is to upload the data to the Aspera server, then use the web interface to create an email with a link you would then send to those outside the Hutch network.
#### Faspex

Aspera Faspex is a web based data transfer tool that enables fast transfer of large data between it and a host using the Aspera client. The primary method of operation is to upload the data to the Aspera server, then use the web interface to create an email with a link you would then send to those outside the Hutch network.

> Note: space is limited. Because of this, data stored here is deleted after a short period of time and not backed up making the Aspera inappropriate for primary storage. Moving data to their final, secure, backed up locations as soon as possible is important when using Aspera.

Visit [the Aspera information page](/scicomputing/store_aspera/) for more details and information on using this storage service.
Visit [the Aspera Faspex information page](/scicomputing/store_aspera/) for more details and information on using this storage service.

#### Shares

Aspera Shares is similar to the web interface used by Faspex but additionally offers a command line interface (CLI). While the Fred Hutch instance of Aspera does not support CLI use, the organization you are collaborating with might. Visit [the Aspera Shares information page](/scicomputing/store_aspera_shares/) for more how to use the Aspera CLI tool on our cluster.

## Data Storage for Collaboration

Expand Down