Skip to content

Commit

Permalink
Reviewed globus-compute README.md. (#130)
Browse files Browse the repository at this point in the history
  • Loading branch information
@adamhgriffith-uofu
adamhgriffith-uofu authored May 19, 2023
1 parent 54a8c8e commit fc651db
Show file tree
Hide file tree
Showing 2 changed files with 82 additions and 76 deletions.
2 changes: 1 addition & 1 deletion charts/globus-compute/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.1.5
version: 0.1.6

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
Expand Down
156 changes: 81 additions & 75 deletions charts/globus-compute/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,116 +1,122 @@
# Kubernetes Endpoint
This chart will deploy a functioning Kubernetes endpoint into your SLATE cluster. It
will launch workers with a specified container image into a namespace.
# Globus Compute Endpoint

This SLATE application will deploy a functioning Kubernetes endpoint into your SLATE cluster and will launch worker pods with a specified container image into a namespace.

## How to Use
There are two required values to specify in the `values.yaml` file:
`endpointUUID` and authentication. The `endpointUUID` is easy:

```
endpointUUID: <someuuid>
There are two required values to specify in the `values.yaml` file, `endpointUUID` and authentication.

### `endpointUUID`

The UUID is of your choosing, but must already be available. In particular, if the UUID you choose has already been taken, the endpoint will fail to register.

```yaml
# app config:
endpointUUID: <your-uuid>
```
The UUID is of your choosing, but must be available. In particular, if the
UUID you choose has already been taken, the endpoint will fail to register.
One method to generate a UUID is with the `uuid` command line tool:
#### Generate Locally
One method to generate a UUID is with the `uuid` command line tool. For example, on a Debian host generate a version 1 UUID:

```shell
$ sudo apt install uuid
...
$ uuid # will generate a version 1 UUID.
$ uuid
e22be136-b3eb-11ed-8611-5b7bc2d2f962
```

Alternatively, Python has the builtin `uuid` module:
```python
Alternatively, use Python's builtin `uuid` module:

```shell
$ python
...
>>> import uuid
>>> uuid.uuid4()
UUID('ea0cab7e-b3eb-11ed-ae8b-719a5541eacb')
UUID('e22be136-b3eb-11ed-8611-5b7bc2d2f962')
```

Getting the authentication setup is slightly more involved. Under the hood,
the Globus Compute Endpoint uses the Globus Compute SDK for communication with the web services,
which requires an authenticated user for most API routes. The Globus Compute SDK can
use either client credentials or user credentials. This README shows how to implement the client credentials.
### Authentication

Getting the authentication setup is slightly more involved.
* Under the hood, the Globus Compute Endpoint uses the [Globus Compute SDK](https://globus-sdk-python.readthedocs.io/en/stable/) for communication with the web services.
* This requires an authenticated user for most API routes.
* The Globus Compute SDK can use either client credentials or user credentials.

This document describes how to implement the client credentials below.

#### Client Credentials
The Globus Compute SDK supports use of Globus Auth Client Credentials. In practice,
that means exporting two variables into the endpoint's environment:

* `FUNCX_SDK_CLIENT_ID`
* `FUNCX_SDK_CLIENT_SECRET`
The Globus Compute SDK supports use of Globus Auth Client Credentials. In practice, that means exporting two variables into the endpoint's environment:

These variables may be generated by following the steps in the [Registering an
Application](https://docs.globus.org/api/auth/developer-guide/#register-app)
section on the [Globus Auth Developer's
Guide](https://docs.globus.org/api/auth/developer-guide/).
1. `FUNCX_SDK_CLIENT_ID`
1. `FUNCX_SDK_CLIENT_SECRET`

Outside of this chart, use of client credentials is also documented for [normal
Globus Compute SDK
These variables may be generated by following the steps in the [Registering an Application](https://docs.globus.org/api/auth/developer-guide/#register-app) section on the [Globus Auth Developer's Guide](https://docs.globus.org/api/auth/developer-guide/). Outside of this application, use of client credentials is also documented for [normal Globus Compute SDK
usage](https://funcx.readthedocs.io/en/latest/sdk.html#client-credentials-with-globus-compute-clients).

Add these variables to a secret object in Kubernetes. For example, to put them
into a Kubernetes store named `my-secrets`, you could create a temporary env file
and load them:
Add these variables to a secret object in Kubernetes. For example, to put them into a Kubernetes store named `my-secrets`, you could create a temporary `.env` file and load them. For example:

```
```shell
$ (umask 077; touch client_creds.env) # create with 0600 (-rw-------) perms
$ cat > client_creds.env
FUNCX_SDK_CLIENT_ID=11111111-2222-4444-8888-000000000000
FUNCX_SDK_CLIENT_SECRET=yoursecret
^D
$ slate secret create my-secrets --grouop <your-group> --cluster <your-cluster> --from-env-file ./client_creds.env
$ slate secret create my-secrets --group <your-group> --cluster <your-cluster> --from-env-file ./client_creds.env
```

Then, specify the secret name in the configuration file, and tell the chart to use
the client credentials:
```
Finally, specify the secret name in the configuration file, and tell the application to use the client credentials.

```yaml
# app config:
secrets: my-secrets
useClientCredentials: true
```

## Install the Globus Compute Endpoint
## Installation

Download the configuration file:
1. Download the application configuration file:

```sehll script
slate app get-conf globus-compute-endpoint > your-config.yaml
```
```shell
slate app get-conf globus-compute-endpoint > your-config.yaml
```

Update the configuration file with the parameters covered above, and install the application with:
1. Update the file with the parameters covered above and install the application.

```shell script
slate app install globus-compute-endpoint --group <your-group> --cluster <your-cluster> --config your-config.yaml
```
```shell
slate app install globus-compute-endpoint --group <your-group> --cluster <your-cluster> --config your-config.yaml
```

## Values
The deployment is configured via values.yaml file.

| Value | Description | Default |
|-------| ----------- | ------- |
| Globus ComputeServiceAddress | URL for the FuncX Webservice. | https://api.funcx.org |
| image.repository | Docker image repository | funcx/kube-endpoint |
| image.tag | Tag name for the endpoint image | endpoint_helm |
| image.pullPolicy | Pod pull policy for the endpoint image | Always |
| workerDebug | Log additional information in the worker logs | False |
| workerImage | Docker image to run in the worker pods | python:3.6-buster |
| workerInit | Command to execute on worker before strating uip | pip install parsl==0.9.0;pip install --force-reinstall globus-compute-sdk>=2.0.0 |
| workerNamespace | Kubernetes namespace to launch worker pods into | default |
| workingDir | Directory inside the container where log files are to be stored | /tmp/worker_logs |
| rbacEnabled | Create service account and roles? | true |
| initMem | Initial memory for worker pod | 2000Mi |
| maxMem| Maximum allowed memory for worker pod | 16000Mi |
| initCPU | Initial CPUs to allocate to worker pod | 1 |
| maxCPU | Maximum CPUs to allocate to worker pod | 2 |
| maxBlocks | Maximum number of worker pods to spawn | 100 |
| maxWorkersPerPod | How many workers will be scheduled in each pod | 1 |
| taskTTLSeconds | (Optional) If set, will stop tasks that run longer than this value, in (fractional) seconds. Example: 1.5 | |
| endpointName | (Optional) Specify a name for registration with the funcX web services | The release name (Release.Name) |
| endpointDisplayName | (Optional) Specify a display name for registration with the funcX web services | The endpoint name (endpointName) or the release name (Release.Name) |
| endpointUUID | (Required) Specify a UUID for this endpoint. | |
| endpointCLIargs | Any additional command line arguments to give to the `globus-compute-endpoint` executable | |
| maxIdleTime | The maximum time to maintain an idle worker. After this time the SimpleStrategy will terminate the idle worker. | 3600 |
| imagePullSecret | The K8s secret to use to deploy worker images. This can refer to an ECR secret. | |
| secrets | Kubernetes secret object in which to find client credential environment variables | |
| useClientCredentials | Whether to use _client_ credentials | false |
| useUserCredentials | Whether to use _user_ credentials (i.e., `storage.db`) | false |

The deployment is configured via the application configuration file.

| Value | Description | Default | Required |
|-------|-----------------------------------------------------------------------------------------------------------------| ------- |----------|
| Globus ComputeServiceAddress | URL for the FuncX Webservice. | https://api.funcx.org | yes |
| image.repository | Container image repository | funcx/kube-endpoint | yes |
| image.tag | Tag name for the endpoint image | endpoint_helm | yes |
| image.pullPolicy | Pod pull policy for the endpoint image | Always | yes |
| workerDebug | Log additional information in the worker logs | False | yes |
| workerImage | Container image to run in the worker pods | python:3.6-buster | yes |
| workerInit | Command to execute on worker before strating uip | pip install parsl==0.9.0;pip install --force-reinstall globus-compute-sdk>=2.0.0 | yes |
| workerNamespace | Kubernetes namespace to launch worker pods into | default | yes |
| workingDir | Directory inside the container where log files are to be stored | /tmp/worker_logs | yes |
| rbacEnabled | Create service account and roles? | true | yes |
| initMem | Initial memory for worker pod | 2000Mi | yes |
| maxMem| Maximum allowed memory for worker pod | 16000Mi | yes |
| initCPU | Initial CPUs to allocate to worker pod | 1 | yes |
| maxCPU | Maximum CPUs to allocate to worker pod | 2 | yes |
| maxBlocks | Maximum number of worker pods to spawn | 100 | yes |
| maxWorkersPerPod | How many workers will be scheduled in each pod | 1 | yes |
| taskTTLSeconds | If set, will stop tasks that run longer than this value, in (fractional) seconds. Example: 1.5 | | no |
| endpointName | Specify a name for registration with the funcX web services | The release name (Release.Name) | no |
| endpointDisplayName | Specify a display name for registration with the funcX web services | The endpoint name (endpointName) or the release name (Release.Name) | no |
| endpointUUID | Specify a UUID for this endpoint. | | yes |
| endpointCLIargs | Any additional command line arguments to give to the `globus-compute-endpoint` executable | | yes |
| maxIdleTime | The maximum time to maintain an idle worker. After this time the SimpleStrategy will terminate the idle worker. | 3600 | yes |
| imagePullSecret | The K8s secret to use to deploy worker images. This can refer to an ECR secret. | | yes |
| secrets | Kubernetes secret object in which to find client credential environment variables | | yes |
| useClientCredentials | Whether to use _client_ credentials | false | yes |
| useUserCredentials | Whether to use _user_ credentials (i.e., `storage.db`) | false | yes |

0 comments on commit fc651db

Please sign in to comment.