-
Notifications
You must be signed in to change notification settings - Fork 102
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(nomad): improved nomad support (#798)
This PR adds better support for our integration with Nomad. Although not officially supported, we would like to improve the developer experience and integrate automated e2e tests to avoid breaking Nomad integrations with future csi-driver releases. The Terraform module for creating a Nomad (+Consul) cluster on Hetzner Cloud can be found [here](https://github.com/hetznercloud/nomad-dev-env). --------- Co-authored-by: Julian Tölle <[email protected]> Co-authored-by: Jonas L. <[email protected]>
- Loading branch information
3 people
authored
Dec 18, 2024
1 parent
d0eba3e
commit 4dbea7d
Showing
17 changed files
with
881 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -115,3 +115,77 @@ jobs: | |
| if: always() | ||
| continue-on-error: true | ||
| run: make -C dev down | ||
|
|
||
| nomad: | ||
| name: nomad ${{ matrix.nomad }} | ||
| runs-on: ubuntu-latest | ||
|
|
||
| permissions: | ||
| id-token: write | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }}-${{ matrix.nomad }} | ||
| cancel-in-progress: true | ||
|
|
||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: | ||
| - nomad: 1.9.3 # renovate: datasource=github-releases depName=hashicorp/nomad | ||
| consul: 1.20.1 # renovate: datasource=github-releases depName=hashicorp/consul | ||
|
|
||
| env: | ||
| TF_VAR_nomad_version: ${{ matrix.nomad }} | ||
| TF_VAR_consul_version: ${{ matrix.consul }} | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: "1.23" | ||
|
|
||
| - uses: opentofu/setup-opentofu@v1 | ||
| with: | ||
| tofu_version: v1.8.6 # renovate: datasource=github-releases depName=opentofu/opentofu | ||
| tofu_wrapper: false | ||
|
|
||
| - uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - uses: yokawasa/[email protected] | ||
| with: | ||
| setup-tools: | | ||
| skaffold | ||
| skaffold: v2.13.2 # renovate: datasource=github-releases depName=GoogleContainerTools/skaffold | ||
|
|
||
| - uses: hashicorp/setup-nomad@main | ||
| id: setup | ||
| with: | ||
| version: ${{ matrix.nomad }} | ||
|
|
||
| # used for generating the certificates | ||
| - name: setup consul binary | ||
| run: | | ||
| curl -o consul.zip https://releases.hashicorp.com/consul/${{ matrix.consul }}/consul_${{ matrix.consul }}_linux_amd64.zip | ||
| unzip consul.zip | ||
| mv consul /usr/local/bin/ | ||
| - uses: hetznercloud/tps-action@main | ||
|
|
||
| - name: Setup environment | ||
| run: make -C nomad/dev up | ||
|
|
||
| - name: Run skaffold | ||
| run: | | ||
| source nomad/dev/files/env.sh | ||
| skaffold -f nomad/skaffold.yaml build | ||
| - name: Run tests | ||
| run: | | ||
| source nomad/dev/files/env.sh | ||
| go test -v -tags e2e ./test/e2e/nomad/... | ||
| - name: Cleanup | ||
| if: always() | ||
| continue-on-error: true | ||
| run: make -C nomad/dev down | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # Nomad | ||
|
|
||
| Hetzner Cloud does not provide official support for running the CSI driver in Nomad. Nonetheless, we would still like to offer a satisfying developer experience and have automated e2e tests to avoid breaking Nomad accidentally. | ||
|
|
||
| ## Nomad Development Environment | ||
|
|
||
| As a prerequisite for developing Nomad, a setup of the [`nomad-dev-env`](https://github.com/hetznercloud/nomad-dev-env) is necessary, which is located in `nomad/dev`. | ||
|
|
||
| 1. Setup the `HCLOUD_TOKEN` environment variable | ||
| 2. Deploy the development cluster: | ||
|
|
||
| ```bash | ||
| make -C nomad/dev up | ||
| ``` | ||
|
|
||
| 3. Load the generated configuration to access the development cluster: | ||
|
|
||
| ```bash | ||
| source nomad/dev/files/env.sh | ||
| ``` | ||
|
|
||
| 4. Check that the cluster is healthy: | ||
|
|
||
| ```bash | ||
| nomad node status | ||
| ``` | ||
|
|
||
| ## Skaffold | ||
|
|
||
| Skaffold commands should be executed from the `csi-driver` root directory and use the `-f` flag to point to the Nomad specific `skaffold.yaml`. | ||
|
|
||
| ```bash | ||
| skaffold -f nomad/skaffold.yaml build | ||
| ``` | ||
|
|
||
| Skaffold does not offer any native support for Nomad. For this reason we use the Nomad post build hooks to deploy/redeploy the csi plugin. To delete the csi plugin a manual execution of `stop_nomad.sh` is necessary. | ||
|
|
||
| ```bash | ||
| bash ./nomad/stop_nomad.sh | ||
| ``` | ||
|
|
||
| ## E2E Tests | ||
|
|
||
| The nomad e2e tests are located in `test/e2e/nomad` and need a working development environment. | ||
|
|
||
| 1. Deploy the csi-driver | ||
|
|
||
| ```bash | ||
| skaffold -f nomad/skaffold.yaml build | ||
| ``` | ||
|
|
||
| 2. Run the e2e tests | ||
|
|
||
| ```bash | ||
| go test -v -tags e2e ./test/e2e/nomad/... | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| .terraform/ | ||
| terraform.tfstate* | ||
| .terraform.tfstate* | ||
| *.auto.tfvars | ||
|
|
||
| files/ |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| SHELL = bash | ||
|
|
||
| ENV ?= dev | ||
|
|
||
| env.auto.tfvars: | ||
| @echo 'hcloud_token = "$(HCLOUD_TOKEN)"' >> "$@" | ||
|
|
||
| .terraform: | ||
| tofu init | ||
|
|
||
| validate: .terraform | ||
| tofu validate | ||
|
|
||
| up: .terraform env.auto.tfvars | ||
| tofu apply -auto-approve | ||
| $(MAKE) port-forward | ||
|
|
||
| down: .terraform env.auto.tfvars | ||
| if test -f files/registry-port-forward.sh; then files/registry-port-forward.sh down; fi | ||
| tofu destroy -auto-approve | ||
|
|
||
| port-forward: | ||
| files/registry-port-forward.sh up | ||
|
|
||
| clean: | ||
| rm -Rf files/ .terraform/ terraform.tfstate* env.auto.tfvars |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| module "dev" { | ||
| source = "github.com/hetznercloud/nomad-dev-env?ref=v0.1.0" # renovate: datasource=github-releases depName=hetznercloud/nomad-dev-env | ||
|
|
||
| hcloud_token = var.hcloud_token | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| variable "hcloud_token" { | ||
| sensitive = true | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| job "hcloud-csi-controller" { | ||
| datacenters = ["dc1"] | ||
| namespace = "default" | ||
| type = "service" | ||
|
|
||
| group "controller" { | ||
| count = 1 | ||
|
|
||
| constraint { | ||
| distinct_hosts = true | ||
| } | ||
|
|
||
| update { | ||
| max_parallel = 1 | ||
| canary = 1 | ||
| min_healthy_time = "10s" | ||
| healthy_deadline = "1m" | ||
| auto_revert = true | ||
| auto_promote = true | ||
| } | ||
|
|
||
| task "plugin" { | ||
| driver = "docker" | ||
|
|
||
| config { | ||
| image = "$SKAFFOLD_IMAGE" | ||
| command = "bin/hcloud-csi-driver-controller" | ||
| } | ||
|
|
||
| env { | ||
| CSI_ENDPOINT = "unix://csi/csi.sock" | ||
| ENABLE_METRICS = true | ||
| } | ||
|
|
||
| template { | ||
| data = <<EOH | ||
| HCLOUD_TOKEN="{{ with nomadVar "secrets/hcloud" }}{{ .hcloud_token }}{{ end }}" | ||
| EOH | ||
| destination = "${NOMAD_SECRETS_DIR}/hcloud-token.env" | ||
| env = true | ||
| } | ||
|
|
||
| csi_plugin { | ||
| id = "csi.hetzner.cloud" | ||
| type = "controller" | ||
| mount_dir = "/csi" | ||
| } | ||
|
|
||
| resources { | ||
| cpu = 100 | ||
| memory = 64 | ||
| } | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.