From 648143c31000eca13636f7c72bdc08338ce6c1f8 Mon Sep 17 00:00:00 2001 From: "svenja.michal" Date: Mon, 18 Nov 2024 10:08:18 +0100 Subject: [PATCH 1/2] Update the tutorial --- .../install-nomad-consul-cluster/01.en.md | 235 ++++++++---------- 1 file changed, 102 insertions(+), 133 deletions(-) diff --git a/tutorials/install-nomad-consul-cluster/01.en.md b/tutorials/install-nomad-consul-cluster/01.en.md index d287c2886..5cbba2357 100644 --- a/tutorials/install-nomad-consul-cluster/01.en.md +++ b/tutorials/install-nomad-consul-cluster/01.en.md @@ -2,13 +2,13 @@ SPDX-License-Identifier: MIT path: "/tutorials/install-nomad-consul-cluster" slug: "install-nomad-consul-cluster" -date: "2021-09-28" +date: "2024-11-18" title: "Install a Nomad cluster with Consul on cloud servers" short_description: "Guide on how to install and set up a HashiCorp Nomad cluster with Consul on Hetzner Cloud servers." tags: ["Nomad", "Consul", "Cluster"] author: "Dominik Langer" author_link: "https://github.com/oniumy" -author_img: "https://avatars.githubusercontent.com/u/31653315?v=4" +author_img: "https://avatars.githubusercontent.com/u/31653315" author_description: "" language: "en" available_languages: ["en"] @@ -18,7 +18,7 @@ cta: "cloud" ## Introduction -In this tutorial we will set up a HashiCorp [Nomad](https://www.nomadproject.io/) cluster with [Consul](https://www.consul.io/) for service and node discovery. With 3 server nodes and a custom amount of client nodes this should serve as a basis for growing projects. We will also create a hetzner cloud snapshot for our clients that enables us to add further clients without any manual setup. +In this tutorial we will set up a HashiCorp [Nomad](https://www.nomadproject.io/) cluster with [Consul](https://www.consul.io/) for service and node discovery. With 3 server nodes and a custom amount of client nodes this should serve as a basis for growing projects. We will also create a Hetzner Cloud Snapshot for our clients that enables us to add further clients without any manual setup. The cluster will run on a private network between the servers and supports all out of the box features of Nomad and Consul, like service discovery and volumes. > This tutorial follows in part the recommended steps from the official [Consul deployment guide](https://learn.hashicorp.com/tutorials/consul/deployment-guide) and [Nomad deployment guide](https://learn.hashicorp.com/tutorials/nomad/production-deployment-guide-vm-with-consul). @@ -26,10 +26,10 @@ The cluster will run on a private network between the servers and supports all o **Prerequisites** * A [Hetzner Cloud](https://www.hetzner.com/cloud) account -* Basic knowledge of linux commands and the shell +* Basic knowledge of Linux commands and the shell * The ability to connect to a server with `ssh` -*This tutorial was tested on Ubuntu 20.04 Hetzner Cloud servers with Nomad 1.1.3 and Consul 1.10.1* +*This tutorial was tested on Ubuntu 24.04 Hetzner Cloud servers with Nomad 1.9.3 and Consul 1.20.1* ### Terminology and Notation @@ -44,78 +44,30 @@ server$ # This command must be executed on the server as root In this step the following resource will be used -* 1 Hetzner cloud CX11 server +* 1 Hetzner cloud CX22 server -We will start by setting up a Consul / Nomad server on a new CX11 cloud server. The resulting snapshot will serve as the base image for all cluster servers and clients in the following steps. +We will start by setting up a Consul / Nomad server on a new CX22 cloud server. The resulting Snapshot will serve as the base image for all cluster servers and clients in the following steps. > This guide will show the setup for 3 servers. This will make the cluster highly-available, but not too expensive. Single server setups are not recommended, but possible. The tutorial will include comments on what to change for only 1 or more than 3 servers in the respective steps. -Go to the Hetzner cloud web interface and create a new CX11 server with Ubuntu 20.04. +Go to the Hetzner cloud web interface and create a new CX22 server with Ubuntu 24.04. -### Step 1.1 - Update the server and install needed packages +### Step 1.1 - Install Consul -To make sure everything is up-to-date, you should update the servers after creation. Connect to your server over ssh and run +Install Consul -```bash -server$ apt update -server$ apt upgrade -``` - -Now we can install `unzip` - -```bash -server$ apt install unzip -``` - -For the following steps, make sure to be in the `/root/` folder - -```bash -server$ cd /root/ -``` - -### Step 1.2 - Install the Consul binary - -To install HashiCorp Consul, we need to download and install the respective binary. First, define the version and host in an environment variable - -```bash -server$ export CONSUL_VERSION="1.10.1" -server$ export CONSUL_URL="https://releases.hashicorp.com/consul" -``` - -> You can use the latest available version found on the [official website](https://developer.hashicorp.com/consul/downloads). Changing the CONSUL_VERSION variable is sufficient. - -Download the binary, decompress it and install it on your server +> For more information about available versions, see the [official website](https://developer.hashicorp.com/consul/downloads). ```bash -server$ curl --silent --remote-name ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip -server$ unzip consul_${CONSUL_VERSION}_linux_amd64.zip -server$ chown root:root consul -server$ mv consul /usr/local/bin/ -server$ rm consul_${CONSUL_VERSION}_linux_amd64.zip +server$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg +server$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list +server$ apt update && apt install consul ``` We can now add autocomplete functionality for Consul (optional) ```bash server$ consul -autocomplete-install -server$ complete -C /usr/local/bin/consul consul -``` - -Create a user for Consul - -```bash -server$ useradd --system --home /etc/consul.d --shell /bin/false consul -server$ mkdir --parents /opt/consul -server$ chown --recursive consul:consul /opt/consul -``` - -Prepare the Consul configuration - -```bash -server$ mkdir --parents /etc/consul.d -server$ touch /etc/consul.d/consul.hcl -server$ chown --recursive consul:consul /etc/consul.d -server$ chmod 640 /etc/consul.d/consul.hcl ``` Prepare the TLS certificates for Consul @@ -146,45 +98,22 @@ dc1-client-consul-0.pem dc1-client-consul-0-key.pem ``` -### Step 1.3 - Install the Nomad binary +### Step 1.2 - Install the Nomad binary -Similar to the Consul binary, we first define the version as a variable +Install nomad -```bash -server$ export NOMAD_VERSION="1.1.3" -``` - -> You can use the latest available version found on the [official website](https://developer.hashicorp.com/nomad/downloads). Changing the NOMAD_VERSION variable is sufficient. - -Download and install the binary +> For more information about available versions, see the [official website](https://developer.hashicorp.com/nomad/downloads). ```bash -server$ curl --silent --remote-name https://releases.hashicorp.com/nomad/${NOMAD_VERSION}/nomad_${NOMAD_VERSION}_linux_amd64.zip -server$ unzip nomad_${NOMAD_VERSION}_linux_amd64.zip -server$ chown root:root nomad -server$ mv nomad /usr/local/bin/ -server$ rm nomad_${NOMAD_VERSION}_linux_amd64.zip +server$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg +server$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list +server$ apt update && apt install nomad ``` Add autocomplete functionality to nomad (optional) ```bash server$ nomad -autocomplete-install -server$ complete -C /usr/local/bin/nomad nomad -``` - -Prepare the data directory - -```bash -server$ mkdir --parents /opt/nomad -``` - -Create the basic configuration file for nomad - -```bash -server$ mkdir --parents /etc/nomad.d -server$ chmod 700 /etc/nomad.d -server$ touch /etc/nomad.d/nomad.hcl ``` Add the following configuration in the file `/etc/nomad.d/nomad.hcl` @@ -194,11 +123,21 @@ datacenter = "dc1" data_dir = "/opt/nomad" ``` -### Step 1.4 - Prepare the systemd services +### Step 1.3 - Prepare the systemd services Consul and Nomad should start automatically after boot. To enable this, create a systemd service for both of them. -Add the configuration file `/etc/systemd/system/consul.service` with the following content +First, set all permissions: + +```bash +server$ chown consul:consul dc1-server-consul* +server$ chown consul:consul dc1-client-consul* +server$ chown -R consul:consul /opt/consul +server$ chown -R nomad:nomad /opt/nomad +server$ mkdir -p /opt/alloc_mounts && chown -R nomad:nomad /opt/alloc_mounts +``` + +Add the configuration file `/etc/systemd/system/consul.service` with the following content: ```config [Unit] @@ -209,10 +148,10 @@ After=network-online.target ConditionFileNotEmpty=/etc/consul.d/consul.hcl [Service] -Type=exec +EnvironmentFile=-/etc/consul.d/consul.env User=consul Group=consul -ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ +ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/ ExecReload=/bin/kill --signal HUP $MAINPID KillMode=process KillSignal=SIGTERM @@ -223,26 +162,29 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -and the configuration file `/etc/systemd/system/nomad.service` with the content +And the configuration file `/etc/systemd/system/nomad.service` with the content: ```config [Unit] Description=Nomad -Documentation=https://developer.hashicorp.com/nomad/docs +Documentation=https://www.nomadproject.io/docs/ Wants=network-online.target After=network-online.target [Service] + +User=nomad +Group=nomad + ExecReload=/bin/kill -HUP $MAINPID -ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d +ExecStart=/usr/bin/nomad agent -config /etc/nomad.d KillMode=process KillSignal=SIGINT -LimitNOFILE=infinity +LimitNOFILE=65536 LimitNPROC=infinity Restart=on-failure RestartSec=2 -StartLimitBurst=3 -StartLimitIntervalSec=10 +OOMScoreAdjust=-1000 TasksMax=infinity [Install] @@ -251,11 +193,11 @@ WantedBy=multi-user.target > Don't enable these services yet, since the setup is not complete. -### Step 1.5 - Create the base snapshot +### Step 1.5 - Create the base Snapshot -Finally, stop the server in the Hetzner cloud web interface and create a snapshot. This snapshot will be used as a foundation for the server and client setup of the cluster. +Finally, stop the server in the Hetzner [Cloud Console](https://console.hetzner.cloud/projects) and create a Snapshot. This Snapshot will be used as a foundation for the server and client setup of the cluster. -After the snapshot creation was successful, delete the CX11 instance of this step. +After the Snapshot creation was successful, delete the CX22 instance of this step. ## Step 2 - Set up the cluster servers @@ -265,14 +207,14 @@ The cloud servers used in this step will be part of the final cluster, so make s We will use the following resources -* 1 Hetzner cloud network -* 3 Hetzner cloud CX11 server +* 1 Hetzner Cloud Network +* 3 Hetzner Cloud CX22 server -In the Hetzner cloud web interface, create 3 CX11 servers from the snapshot created in Step 1 and a common cloud network attached. This tutorial will use a `10.0.0.0/8` network, but smaller networks work as well. +In the Hetzner [Cloud Console](https://console.hetzner.cloud/projects), create 3 CX22 servers from the Snapshot created in Step 1 and a common Cloud Network attached. This tutorial will use a `10.0.0.0/8` network, but smaller networks work as well. In the following steps, the tutorial will refer to the servers by their internal IP addresses `10.0.0.2`, `10.0.0.3` and `10.0.0.4`. If your servers have different internal addresses, make sure to replace them in the following steps. -> If you plan to only run a single cluster server, one cloud server is enough in this step. You still need the cloud network, since it will be used by the clients as well. +> If you plan to only run a single cluster server, one cloud server is enough in this step. You still need the Cloud Network, since it will be used by the clients as well. ### Step 2.1 - Create the symmetric encryption key @@ -286,7 +228,7 @@ server$ consul keygen ### Step 2.2 - Distribute the certificates -Now we can copy the right certificates from step 1 to the Consul configuration directory. Run the following command on all servers +Now we can copy the right certificates from step 1 to the Consul configuration directory. Run the following command on all servers: ```bash server$ cp consul-agent-ca.pem /etc/consul.d/ @@ -294,6 +236,12 @@ server$ cp consul-agent-ca.pem /etc/consul.d/ The 3 certificates created in step 1 need to be distributed, so that every server gets a unique certificate with matching key. This tutorial will copy `dc1-server-consul-0*` on the server `10.0.0.2` into the folder `/etc/consul.d/`. Similar, copy `dc1-server-consul-1*` on `10.0.0.3` and `dc1-server-consul-2*` on `10.0.0.4`. +```bash +[10.0.0.2] server$ cp -a dc1-server-consul-0* /etc/consul.d/ +[10.0.0.3] server$ cp -a dc1-server-consul-1* /etc/consul.d/ +[10.0.0.4] server$ cp -a dc1-server-consul-2* /etc/consul.d/ +``` + To check if you have all the files you need, you should get a similar output for your servers (X being the respective certificate number) ```console @@ -319,12 +267,18 @@ On all servers, edit the configuration file `/etc/consul.d/consul.hcl` and add t datacenter = "dc1" data_dir = "/opt/consul" encrypt = "your-symmetric-encryption-key" -ca_file = "/etc/consul.d/consul-agent-ca.pem" -cert_file = "/etc/consul.d/dc1-server-consul-0.pem" -key_file = "/etc/consul.d/dc1-server-consul-0-key.pem" -verify_incoming = true -verify_outgoing = true -verify_server_hostname = true +tls { + defaults { + ca_file = "/etc/consul.d/consul-agent-ca.pem" + cert_file = "/etc/consul.d/dc1-server-consul-0.pem" + key_file = "/etc/consul.d/dc1-server-consul-0-key.pem" + verify_incoming = true + verify_outgoing = true + }, + internal_rpc { + verify_server_hostname = true + } +} retry_join = ["10.0.0.2"] bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}" @@ -390,13 +344,22 @@ server$ systemctl enable consul server$ systemctl enable nomad ``` -and start the services +And start the services ```bash server$ systemctl start consul server$ systemctl start nomad ``` +And check the status + +> If you run into any issues, you can add `log_level = "debug"` in `/etc/consul.d/consul.hcl` or `/etc/nomad.d/nomad.hcl`, restart the service, and run `journalctl -u consul` or `journalctl -u nomad`. + +```bash +server$ systemctl status consul +server$ systemctl status nomad +``` + If everything went well, you should now have your cluster up and running. Wait up to 30 seconds for the servers to sync and elect a leader. To check the cluster, run the following command on one of your servers @@ -418,13 +381,13 @@ server$ nomad acl bootstrap > This command only works once. Make sure to save the acl token in a secure location. -This gives you the `Secret ID` used for all Nomad requests. With that copied, create a variable with your secret ID +This gives you the `Secret ID` used for all Nomad requests. With that copied, create a variable with your secret ID: ```bash server$ export NOMAD_TOKEN="your-secret-id" ``` -and check the status of the Nomad cluster +And check the status of the Nomad cluster: ```bash server$ nomad server members @@ -434,13 +397,13 @@ Similar to Consul, you should get a list of all servers. ## Step 3 - Create the client image -In this step you will set up a client. The snapshot will serve as an image for easy deployment of multiple clients. +In this step you will set up a client. The Snapshot will serve as an image for easy deployment of multiple clients. We will use the following resources -* 1 Hetzner cloud CX11 server +* 1 Hetzner Cloud CX22 server -In the Hetzner cloud web interface, create 1 CX11 servers from the snapshot created in step 1. The cloud server name is not important in this step, since we will only prepare the image on it. +In the Hetzner [Cloud Console](https://console.hetzner.cloud/projects), create 1 CX22 server from the Snapshot created in step 1. The cloud server name is not important in this step, since we will only prepare the image on it. ### Step 3.1 - Configure Consul @@ -451,7 +414,7 @@ Run the following commands on the cloud server ```bash server$ cd /root/ server$ cp consul-agent-ca.pem /etc/consul.d/ -server$ cp dc1-client-consul-0* /etc/consul.d/ +server$ cp -a dc1-client-consul-0* /etc/consul.d/ server$ rm *.pem ``` @@ -461,12 +424,18 @@ Open the configuration file `/etc/consul.d/consul.hcl` and add the content datacenter = "dc1" data_dir = "/opt/consul" encrypt = "your-symmetric-encryption-key" -ca_file = "/etc/consul.d/consul-agent-ca.pem" -cert_file = "/etc/consul.d/dc1-client-consul-0.pem" -key_file = "/etc/consul.d/dc1-client-consul-0-key.pem" -verify_incoming = true -verify_outgoing = true -verify_server_hostname = true +tls { + defaults { + ca_file = "/etc/consul.d/consul-agent-ca.pem" + cert_file = "/etc/consul.d/dc1-client-consul-0.pem" + key_file = "/etc/consul.d/dc1-client-consul-0-key.pem" + verify_incoming = true + verify_outgoing = true + }, + internal_rpc { + verify_server_hostname = true + } +} retry_join = ["10.0.0.2"] bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}" @@ -505,22 +474,22 @@ If you don't use `10.0.0.0/8` as your private network, replace this part in the ### Step 3.3 - Enable systemd services for Consul and Nomad -To make the snapshot as small as possible, we will only enable the services, but won't start them yet. +To make the Snapshot as small as possible, we will only enable the services, but won't start them yet. ```bash server$ systemctl enable consul server$ systemctl enable nomad ``` -### Step 3.4 - Create the snapshot +### Step 3.4 - Create the Snapshot -Go to the Hetzner cloud web interface, stop the server and create a snapshot. This snapshot will be used for all clients, so choose a meaningful name. You can delete the cloud server after the snapshot creation was successful. +Go to the Hetzner Cloud Console, stop the server and create a Snapshot. This Snapshot will be used for all clients, so choose a meaningful name. You can delete the cloud server after the Snapshot creation was successful. -> If you want to update this image later, create a cloud server from the snapshot, but without any cloud network attached. This way both Consul and Nomad will fail and won't sync any data, resulting in a smaller image. +> If you want to update this image later, create a cloud server from the Snapshot, but without any Cloud Network attached. This way both Consul and Nomad will fail and won't sync any data, resulting in a smaller image. ## Step 4 - Adding clients -To add a client, start a cloud server from the snapshot created in step 3. Make sure to attach the cloud server to the private cloud network also used by the servers from step 2. Keep in mind that the cloud server name will be used in the cluster. +To add a client, start a cloud server from the Snapshot created in step 3. Make sure to attach the cloud server to the private Cloud Network also used by the servers from step 2. Keep in mind that the cloud server name will be used in the cluster. Startup may take up to a minute for the client to join the network. You can check the status on any node in the cluster with From 9c4b306e332b4e80c08ef71fc01fcb11c10c4d17 Mon Sep 17 00:00:00 2001 From: "svenja.michal" Date: Mon, 18 Nov 2024 10:12:33 +0100 Subject: [PATCH 2/2] Fix link --- tutorials/install-nomad-consul-cluster/01.en.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tutorials/install-nomad-consul-cluster/01.en.md b/tutorials/install-nomad-consul-cluster/01.en.md index 5cbba2357..10922c9f3 100644 --- a/tutorials/install-nomad-consul-cluster/01.en.md +++ b/tutorials/install-nomad-consul-cluster/01.en.md @@ -195,7 +195,7 @@ WantedBy=multi-user.target ### Step 1.5 - Create the base Snapshot -Finally, stop the server in the Hetzner [Cloud Console](https://console.hetzner.cloud/projects) and create a Snapshot. This Snapshot will be used as a foundation for the server and client setup of the cluster. +Finally, stop the server in the Hetzner [Cloud Console](https://console.hetzner.cloud/) and create a Snapshot. This Snapshot will be used as a foundation for the server and client setup of the cluster. After the Snapshot creation was successful, delete the CX22 instance of this step. @@ -210,7 +210,7 @@ We will use the following resources * 1 Hetzner Cloud Network * 3 Hetzner Cloud CX22 server -In the Hetzner [Cloud Console](https://console.hetzner.cloud/projects), create 3 CX22 servers from the Snapshot created in Step 1 and a common Cloud Network attached. This tutorial will use a `10.0.0.0/8` network, but smaller networks work as well. +In the Hetzner [Cloud Console](https://console.hetzner.cloud/), create 3 CX22 servers from the Snapshot created in Step 1 and a common Cloud Network attached. This tutorial will use a `10.0.0.0/8` network, but smaller networks work as well. In the following steps, the tutorial will refer to the servers by their internal IP addresses `10.0.0.2`, `10.0.0.3` and `10.0.0.4`. If your servers have different internal addresses, make sure to replace them in the following steps. @@ -403,7 +403,7 @@ We will use the following resources * 1 Hetzner Cloud CX22 server -In the Hetzner [Cloud Console](https://console.hetzner.cloud/projects), create 1 CX22 server from the Snapshot created in step 1. The cloud server name is not important in this step, since we will only prepare the image on it. +In the Hetzner [Cloud Console](https://console.hetzner.cloud/), create 1 CX22 server from the Snapshot created in step 1. The cloud server name is not important in this step, since we will only prepare the image on it. ### Step 3.1 - Configure Consul