From 8709ca2c78f171da3727bc6c409c356130ea40cf Mon Sep 17 00:00:00 2001 From: Paul Gottschling Date: Mon, 24 Jun 2024 11:29:12 -0400 Subject: [PATCH] Consolidate updating guides Backports #41160 * Consolidate updating guides Include a how-to guide for the fast path we want most users to move to: using cloud-hosted Teleport Enterprise and enrolling agents in automatic updates. For other use cases and edge cases, include a general Updating Reference. * Respond to hugoShaka feedback - Fix step numbering. - Add cloud prerequisite and mention cloud-hosted clusters in the introduciton. * Respond to bernardjkim feedback - Remove unintentionally preserved text about release channels for Teleport v14. - Add instructions for determining the Teleport version to install. * Add an intro paragraph with a recommendation Urge users to enroll their agents in automatic upgrades per feedback from bernardjkim. * Add check for the Enterprise teleport-kube-agent Incorporates #41132 * Respond to r0mant feedback - Include self-hosted instructions in the how-to guide - Include a journalctl command to check logs - Edit the upgrading menu page and intro of the how-to guide to provide stronger language re: using automatic agent upgrades in cloud accounts. * Add `helm repo update` step * Fix linter errors * Respond to r0mant feedback --- docs/config.json | 27 +- .../architecture/agent-update-management.mdx | 2 +- .../choose-an-edition/teleport-cloud/faq.mdx | 12 +- .../server-access/custom-installer.mdx | 2 +- docs/pages/upgrading.mdx | 48 +- .../upgrading/automatic-agent-updates.mdx | 373 ++++++++++++++++ docs/pages/upgrading/cloud-kubernetes.mdx | 199 --------- docs/pages/upgrading/cloud-linux.mdx | 282 ------------ docs/pages/upgrading/reference.mdx | 411 ++++++++++++++++++ .../self-hosted-automatic-agent-updates.mdx | 311 ------------- .../upgrading/self-hosted-kubernetes.mdx | 78 ---- docs/pages/upgrading/self-hosted-linux.mdx | 43 -- 12 files changed, 807 insertions(+), 981 deletions(-) create mode 100644 docs/pages/upgrading/automatic-agent-updates.mdx delete mode 100644 docs/pages/upgrading/cloud-kubernetes.mdx delete mode 100644 docs/pages/upgrading/cloud-linux.mdx create mode 100644 docs/pages/upgrading/reference.mdx delete mode 100644 docs/pages/upgrading/self-hosted-automatic-agent-updates.mdx delete mode 100644 docs/pages/upgrading/self-hosted-kubernetes.mdx delete mode 100644 docs/pages/upgrading/self-hosted-linux.mdx diff --git a/docs/config.json b/docs/config.json index e3aa6226a56b2..4ec570ee8f31a 100644 --- a/docs/config.json +++ b/docs/config.json @@ -29,29 +29,12 @@ "slug": "/upgrading/overview/" }, { - "title": "Teleport Cloud Agents (Linux)", - "slug": "/upgrading/cloud-linux/", - "forScopes": ["cloud"] - }, - { - "title": "Teleport Cloud Agents (Kubernetes)", - "slug": "/upgrading/cloud-kubernetes/", - "forScopes": ["cloud"] - }, - { - "title": "Self-Hosted Linux", - "slug": "/upgrading/self-hosted-linux/", - "forScopes": ["enterprise", "oss"] - }, - { - "title": "Self-Hosted Kubernetes", - "slug": "/upgrading/self-hosted-kubernetes/", - "forScopes": ["enterprise", "oss"] + "title": "Set up Automatic Agent Updates", + "slug": "/upgrading/automatic-agent-updates/" }, { - "title": "Self-Hosted Automatic Upgrades", - "slug": "/upgrading/self-hosted-automatic-agent-updates/", - "forScopes": ["enterprise"] + "title": "Updating Reference", + "slug": "/upgrading/reference/" } ] }, @@ -2685,7 +2668,7 @@ }, { "source": "/management/operations/self-hosted-automatic-agent-updates/", - "destination": "/upgrading/self-hosted-automatic-agent-updates/", + "destination": "/upgrading/", "permanent": true }, { diff --git a/docs/pages/architecture/agent-update-management.mdx b/docs/pages/architecture/agent-update-management.mdx index 8fab7425c70c4..84768a9741426 100644 --- a/docs/pages/architecture/agent-update-management.mdx +++ b/docs/pages/architecture/agent-update-management.mdx @@ -88,7 +88,7 @@ ensure every agent is healthy and running the correct version. ## Next steps Self-hosted users must first [set up self-hosted automatic agent upgrades -](../upgrading/self-hosted-automatic-agent-updates.mdx). +](../upgrading/automatic-agent-updates.mdx). After that, you can set enroll agents in automatic updates as part of the [upgrading procedure](../upgrading.mdx). diff --git a/docs/pages/choose-an-edition/teleport-cloud/faq.mdx b/docs/pages/choose-an-edition/teleport-cloud/faq.mdx index 616b18b641d14..5631ae188582d 100644 --- a/docs/pages/choose-an-edition/teleport-cloud/faq.mdx +++ b/docs/pages/choose-an-edition/teleport-cloud/faq.mdx @@ -190,14 +190,10 @@ Teleport Cloud version server for automatic agent updates. With automatic agent updates, agents periodically check the version server for new releases and update the software when new versions are found. -If you enroll in automatic agent updates, Teleport agents are automatically -updated after your Teleport cluster is updated during your scheduled maintenance -period. For more information, read one of the following guides: - -- [Upgrading Teleport Cloud Agents on - Kubernetes](../../upgrading/cloud-kubernetes.mdx) -- [Upgrading Teleport Cloud Agents on - Linux Servers](../../upgrading/cloud-linux.mdx) +If you enroll in automatic agent updates, Teleport agents are automatically +updated after your Teleport cluster is updated during your scheduled maintenance +period. For more information, read the [Automatic Agent +Updates](../../upgrading/automatic-agent-updates.mdx) guide. ### How can I find version information on connected agents? diff --git a/docs/pages/includes/server-access/custom-installer.mdx b/docs/pages/includes/server-access/custom-installer.mdx index c507239899424..bfd999c573448 100644 --- a/docs/pages/includes/server-access/custom-installer.mdx +++ b/docs/pages/includes/server-access/custom-installer.mdx @@ -71,7 +71,7 @@ connect to. Has format `/` e.g. stable/v(=teleport.major_version=). See [installation](../../installation.mdx#linux) for more details. - `{{ .AutomaticUpgrades }}`: indicates whether Automatic Updates are enabled or disabled. Its value is either `true` or `false`. See - [self-hosted automatic agent updates](../../upgrading/self-hosted-automatic-agent-updates.mdx) + [Automatic Agent Updates](../../upgrading/automatic-agent-updates.mdx) for more information. - `{{ .TeleportPackage }}`: the Teleport package to use. Its value is either `teleport-ent` or `teleport` depending on whether the diff --git a/docs/pages/upgrading.mdx b/docs/pages/upgrading.mdx index e14690dc5effe..fc73c1f70a270 100644 --- a/docs/pages/upgrading.mdx +++ b/docs/pages/upgrading.mdx @@ -1,47 +1,23 @@ --- title: Upgrading Teleport description: Explains how to upgrade Teleport depending on your environment and edition. -layout: tocless-doc --- The guides in this section show you how to upgrade Teleport to a more recent version. -## Upgrading compatibility overview +Read the [Upgrading Compatibility Overview](./upgrading/overview.mdx) to +understand how to upgrade components in your Teleport cluster while ensuring +compatibility between all components. -Since Teleport is a distributed system with a number of services that run on -potentially many hosts, you should take care when upgrading the cluster to -ensure that all components remain compatible. +If you have a Teleport Enterprise (Cloud) account, you **must** [set up automatic +Teleport agent updates](./upgrading/automatic-agent-updates.mdx) to ensure that +the version of Teleport running on agents is always compatible with that of the +Teleport cluster. You can also set up automatic agent upgrades in a self-hosted +cluster. -The [Upgrading Compatibility Overview](./upgrading/overview.mdx) explains how to -upgrade components in your Teleport cluster to ensure that they communicate as -expected. +For more information about upgrading, for example, to upgrade manually, read the +[Upgrading Reference](./upgrading/reference.mdx). -## Upgrading guides - -After getting familiar with the Compatibility Overview, read one of the -following guides to upgrade the components in your Teleport cluster. Choose the -guide that matches your Teleport edition and the platform where you run Teleport -services: - -Teleport Cloud: - -- [Linux Servers](./upgrading/cloud-linux.mdx) -- [Kubernetes](./upgrading/cloud-kubernetes.mdx) - -Self-hosted deployments: - -- [Linux Servers](./upgrading/self-hosted-linux.mdx) -- [Kubernetes](./upgrading/self-hosted-kubernetes.mdx) - -## Automatic agent upgrades - -We recommend that Teleport users set up automatic agent upgrades, where agents -query a **version server** in order to determine whether they are out of date -and need to install a new version of Teleport. - -On Teleport Cloud, the version server is managed for you. - -If you are running Teleport Enterprise, read the [Self-Hosted Automatic Agent -Updates](./upgrading/self-hosted-automatic-agent-updates.mdx) guide to set up -the version server yourself. +You can find more information regarding the automatic updates architecture in the +[Agent Update Management](./architecture/agent-update-management.mdx) page. diff --git a/docs/pages/upgrading/automatic-agent-updates.mdx b/docs/pages/upgrading/automatic-agent-updates.mdx new file mode 100644 index 0000000000000..b466572de30a4 --- /dev/null +++ b/docs/pages/upgrading/automatic-agent-updates.mdx @@ -0,0 +1,373 @@ +--- +title: Set up Automatic Agent Updates +description: Describes how to set up automatic agent updates. +--- + +On cloud-hosted Teleport Enterprise accounts, users must set up automatic agent +updates to ensure that the version of Teleport running on agents remains +compatible with the version running on the Auth Service and Proxy Service. If an +agent does not maintain [version compatibility](./overview.mdx) with your +Teleport cluster, connections to those agents will become degraded or lost. + +Cloud-hosted Teleport clusters are updated on a weekly basis. Major version +updates are performed every 4 months. You can monitor and subscribe to the +[Teleport Status](https://status.teleport.sh/) page to be notified of scheduled +updates. + +Teleport supports automatic agent updates for systemd-based Linux distributions +using `apt`, `yum`, and `zypper` package managers, as well as Kubernetes +clusters. + +This guide explains how to enable automatic updates for Teleport agents on +Teleport Enterprise clusters, including both self-hosted and cloud-hosted +clusters. + +## How it works + +When automatic updates are enabled, a Teleport updater is installed alongside +each Teleport agent. The updater communicates with the Teleport Proxy Service to +determine when an update is available. When an update is available, the updater +will update the Teleport agent during the next maintenance window. However, if a +critical update is available, the Teleport agent will be updated outside the +regular maintenance window. + +## Prerequisites + +- A Teleport Enterprise cluster. If you do not have one, [sign + up](https://goteleport.com/signup) for a free trial or consult the [Update + Reference](./reference.mdx) to read about manual updates. +- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide, + which describes the sequence in which to upgrade components in your cluster. +- Teleport agents that are not yet enrolled in automatic updates. +- (!docs/pages/includes/tctl-tsh-prerequisite.mdx!) +- (!docs/pages/includes/tctl.mdx!) + +## Step 1/4. Enable automatic agent upgrades + +If you are running a cloud-hosted Teleport Enterprise cluster, skip to [Step +2](#step-24-find-agents-to-enroll-in-automatic-updates). + +Before enabling automatic upgrades in your self-hosted Teleport cluster, you +must enable a **version server**. This section shows you how to enable a version +server in your cluster. Automatic upgrades in self-hosted Teleport clusters +require at least v14.3.7 or v15.1.3. + +### Configure a maintenance schedule + +To enable automatic upgrades in your cluster, you must create a cluster +maintenance configuration. This configures a maintenance schedule for the +Teleport cluster that agents use to determine when to check for upgrades. + +1. Create a Teleport role that can manage cluster maintenance configurations + through the `cluster_maintenance_config` dynamic resource. No preset Teleport + roles provide this ability, so you will need to create one. + + Create a file called `cmc-editor.yaml` with the following content: + + ```yaml + kind: role + version: v7 + metadata: + name: cmc-editor + spec: + allow: + rules: + - resources: ['cluster_maintenance_config'] + verbs: ['create', 'read', 'update', 'delete'] + ``` + + Create the role resource: + + ```code + $ tctl create cmd-editor.yaml + ``` + +1. Add the role to your Teleport user: + + (!docs/pages/includes/add-role-to-user.mdx!) + +1. Create a cluster maintenance config in a file called `cmc.yaml`. The + following example allows maintenance on Monday, Wednesday and Friday between + 02:00 and 03:00 UTC: + + (!docs/pages/includes/cluster-maintenance-config-spec.mdx!) + +1. Apply the manifest using `tctl`: + + ```code + $ tctl create cmc.yaml + maintenance window has been updated + ``` + +### [Optional] Assign the version served by the version server + +By default, the version server has a single `default` channel, serving the +version of the Teleport Proxy Service. If you want to override the default +version or add other channels you can use the `automatic_upgrades_channels` +field in the Proxy Service configuration file: + +```yaml +proxy_service: + enabled: "yes" + automatic_upgrades_channels: + # Override the default version channel reachable at + # https:///v1/webapi/automaticupgrades/channel/default/version + default: + static_version: v14.2.1 + # Define a new version channel with a static version reachable at + # https:///v1/webapi/automaticupgrades/channel/m-static-channel/version + my-static-channel: + static_version: v14.2.0 + # Define a new version channel forwarding requests to an upstream version server + my-remote-channel: + forward_url: https://updates.releases.teleport.dev/v1/stable/cloud +``` + +You must ensure that all Proxy Service instances share the same +`automatic_upgrades_channels` configuration. If some Proxy Service instances are +configured differently, you will experience agents flickering between versions +as the version served is not consistent across instances. + +If your Proxy Service public address is , +you can query the version server with the following command: + +```code +$ curl "https:///v1/webapi/automaticupgrades/channel/default/version" +(=teleport.version=) +``` + +## Step 2/4. Find agents to enroll in automatic updates + +Use the `tctl inventory ls` command to list connected agents along with their current +version. Use the `--upgrader=none` flag to list agents that are not enrolled in automatic +updates. + +```code +$ tctl inventory ls --upgrader=none +Server ID Hostname Services Version Upgrader +------------------------------------ ------------- -------- ------- -------- +00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node v14.4.5 none +... +``` + +Note that the `inventory ls` command will also list `teleport-auth` and +`teleport-proxy` services. If you use managed Teleport Enterprise, the Teleport +team updates these services automatically. + +## Step 3/4. Enroll agents on Linux servers in automatic updates + +1. For each agent ID returned by the `tctl inventory ls` command, copy the ID + and run the following `tctl` command to access the host via `tsh`: + + ```code + $ HOST=00000000-0000-0000-0000-000000000000 + $ USER=root + $ tsh ssh "${USER?}@${HOST?}" + ``` + +1. Determine the Teleport version to install by querying the Teleport Proxy + Service. This way, the Teleport installation has the same major version as + the automatic updater. + + Replace with the domain name of the + Teleport Proxy Service and with the name of your + automatic update channel. For cloud-hosted Teleport Enterprise accounts, this + is always `stable/cloud`: + + ```code + $ TELEPORT_VERSION="$(curl https:///v1/webapi/automaticupgrades/channel//version | sed 's/v//')" + ``` + +1. Ensure that the Teleport repository is properly configured to use the channel, and install the `teleport-ent-updater` + package. You must install `teleport-ent-updater` on each agent you would like + to enroll into automatic updates: + + + + + ```code + $ curl https://goteleport.com/static/install.sh | bash -s ${TELEPORT_VERSION?} cloud + ``` + + + + + ```code + $ curl https://goteleport.com/static/install.sh | bash -s ${TELEPORT_VERSION?} enterprise + ``` + + + + + The installation script detects the package manager on your Linux server and + uses it to install Teleport binaries. To customize your installation, learn + about the Teleport package repositories in the [installation + guide](../installation.mdx#linux). + +1. Confirm that the version of the `teleport` binary is the one you expect: + + ```code + $ teleport version + ``` + +
+ +If you changed the agent user to run as non-root, create +`/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user: + +```code +$ sudo mkdir -p /etc/teleport-upgrade.d/ +$ sudo touch /etc/teleport-upgrade.d/schedule +$ sudo chown /etc/teleport-upgrade.d/schedule +``` + +
+ +1. Verify that the upgrader can see your version endpoint by checking for + upgrades: + + ```code + $ sudo teleport-upgrade dry-run + ``` + +1. You should see one of the following messages, depending on the target version + you are currently serving: + + ```text + no upgrades available (1.2.3 == 1.2.3) + an upgrade is available (1.2.3 -> 2.3.4) + ``` + + `teleport-upgrade` may display warnings about not having a valid upgrade + schedule. This is expected immediately after install as the maintenance + schedule might not be exported yet. + +## Step 4/4. Enroll Kubernetes agents in automatic updates + +This section assumes that the name of your `teleport-kube-agent` release is +`teleport-agent`, and that you have installed it in the `teleport` namespace. + +1. Confirm that you are using the Teleport Enterprise edition of the + `teleport-kube-agent` chart. You should see the following when you query your + `teleport-kube-agent` release: + + ```code + $ helm -n `teleport` get values `teleport-agent` -o json | jq '.enterprise' + true + ``` + + If another value such as `null` is returned, update your existing agent + `values.yaml` to use the Enterprise version: + + ```yaml + enterprise: true + ``` + +1. Add the following chart values to the values file for the + `teleport-kube-agent` chart: + + ```yaml + updater: + enabled: true + ``` + +1. Update the Teleport Helm repository to include any new versions of the + `teleport-kube-agent` chart: + + ```code + $ helm repo update teleport + ``` + +1. Update the Helm chart release with the new values: + + + + + ```code + $ helm -n upgrade teleport/teleport-kube-agent \ + --values=values.yaml \ + --version= + ``` + + + + ```code + $ helm -n upgrade teleport/teleport-kube-agent \ + --values=values.yaml \ + --version= + ``` + + + +1. You can validate the updater is running properly by checking if its pod is + ready: + + ```code + $ kubectl -n teleport-agent get pods + NAME READY STATUS RESTARTS AGE + -0 1/1 Running 0 14m + -1 1/1 Running 0 14m + -2 1/1 Running 0 14m + -updater-d9f97f5dd-v57g9 1/1 Running 0 16m + ``` + +1. Check for any deployment issues by checking the updater logs: + + ```code + $ kubectl -n logs deployment/-updater + 2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"} + ``` + +## Troubleshooting + +Teleport agents are not updated immediately when a new version of Teleport is +released, and agent updates can lag behind the cluster by a few days. + +If the Teleport agent has not been automatically updating for several weeks, you +can consult the updater logs to help troubleshoot the problem: + +```code +$ journalctl -u teleport-upgrade +``` + +### Troubleshooting automatic agent upgrades on Kubernetes + +The updater is a controller that periodically reconciles expected Kubernetes +resources with those in the cluster. The updater executes a reconciliation loop +every 30 minutes or in response to a Kubernetes event. If you don't want to wait +until the next reconciliation, you can trigger an event. + +1. Any deployment update will send an event, so you can trigger the upgrader by + annotating the resource: + + ```code + $ kubectl -n annotate statefulset/ 'debug.teleport.dev/trigger-event=1' + ``` + +1. To suspend automatic updates for an agent, annotate the agent deployment + with `teleport.dev/skipreconcile: "true"`, either by setting the + `annotations.deployment` value in Helm, or by patching the deployment + directly with `kubectl`. + +### Troubleshooting automatic agent upgrades on Linux + +1. If an agent is not automatically upgraded, you can invoke the upgrader + manually and look at its logs: + + ```code + $ sudo teleport-upgrade run + ``` + +1. To suspend automatic updates, disable the systemd timer: + + ```code + $ sudo systemctl disable --now teleport-upgrade.timer + ``` + +1. To enable and start the systemd timer after suspending: + + ```code + $ sudo systemctl enable --now teleport-upgrade.timer + ``` + diff --git a/docs/pages/upgrading/cloud-kubernetes.mdx b/docs/pages/upgrading/cloud-kubernetes.mdx deleted file mode 100644 index 9060c6c278d25..0000000000000 --- a/docs/pages/upgrading/cloud-kubernetes.mdx +++ /dev/null @@ -1,199 +0,0 @@ ---- -title: Upgrade Teleport Cloud Agents on Kubernetes -description: Provides instructions for upgrading Teleport Cloud agents that run on Kubernetes. ---- - -This guide explains how to upgrade Teleport Cloud agents running on Kubernetes. -On Teleport Cloud, Auth Service and Proxy Service upgrades are managed for you. -To keep agents up to date, you can either enroll them in automatic updates or -upgrade them manually using the method you used to install Teleport. - -## Prerequisites - -- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide, - which describes the sequence in which to upgrade components of your cluster. - -- Teleport Enterprise Cloud. You can determine the current version of these - services by running the following command, where `mytenant` is the name of - your Teleport Enterprise Cloud tenant. This requires the - [`jq` CLI tool](https://jqlang.github.io/jq/): - - ```code - $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.server_version' - ``` - -- (!docs/pages/includes/cloud/tctl-tsh-prerequisite.mdx!) - -- (!docs/pages/includes/tctl.mdx!) - -Teleport supports automatic agent updates for the `teleport-kube-agent` Helm -chart. The [Automatic Update Architecture -](../architecture/agent-update-management.mdx) guide describes how agent -updating works. Automatic agent upgrades require: - -- A Teleport Cloud account that supports automatic updates. To determine if - your account supports automatic updates, run the following command, replacing - `mytenant.teleport.sh` with the address of your Teleport Cloud account: - - ```code - $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.automatic_upgrades' - true - ``` -- At least one Teleport Enterprise agent deployed via the `teleport-kube-agent` - Helm chart. - -## Step 1/2. Determine whether any agents require automatic updates - -To determine if any agents in your cluster are not configured for automatic -upgrades, run the following command. This prints a list of all Teleport services -with no upgrader, filtering these to ignore the Auth Service and Proxy Service -(which are maintained for you in Teleport Cloud): - -```code -$ tctl inventory ls \ - --services=node,kube,db,app,windows_desktop,discovery,okta \ - --upgrader=none -Server ID Hostname Services Version Upgrader ------------------------------------- ------------------------------- -------- ------- -------- -00000000-0000-0000-0000-000000000000 teleport-agent-0 Kube v13.4.3 none -``` - -If the list is nonempty, proceed to the next step to enable automatic updates. - -## Step 2/2. Enroll agents in automatic updates - -This section assumes that the name of your `teleport-kube-agent` release is -`teleport-agent`, and that you have installed it in the `teleport` namespace. - -1. Confirm you are using the Teleport Enterprise edition of the - `teleport-kube-agent` chart. You should see the following when you query your - `teleport-kube-agent` release: - - ```code - $ helm -n get values -o json | jq '.enterprise' - true - ``` - -1. Add the following chart values to your existing agent `values.yaml` to enable - the automatic updater: - - ```yaml - updater: - enabled: true - ``` - -1. Upgrade the Helm chart release with the new values by running `helm upgrade`. - The command should resemble the following: - - ```code - $ helm -n upgrade teleport/teleport-kube-agent \ - --values=values.yaml \ - --version= - ``` - -1. Validate the updater is running properly by checking if its pod is ready: - - ```code - $ kubectl -n get pods - NAME READY STATUS RESTARTS AGE - my-agent-0 1/1 Running 0 14m - my-agent-1 1/1 Running 0 14m - my-agent-2 1/1 Running 0 14m - my-agent-updater-d9f97f5dd-v57g9 1/1 Running 0 16m - ``` - - And by consulting its logs: - - ```code - $ kubectl -n logs deployment/-updater - 2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"} - ``` - -The updater is a controller that periodically reconciles expected Kubernetes -resources with those in the cluster. The updater executes a reconciliation loop -every 30 minutes or in response to a Kubernetes event. If you don't want to wait -until the next reconciliation, you can trigger an event. Any deployment update -will send an event, so the updater can be triggered by annotating the resource: - -```code -$ kubectl -n annotate statefulset/ 'debug.teleport.dev/trigger-event=1' -``` - -To suspend automatic updates for an agent, annotate the agent deployment with -`teleport.dev/skipreconcile: "true"`, either by setting the -`annotations.deployment` value in Helm, or by patching the deployment directly -with `kubectl`. - -## GitOps - -Automatic updates are incompatible with some GitOps tools used for continuous deployment. -The `teleport-kube-agent` Helm chart owns the version of the `teleport-agent` resource, -so when the `teleport-agent-updater` modifies the image version of the `teleport-agent` -resource, the GitOps tool will detect a drift or a diff in the `teleport-agent` resource. - -### ArgoCD deployments - -After an automatic update, ArgoCD reports the `teleport-agent` resource as `OutOfSync`. -As a workaround to this problem use a [Diff Customization](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#diffing-customization) -to ignore the difference in image version. Here is an example deployment using the -name `teleport-agent` and namespace `teleport`. - -```yaml -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: teleport-agent - namespace: teleport -spec - ignoreDifferences: - - group: apps - kind: StatefulSet - name: teleport-agent - namespace: teleport - jqPathExpressions: - - .spec.template.spec.containers[] | select(.name == "teleport").image -... -``` - -### FluxCD deployments - -After an automatic update, FluxCD reports a `DriftDetected` event. As a workaround -to this problem modify the [drift detection](https://fluxcd.io/flux/components/helm/helmreleases/#drift-detection) -configuration to ignore the difference in image version. Here is an example deployment -using the name `teleport-agent` and namespace `teleport`. - -```yaml -apiVersion: helm.toolkit.fluxcd.io/v2beta2 -kind: HelmRelease -metadata: - name: teleport-agent - namespace: teleport -spec - driftDetection: - mode: enabled - ignore: - - paths: ["/spec/template/spec/containers/0/image"] - target: - kind: StatefulSet - name: teleport-agent - namespace: teleport -... -``` - -## Manually upgrading agents - -Run the following commands to upgrade Teleport agents running on Kubernetes. - -1. Update the Teleport Helm chart repository so you can install the latest - version of the `teleport-kube-agent` chart: - - (!docs/pages/kubernetes-access/helm/includes/helm-repo-add.mdx!) - -1. Upgrade the Helm release: - - ```code - $ helm -n upgrade teleport/teleport-kube-agent \ - --values=values.yaml \ - --version= - ``` - diff --git a/docs/pages/upgrading/cloud-linux.mdx b/docs/pages/upgrading/cloud-linux.mdx deleted file mode 100644 index 5acf6e5f080c2..0000000000000 --- a/docs/pages/upgrading/cloud-linux.mdx +++ /dev/null @@ -1,282 +0,0 @@ ---- -title: Upgrade Teleport Cloud Agents on Linux -description: Provides instructions for upgrading Teleport Cloud agents that run on Linux servers. ---- - -This guide explains how to upgrade Teleport Cloud agents running on Linux -servers. On Teleport Cloud, Auth Service and Proxy Service upgrades are managed -for you. To keep agents up to date, you can either enroll them in automatic -upgrades or upgrade them manually using the method you used to install Teleport. - -## Prerequisites - -- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide, - which describes the sequence in which to upgrade components in your cluster. - -- A Teleport Enterprise Cloud account. You can determine the current version of - the Auth Service and Proxy Service by running the following command, where - `mytenant` is the name of your Teleport Enterprise Cloud account. This - requires the [`jq` CLI tool](https://jqlang.github.io/jq/): - - ```code - $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.server_version' - ``` - -- (!docs/pages/includes/cloud/tctl-tsh-prerequisite.mdx!) - -- (!docs/pages/includes/tctl.mdx!) - -Teleport supports automatic agent updates for systemd-based Linux distributions -using `apt`, `yum`, and `zypper` package managers. The [Automatic Update -Architecture](../architecture/agent-update-management.mdx) guide describes -how agent updating works. Automatic agent upgrades require: - -- A Teleport Cloud account that supports automatic updates. To determine if - your account supports automatic updates, run the following command, replacing - `mytenant.teleport.sh` with the address of your Teleport Cloud account: - - ```code - $ curl -s https://mytenant.teleport.sh/webapi/ping | jq '.automatic_upgrades' - true - ``` - -- At least one Teleport Enterprise agent, started via systemd on a distribution - using the `apt` or `yum` package managers. - - So you can access each agent, we assume that each agent runs the Teleport SSH - Service in addition to any other services. See the [Deploy Teleport Agents - with Terraform](../agents/deploy-agents-terraform.mdx) guide for an example. - -## Step 1/2. Find agents to upgrade - -To list agents that are not enrolled in automatic updates, use the `tctl -inventory ls` command with the `--upgrader=none` flag. This command also uses -the `--services` flag to constrain the search to Teleport processes running -agent services: - -```code -$ tctl inventory ls \ - --services=node,kube,db,app,windows_desktop,discovery,okta \ - --upgrader=none -Server ID Hostname Services Version Upgrader ------------------------------------- ------------- -------- ------- -------- -00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node v13.4.5 none -``` - -If you have a lot of agents on different versions and want to prioritize enrolling -your oldest agents, you can limit your search using the `--older-than` filter: - -```code -$ tctl inventory ls --upgrader=none --older-than=v1.2.3 -Server ID Hostname Services Version Upgrader ------------------------------------- --------------- -------------- ------- -------- -00000000-0000-0000-0000-000000000000 old.example.com Node v1.1.1 none -00000000-0000-0000-0000-000000000001 teleport-proxy Proxy v14.3.6 none -00000000-0000-0000-0000-000000000002 teleport-auth Auth,Discovery v14.3.6 none -... -``` - -Note that the `inventory ls` command will also list `teleport-auth` and `teleport-proxy` services. -You do not need to update these services or enroll them in automatic updates. -Teleport Cloud manages updates for these services. - -When you enroll each agent in automatic updates in the next section, you can -run the following commands to fetch the hostname for each agent and access it -via Teleport. - -For each agent ID returned by the `tctl inventory ls` command, copy the ID and -run the following `tctl` command to access the host via `tsh`: - -```code -$ HOST=00000000-0000-0000-0000-000000000000 -$ USER=root -$ tsh ssh "${USER?}@${HOST?}" -``` - -You can then follow the steps in the next section on each agent to enroll it in -automatic updates. - -## Step 2/2. Enroll agents in automatic updates - -Complete the following instructions on each agent you would like to enroll into -automatic updates: - -1. Ensure the Teleport repository is added and Teleport Enterprise is installed. - - To verify if the Teleport repository was added to the system, check if either of - the follow files exists: - - ```code - $ ls /etc/apt/sources.list.d/teleport.list - # or - $ ls /etc/yum.repos.d/teleport.repo - ``` - - The upgrader checks the repository for available releases, so make sure that - it is up to date. - -1. If the repository was added, make sure the Teleport binary installed on the - agent can run the automatic updater: - - ```code - $ which teleport-upgrade || echo "Install the upgrader" - Install the upgrader - ``` - -1. If the Teleport repository is not found, or the Teleport binary you installed - does not include the upgrader, add the appropriate repository and reinstall - Teleport: - - (!docs/pages/includes/cloud/install-linux-cloud.mdx!) - -1. If you changed the agent user to run as non-root, create - `/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user. - Otherwise, you can skip this step: - - ```code - $ sudo mkdir -p /etc/teleport-upgrade.d/ - $ sudo touch /etc/teleport-upgrade.d/schedule - $ sudo chown /etc/teleport-upgrade.d/schedule - ``` - -1. Verify that the upgrader can see your version endpoint by checking for - updates: - - ```code - $ sudo teleport-upgrade dry-run - ``` - - You should see one of the following messages, depending on the target version - you are currently serving: - - ```text - no upgrades available (1.2.3 == 1.2.3) - an upgrade is available (1.2.3 -> 2.3.4) - ``` - - The upgrader may display warnings about not having a valid upgrade schedule. - This is expected immediately after install as the maintenance schedule might - not be exported yet. - -1. Ensure that the upgrader is running: - - ```code - $ sudo systemctl status teleport-upgrade - ``` - -## Version locking - -As of Teleport `15.1.10`, a version locking mechanism is enabled by the updater. -This mechanism locks the version of Teleport and prevents manual updates of the Teleport -package. This prevents unintentional updates during routine system maintenance, or -an accidental update by a user. The updater still has the capability to update the -Teleport package, and all updates of Teleport are expected to be performed by the -updater. - -The version locking mechanism is implemented using the features of the package managers. -In case a user would like to manually update the Teleport version, this can be done -with the following commands. - -With the APT package manager CLI, the `--allow-change-held-packages` flag must be provided -to bypass the version lock. -```code -$ apt-get install --allow-change-held-packages "teleport-ent=" -``` - -With the YUM package manager CLI, the `--disableexcludes="teleport"` flag must be provided -to bypass the version lock. -```code -$ yum install --disablerepo="*" --enablerepo="teleport" --disableexcludes="teleport" "teleport-ent-" -``` - -With the Zypper package manager CLI, the lock must be disabled and then re-enabled -after the update. -```code -$ zypper removelock "teleport-ent" -$ zypper install --repo="teleport" "teleport-ent-" -$ zypper addlock "teleport-ent" -``` - -## Updating the updater - -The updater is designed to be minimal and relatively stable, but the updater will -receive updates on occasion. Currently, the updater does not have the capability -to update itself. Updates to the `teleport-ent-updater` package must be done manually. - -The `teleport-ent-updater` will be backwards compatible with older versions of Teleport, -so you can always update the `teleport-ent-updater` package to the latest available -version. - -## Troubleshooting - -The `teleport-upgrade` tool provides some basic commands to verify and perform an -update of the Teleport agent. - -```code -$ teleport-upgrade help -USAGE: /usr/sbin/teleport-upgrade - -Tool for automatic upgrades of Teleport agents. - -Commands: - run check for and potentially apply a teleport upgrade. - dry-run check for new teleport version but do not upgrade. - version print the current version of /usr/sbin/teleport-upgrade. - help show this help text. -``` - -The `dry-run` command can be used to check for an available update without performing -an update. -```code -# Example output when teleport is already on the latest compatible version. -$ teleport-upgrade dry-run -[i] no upgrades available (14.3.14 == 14.3.14) [ 582 ] - -# Example output when an update is available. -$ teleport-upgrade dry-run -[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] -[i] within maintenance window, upgrade will be attempted. [ 596 ] -``` - -The `run` command performs an update if available. -```code -# Successful teleport update from 13.4.14 to 14.3.14. -$ teleport-upgrade run -[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] -[i] within maintenance window, upgrade will be attempted. [ 596 ] -[i] attempting apt install teleport-ent=14.3.14... [ 480 ] -[...] -[i] gracefully restarting Teleport (if already running) [ 449 ] - -# Teleport updates are not attempted when outside the maintenance window. -$ teleport-upgrade run -[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] -[i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ] -``` - -Teleport agents are not updated immediately when a new version of Teleport is released, -and agent updates can lag behind the cluster by a few days. - -If the Teleport agent has not been automatically updating for several weeks, you -can view the logs of the `teleport-upgrade.service` to help troubleshoot the problem. - -```code -$ journalctl -t teleport-upgrade -Apr 16 16:44:32 example teleport-upgrade[168983]: [i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ] -Apr 16 16:50:32 example teleport-upgrade[169084]: [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] -Apr 16 16:50:33 example teleport-upgrade[169084]: [i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ] -Apr 16 16:56:32 example teleport-upgrade[169187]: [i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] -... -``` - -To suspend automatic updates, disable the `teleport.upgrade.timer` systemd timer: - -```code -$ sudo systemctl disable --now teleport-upgrade.timer -``` - -To enable and start the systemd timer after suspending: - -```code -$ sudo systemctl enable --now teleport-upgrade.timer -``` diff --git a/docs/pages/upgrading/reference.mdx b/docs/pages/upgrading/reference.mdx new file mode 100644 index 0000000000000..8992492c2506c --- /dev/null +++ b/docs/pages/upgrading/reference.mdx @@ -0,0 +1,411 @@ +--- +title: Upgrading Reference +description: Provides detailed information on upgrading Teleport in various situations. +tocDepth: 3 +--- + +This guide explains how to upgrade components of a Teleport cluster in +non-standard situations. + +To ensure that your Teleport cluster remains up to date with the lowest amount +of manual overhead, we recommend [signing up](https://goteleport.com/signup) for +a cloud-hosted Teleport Enterprise account and following [Enroll Agents in +Automatic Upgrades](./automatic-agent-updates.mdx). + +If your infrastructure does not support automatic agent updates, follow this +guide to determine the best approach for keeping your Teleport cluster up to +date. + +Before reading this guide, become familiar with the [Upgrading Compatibility +Overview](./overview.mdx) guide, which describes the sequence in which to +upgrade components in your cluster. + +## Working with the automatic updater + +This section explains how to manage the automatic updater. + +On Kubernetes deployments, the updater is a controller that periodically +reconciles expected Kubernetes resources with those in the cluster. On a Linux +server, the updater is an executable script called `teleport-upgrade`. + +### `teleport-upgrade` commands + +The `teleport-upgrade` tool provides some basic commands to verify and perform an +update of the Teleport agent. + +```code +$ teleport-upgrade help +USAGE: /usr/sbin/teleport-upgrade + +Tool for automatic upgrades of Teleport agents. + +Commands: + run check for and potentially apply a teleport upgrade. + dry-run check for new teleport version but do not upgrade. + force performs an upgrade if an upgrade is available. + version print the current version of /usr/sbin/teleport-upgrade. + help show this help text. +``` + +The `dry-run` command can be used to check for an available update without performing +an update. +```code +# Example output when teleport is already on the latest compatible version. +$ teleport-upgrade dry-run +[i] no upgrades available (14.3.14 == 14.3.14) [ 582 ] + +# Example output when an update is available. +$ teleport-upgrade dry-run +[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] +[i] within maintenance window, upgrade will be attempted. [ 596 ] +``` + +The `run` command performs an update if available. +```code +# Successful teleport update from 13.4.14 to 14.3.14. +$ teleport-upgrade run +[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] +[i] within maintenance window, upgrade will be attempted. [ 596 ] +[i] attempting apt install teleport-ent=14.3.14... [ 480 ] +[...] +[i] gracefully restarting Teleport (if already running) [ 449 ] + +# Teleport updates are not attempted when outside the maintenance window. +$ teleport-upgrade run +[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] +[i] upgrade is non-critical and we are outside of maintenance window, not attempting. [ 618 ] +``` + +The `force` command performs an update immediately even when outside the maintenance +window. +```code +$ teleport-upgrade force +[i] an upgrade is available (13.4.14 -> 14.3.14) [ 585 ] +[i] attempting apt install teleport-ent=14.3.14... [ 480 ] +[...] +[i] gracefully restarting Teleport (if already running) [ 449 ] +``` + +### Configuring the `teleport-upgrade` tool + +1. Create the upgrade configuration directory: + + ```code + $ sudo mkdir -p /etc/teleport-upgrade.d/ + ``` + +1. If you changed the agent user to run as non-root, create + `/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user. + Otherwise, you can skip this step: + + ```code + $ sudo touch /etc/teleport-upgrade.d/schedule + $ sudo chown /etc/teleport-upgrade.d/schedule + ``` + +1. Configure the upgrader to connect to your version server and subscribe to the + right release channel: + + ```code + $ echo "/v1/webapi/automaticupgrades/channel/default" | sudo tee /etc/teleport-upgrade.d/endpoint + ``` + + Make sure not to include `https://` as a prefix to the server address, nor + suffix the endpoint with `/version`. + +### Choosing a release channel + +When [configuring the updater](#configuring-the-teleport-upgrade-tool), you can +select a release channel. + +The following channels are available for APT, YUM, and Zypper repos: + +| Channel name | Description | +|-------------------|--------------------------------------------------------------------------------------------| +| `stable/` | Receives releases for the specified major release line, i.e. `v(=teleport.major_version=)` | +| `stable/cloud` | Rolling channel that receives releases compatible with current Cloud version | +| `stable/rolling` | Rolling channel that receives all published Teleport releases | +### Updating the updater + +The updater is designed to be minimal and relatively stable, but the updater will +receive updates on occasion. Currently, the updater does not have the capability +to update itself. Updates to the `teleport-ent-updater` package must be done manually. + +The `teleport-ent-updater` will be backwards compatible with older versions of Teleport, +so you can always update the `teleport-ent-updater` package to the latest available +version. + +### Version locking + +As of Teleport `15.1.10`, a version locking mechanism is enabled by the updater. +This mechanism locks the version of Teleport and prevents manual updates of the Teleport +package. This prevents unintentional updates during routine system maintenance, or +an accidental update by a user. The updater still has the capability to update the +Teleport package, and all updates of Teleport are expected to be performed by the +updater. + +The version locking mechanism is implemented using the features of the package managers. +In case a user would like to manually update the Teleport version, this can be done +with the following commands. + +With the `apt` package manager CLI, the `--allow-change-held-packages` flag must be provided +to bypass the version lock. +```code +$ apt-get install --allow-change-held-packages "teleport-ent=" +``` + +With the `yum` package manager CLI, the `--disableexcludes="teleport"` flag must be provided +to bypass the version lock. +```code +$ yum install --disablerepo="*" --enablerepo="teleport" --disableexcludes="teleport" "teleport-ent-" +``` + +With the `zypper` package manager CLI, the lock must be disabled and then re-enabled +after the update. +```code +$ zypper removelock "teleport-ent" +$ zypper install --repo="teleport" "teleport-ent-" +$ zypper addlock "teleport-ent" +``` + +## Automatic update limitations + +Automatic updates are not available in all Teleport editions and installation +methods. If you cannot use automatic updates, read [Manual +updates](#manual-updates) for possible steps. + +### Automatic updates with Teleport Community Edition + +Automatic updates is not currently supported with the community editions of Teleport. +Ensure that you are using the Teleport Enterprise edition of the `teleport-kube-agent` +chart. You should see the following when you query your `teleport-kube-agent` release: + +```code +$ helm -n get values -o json | jq '.enterprise' +true +``` + +### Automatic updates with direct binary installation + +Automatic updates is not currently supported with the direct binary installation method. +Automatic updates is only supported with installations via the `apt`, `yum`, and +`zypper` package managers. + +## Automatic updates with GitOps tools + +Automatic updates are incompatible with some GitOps tools used for continuous +deployment. The `teleport-kube-agent` Helm chart owns the version of the +`teleport-agent` resource, so when the `teleport-agent-updater` modifies the +image version of the `teleport-agent` resource, the GitOps tool will detect a +drift or a diff in the `teleport-agent` resource. + +### ArgoCD deployments + +After an automatic update, ArgoCD reports the `teleport-agent` resource as `OutOfSync`. +As a workaround to this problem use a [Diff Customization](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#diffing-customization) +to ignore the difference in image version. Here is an example deployment using the +name `teleport-agent` and namespace `teleport`. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: teleport-agent + namespace: teleport +spec + ignoreDifferences: + - group: apps + kind: StatefulSet + name: teleport-agent + namespace: teleport + jqPathExpressions: + - .spec.template.spec.containers[] | select(.name == "teleport").image +... +``` + +### FluxCD deployments + +After an automatic update, FluxCD reports a `DriftDetected` event. As a workaround +to this problem modify the [drift detection](https://fluxcd.io/flux/components/helm/helmreleases/#drift-detection) +configuration to ignore the difference in image version. Here is an example deployment +using the name `teleport-agent` and namespace `teleport`. + +```yaml +apiVersion: helm.toolkit.fluxcd.io/v2beta2 +kind: HelmRelease +metadata: + name: teleport-agent + namespace: teleport +spec + driftDetection: + mode: enabled + ignore: + - paths: ["/spec/template/spec/containers/0/image"] + target: + kind: StatefulSet + name: teleport-agent + namespace: teleport +... +``` + +## Manual updates + +This section shows you how to upgrade Teleport manually. You can perform manual +upgrades on Teleport Auth Service and Proxy Service instances running in +self-hosted clusters, as well as all Teleport agents. + +### Teleport agents + +1. Identify the latest compatible Teleport agent version by querying the + `webapi` endpoint of the Teleport Proxy Service: + + ```code + $ curl https:///webapi/automaticupgrades/channel/stable/cloud/version + v15.2.1 + ``` + +1. Use the `tctl inventory ls` command to list connected agents along with their + current version. Use the `--older-than` flag to list agents that are + upgradable: + + ```code + $ tctl inventory ls --older-than=v15.2.1 + Server ID Hostname Services Version Upgrader + ------------------------------------ -------------- -------------- ------- -------- + 00000000-0000-0000-0000-000000000000 ip-10-1-6-130 Node v14.4.5 none + 00000000-0000-0000-0000-000000000001 teleport-proxy Proxy v15.2.0 none + 00000000-0000-0000-0000-000000000002 teleport-auth Auth,Discovery v15.2.0 none + ... + ``` + + Note that the `inventory ls` command will also list `teleport-auth` and + `teleport-proxy` services. These services are managed by the Teleport team, and + updates will be performed automatically. + +1. For each agent ID returned by the `tctl inventory ls` command, copy the ID + and run the following `tctl` command to access the host via `tsh`: + + ```code + $ HOST=00000000-0000-0000-0000-000000000000 + $ USER=root + $ tsh ssh "${USER?}@${HOST?}" + ``` + +1. On each Linux server, follow the instructions in the [next + section](#single-teleport-binaries-on-linux-servers) to install the new + version of the `teleport` binary. + +1. If you have deployed any agents on Kubernetes using the `teleport-kube-agent` + Helm chart, [follow the instructions](#teleport-agents-running-on-kubernetes) + to upgrade the Helm release. + +### Single Teleport binaries on Linux servers + +You can upgrade a single Teleport binary running on a Linux host by running the +one-line installation script with a higher version than the current one. + +Before upgrading Teleport across a self-hosted cluster, read the [Compatibility +Overview](./overview.mdx) to ensure you are upgrading components in +the correct order. + +Complete the following steps on all servers that run the Auth Service and Proxy +Service, then on each of your agents: + +1. Get the current version: + + ```code + $ teleport version + ``` + +1. Assign to one of the following, depending on your + Teleport edition: + + | Edition | Value | + |-----------------------------------|--------------| + | Teleport Enterprise (Cloud) | `cloud` | + | Teleport Enterprise (Self-Hosted) | `enterprise` | + | Teleport Community Edition | `oss` | + +1. Assign to the version you want to install. + +1. Install the new Teleport version on your Linux server: + + ```code + $ curl https://goteleport.com/static/install.sh | bash -s + ``` + + The installation script detects the package manager on your Linux server and + uses it to install Teleport binaries. To customize your installation, learn + about the Teleport package repositories in the [installation + guide](../installation.mdx#linux). + +1. Confirm that the version of the `teleport` binary is the one you expect: + + ```code + $ teleport version + ``` + +1. Now that you have installed a more recent `teleport` binary on your Auth + Service and Proxy Service instances, restart Teleport on these servers to run + the new version. + + (!docs/pages/includes/start-teleport.mdx!) + +### Self-hosted Teleport clusters on Kubernetes + +The instructions in this section assume that you have configured the +`teleport-cluster` Helm chart with a values file called `values.yaml`, and that +your `teleport-cluster` release is called `teleport-cluster`. + +1. Shrink the Auth Service pool. You must reduce the number of Auth Service + instances to one in order to ensure a consistent cluster state during the + upgrade. + + Ensure that your `teleport-cluster` values file includes the following + configuration: + + ```yaml + auth: + highAvailability: + replicaCount: 1 + ``` + +1. Update the Teleport Helm chart repository so you can install the latest + version of the `teleport-cluster` chart: + + (!docs/pages/kubernetes-access/helm/includes/helm-repo-add.mdx!) + +1. Upgrade the `teleport-cluster` Helm release: + + ```code + $ helm upgrade teleport-cluster teleport/teleport-cluster \ + --version= \ + --values=values.yaml + ``` + + The `teleport-cluster` Helm chart automatically waits for the previous + version of the Proxy Service to stop responding to requests before running a + new version of the Auth Service. + +1. Once you have completed this guide and upgraded the cluster, you can + configure your cluster for high availability again. + +### Teleport agents running on Kubernetes + +The instructions in this section assume that you have configured the +`teleport-kube-agent` Helm chart with a values file called `values.yaml`, and +that your `teleport-kube-agent` release is called `teleport-agent`. + +1. Update the Teleport Helm chart repository so you can install the latest + version of the `teleport-kube-agent` chart: + + (!docs/pages/kubernetes-access/helm/includes/helm-repo-add.mdx!) + +1. Upgrade the Helm release: + + ```code + $ helm -n upgrade teleport-agent teleport/teleport-kube-agent \ + --values=values.yaml \ + --version= + ``` + diff --git a/docs/pages/upgrading/self-hosted-automatic-agent-updates.mdx b/docs/pages/upgrading/self-hosted-automatic-agent-updates.mdx deleted file mode 100644 index 7ea7df78e9f7b..0000000000000 --- a/docs/pages/upgrading/self-hosted-automatic-agent-updates.mdx +++ /dev/null @@ -1,311 +0,0 @@ ---- -title: Set up Automatic Agent Upgrades -description: Describes how to set up automatic agent upgrades for self-hosted Teleport deployments. ---- - -Teleport supports automatic agent upgrades for systemd-based Linux distributions -using `apt`, `yum`, and `zypper` package managers, as well as Kubernetes -clusters. - -Teleport agents run an **upgrader** that queries a **version server** to -determine whether they are out of date. This guide describes how to set up your -infrastructure to support automatic updates. If you are a Teleport Cloud user -or run a version server already, return to the [Upgrading](../upgrading.mdx) -menu for the appropriate next steps to upgrade Teleport. - -The [Automatic Update Architecture](../architecture/agent-update-management.mdx) -guide explains how automatic agent upgrades work in more detail. - - -Systemd agents enrolled into automatic updates can only install versions -present in their package repositories. As Teleport 14 won't be published to -`stable/v13`, those agents will require manual intervention to be upgraded to -the next major version (adding a new APT/YUM/zypper repo for `stable/v14`). -Alternatively, you can use the `stable/rolling` channel, which contains -Teleport v13.3.2 forward, including future major releases. - - -## Prerequisites - -- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide, - which describes the sequence in which to upgrade components of your cluster. -- Self-hosted Teleport cluster running at least v14.3.7 or v15.1.3. -- (!docs/pages/includes/tctl-tsh-prerequisite.mdx!) -- (!docs/pages/includes/tctl.mdx!) - -## Step 1/4. Check the version advertised by the Teleport Proxy Service - -The Teleport Proxy service exposes by default a version channel called `default` -that serves the same version the Proxy Service is running. - -If your proxy public address is , you can -query the version server with the following command: - -```code -$ curl "https:///v1/webapi/automaticupgrades/channel/default/version" -(=teleport.version=) -``` - -
- -By default, the version server has a single `default` channel, serving the -version of the Teleport Proxy Service. If you want to override the default version or add other channels you -can use the `automatic_upgrades_channels` field in the Proxy Service configuration file: - -```yaml -proxy_service: - enabled: "yes" - automatic_upgrades_channels: - # Override the default version channel reachable at - # https:///v1/webapi/automaticupgrades/channel/default/version - default: - static_version: v14.2.1 - # Define a new version channel with a static version reachable at - # https:///v1/webapi/automaticupgrades/channel/m-static-channel/version - my-static-channel: - static_version: v14.2.0 - # Define a new version channel forwarding requests to an upstream version server - my-remote-channel: - forward_url: https://updates.releases.teleport.dev/v1/stable/cloud -``` - -You must ensure all proxies share the same `automatic_upgrades_channels` -configuration. If Proxy Service instances are configured differently, you will experience agents -flickering between versions as the version served is not consistent across -instances. - -
- -## Step 2/4. Configure the maintenance schedule - -At this point, the updaters can be configured to pull the version from the -release channel and upgrade the agents. In this step you'll configure a -maintenance schedule for the Teleport cluster that agents will use to determine -when to check for upgrades. - -### Create a Teleport role - -Create a Teleport role that can manage cluster maintenance configurations -through the `cluster_maintenance_config` dynamic resource. No preset Teleport -roles provide this ability, so you will need to create one. - -1. Create a file called `cmc-editor.yaml` with the following content: - - ```yaml - kind: role - version: v7 - metadata: - name: cmc-editor - spec: - allow: - rules: - - resources: ['cluster_maintenance_config'] - verbs: ['create', 'read', 'update', 'delete'] - ``` - -1. Create the role resource: - - ```code - $ tctl create cmd-editor.yaml - ``` - -1. Add the role to your Teleport user: - - (!docs/pages/includes/add-role-to-user.mdx!) - -### Create a cluster maintenance configuration - -Create the following `cmc.yaml` manifest allowing maintenance on Monday, -Wednesday and Friday between 02:00 and 03:00 UTC. - -(!docs/pages/includes/cluster-maintenance-config-spec.mdx!) - -Finally, apply the manifest using `tctl`: - -```code -$ tctl create cmc.yaml -maintenance window has been updated -``` - -## Step 3/4. Enroll Kubernetes agents in automatic updates - -Now that you have deployed a version server, you can enroll agents in automatic -upgrades. This guide begins with agents deployed on Kubernetes. If all of your -agents run on Linux servers, you can skip to [Step -4](#step-44-enroll-linux-agents-in-automatic-updates). - -### Install the agent upgrader Helm chart - -This section assumes that the name of your `teleport-kube-agent` release is -`teleport-agent`, and that you have installed it in the `teleport` namespace. - -1. Add the following chart values to the values file for the - `teleport-kube-agent` chart: - - ```yaml - updater: - enabled: true - # No need to set the `versionServer` value anymore since v13.4.15, v14.3.1 and v15. - releaseChannel: default - ``` - -1. Update the Helm chart release with the new values: - - ```code - $ helm -n upgrade teleport/teleport-kube-agent \ - --values=values.yaml \ - --version= - ``` - -### Verify that the upgrader is working properly - -1. You can validate the updater is running properly by checking if its pod is - ready: - - ```code - $ kubectl -n teleport-agent get pods - NAME READY STATUS RESTARTS AGE - -0 1/1 Running 0 14m - -1 1/1 Running 0 14m - -2 1/1 Running 0 14m - -updater-d9f97f5dd-v57g9 1/1 Running 0 16m - ``` - -1. Check for any deployment issues by checking the updater logs: - - ```code - $ kubectl -n logs deployment/-updater - 2023-04-28T13:13:30Z INFO StatefulSet is already up-to-date, not updating. {"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"} - ``` - -### Troubleshooting automatic agent upgrades on Kubernetes - -The updater is a controller that periodically reconciles expected Kubernetes -resources with those in the cluster. The updater executes a reconciliation loop -every 30 minutes or in response to a Kubernetes event. If you don't want to wait -until the next reconciliation, you can trigger an event. - -1. Any deployment update will send an event, so you can trigger the upgrader by - annotating the resource: - - ```code - $ kubectl -n annotate statefulset/ 'debug.teleport.dev/trigger-event=1' - ``` - -1. To suspend automatic updates for an agent, annotate the agent deployment - with `teleport.dev/skipreconcile: "true"`, either by setting the - `annotations.deployment` value in Helm, or by patching the deployment - directly with `kubectl`. - -## Step 4/4. Enroll Linux agents in automatic updates - -This section shows you how to enroll Teleport agents running on Linux virtual or -bare-metal machines into automatic updates. - -Follow these instructions on each of your Teleport agents. - -### Install the agent upgrader - -1. Ensure the Teleport repository is added and Teleport Enterprise is installed. - - To verify if the Teleport repository was added to the system, check if either of the follow files exist: - - ```code - $ ls /etc/apt/sources.list.d/teleport.list - # or - $ ls /etc/yum.repos.d/teleport.repo - ``` - - The upgrader checks the repository for available releases, so make sure that - it is up to date. - -1. If the repository was added, make sure the Teleport binary installed on the - agent can run the automatic updater: - - ```code - $ which teleport-upgrade || echo "Install the upgrader" - Install the upgrader - ``` - -1. If the Teleport repository is not found, or the Teleport binary you installed - does not include the upgrader, add the appropriate repository and reinstall - Teleport. - - (!docs/pages/includes/install-linux-ent-self-hosted.mdx to-install="teleport-ent-updater" !) - -### Configure the upgrader - -1. Create the upgrade configuration directory: - - ```code - $ sudo mkdir -p /etc/teleport-upgrade.d/ - ``` - -1. If you changed the agent user to run as non-root, create - `/etc/teleport-upgrade.d/schedule` and grant ownership to your Teleport user. - Otherwise, you can skip this step: - - ```code - $ sudo touch /etc/teleport-upgrade.d/schedule - $ sudo chown /etc/teleport-upgrade.d/schedule - ``` - -1. Configure the upgrader to connect to your custom version server and subscribe - to the right release channel: - - ```code - $ echo "/v1/webapi/automaticupgrades/channel/default" | sudo tee /etc/teleport-upgrade.d/endpoint - ``` - - Make sure not to include `https://` as a prefix to the server address, nor - suffix the endpoint with `/version`. - -### Verify that the upgrader is working properly - -1. Verify that the upgrader can see your version endpoint by checking for - upgrades: - - ```code - $ sudo teleport-upgrade dry-run - ``` - -1. You should see one of the following messages, depending on the target version - you are currently serving: - - ```text - no upgrades available (1.2.3 == 1.2.3) - an upgrade is available (1.2.3 -> 2.3.4) - ``` - - `teleport-upgrade` may display warnings about not having a valid upgrade - schedule. This is expected immediately after install as the maintenance - schedule might not be exported yet. - -### Troubleshooting automatic agent upgrades on Linux - -1. If an agent is not automatically upgraded, you can invoke the upgrader - manually and look at its logs: - - ```code - $ sudo teleport-upgrade run - ``` - -1. To suspend automatic updates, disable the systemd timer: - - ```code - $ sudo systemctl disable --now teleport-upgrade.timer - ``` - -1. To enable and start the systemd timer after suspending: - - ```code - $ sudo systemctl enable --now teleport-upgrade.timer - ``` - -## Next steps - -While this guide showed you how to apply a `cluster_maintenance_config` resource -using `tctl`, we recommend using infrastructure as code to maintain your -Teleport resources. See the -[`teleport_cluster_maintenance_config`](../reference/terraform-provider.mdx#teleport_cluster_maintenance_config) -for how to declare a cluster maintenance configuration with Terraform. diff --git a/docs/pages/upgrading/self-hosted-kubernetes.mdx b/docs/pages/upgrading/self-hosted-kubernetes.mdx deleted file mode 100644 index fb3753eb0dbd2..0000000000000 --- a/docs/pages/upgrading/self-hosted-kubernetes.mdx +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: Upgrade Self-Hosted Teleport Clusters on Kubernetes -description: Provides instructions for upgrading self-hosted Teleport clusters that run on Kubernetes. ---- - -This guide explains how to upgrade self-hosted Teleport clusters running on -Kubernetes. - -## Prerequisites - -- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide, - which describes the sequence in which to upgrade components of your cluster. -- A self-hosted Teleport cluster in which the Auth Service and Proxy Service run - on Kubernetes. This guide assumes that you have deployed the Teleport cluster - using the `teleport-cluster` Helm chart. -- (!docs/pages/includes/tctl-tsh-prerequisite.mdx!) -- (!docs/pages/includes/tctl.mdx!) - -This guide assumes that you have configured the `teleport-cluster` Helm chart -with a values file called `values.yaml`, and that your `teleport-cluster` -release is called `teleport-cluster`. - -## Step 1/3. Shrink the Auth Service pool - -You must reduce the number of Auth Service instances to one in order to ensure a -consistent cluster state during the upgrade. - -Ensure that your `teleport-cluster` values file includes the following -configuration: - -```yaml -auth: - highAvailability: - replicaCount: 1 -``` - -Once you have completed this guide and upgraded the cluster, you can configure -your cluster for high availability again. - -## Step 2/3. Upgrade the Auth Service and Proxy Service - -Run the following commands to upgrade Auth Service and Proxy Service instances -running on Kubernetes. - -1. Update the Teleport Helm chart repository so you can install the latest - version of the `teleport-cluster` chart: - - (!docs/pages/kubernetes-access/helm/includes/helm-repo-add.mdx!) - -1. Upgrade the Helm release: - - ```code - $ helm upgrade teleport-cluster teleport/teleport-cluster \ - --version= \ - --values=values.yaml - ``` - -The `teleport-cluster` Helm chart automatically waits for the previous version -of the Proxy Service to stop responding to requests before running a new version -of the Auth Service. - -## Step 3/3. Upgrade agents - -Run the following commands to upgrade Teleport agents running on Kubernetes. - -1. Update the Teleport Helm chart repository so you can install the latest - version of the `teleport-kube-agent` chart: - - (!docs/pages/kubernetes-access/helm/includes/helm-repo-add.mdx!) - -1. Upgrade the Helm release: - - ```code - $ helm -n upgrade teleport/teleport-kube-agent \ - --values=values.yaml \ - --version= - ``` - diff --git a/docs/pages/upgrading/self-hosted-linux.mdx b/docs/pages/upgrading/self-hosted-linux.mdx deleted file mode 100644 index e1abdd7217ce6..0000000000000 --- a/docs/pages/upgrading/self-hosted-linux.mdx +++ /dev/null @@ -1,43 +0,0 @@ ---- -title: Upgrade Self-Hosted Teleport Clusters on Linux -description: Provides instructions for upgrading self-hosted Teleport clusters that run on Linux servers. ---- - -This guide explains how to upgrade self-hosted Teleport clusters running on -Linux servers. - -## Prerequisites - -- Familiarity with the [Upgrading Compatibility Overview](./overview.mdx) guide, - which describes the sequence in which to upgrade components of your cluster. - -- A self-hosted Teleport cluster in which the Auth Service and Proxy Service run - on Linux servers. - - If you are running more than one Auth Service instance, you must reduce the - size of the Auth Service instance pool to one in order to perform an upgrade. - -- (!docs/pages/includes/tctl-tsh-prerequisite.mdx!) - -- (!docs/pages/includes/tctl.mdx!) - -## Upgrade Teleport binaries - -Complete the following steps on all servers that run the Auth Service and Proxy -Service, then on each of your agents: - -1. Install the latest Teleport version on the host. - - (!docs/pages/includes/install-linux-self-hosted.mdx!) - -1. Confirm that the version of the `teleport` binary is the one you expect: - - ```code - $ teleport version - ``` - -1. Now that you have installed a more recent `teleport` binary on your Auth - Service and Proxy Service servers, restart Teleport on these servers to run - the new version. - - (!docs/pages/includes/start-teleport.mdx!)