From f1c4633b417c2ac9c822727f012b2045e8add548 Mon Sep 17 00:00:00 2001 From: Julia Longtin Date: Thu, 21 Apr 2022 09:27:57 +0100 Subject: [PATCH] Update upgrade docs (#557) * upgrade upgrade documentation. * add workaround. * Capitalize character * Capitalizing characters and mark a command as a command Co-authored-by: Arthur Wolf --- offline/upgrading.md | 94 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 77 insertions(+), 17 deletions(-) diff --git a/offline/upgrading.md b/offline/upgrading.md index 79ae99188..ef8532f42 100644 --- a/offline/upgrading.md +++ b/offline/upgrading.md @@ -20,7 +20,14 @@ df -h sudo docker container prune ``` +Prune old security update deployment archives: +``` +sudo apt clean +``` + ### Kubernetes hosts: + +#### Wire Remove wire-server images from two releases ago, or from the current release that we know are unused. For instance, ``` @@ -30,17 +37,18 @@ sudo docker image ls | grep -E "^quay.io/wire/" | grep $VERSION | sed "s/.*[ ]*\ ``` +If you are not running SFT in your main cluster (for example, do not use SFT, or have SFT in a separate DMZ'd cluster).. then remove SFT images from the Wire Kubernetes. +``` +sudo docker image ls | grep -E "^quay.io/wire/sftd" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/" +``` + +#### SFT If you are running a DMZ deployment, prune the old wire-server images and their dependencies on the SFT kubernetes hosts... ``` sudo docker image ls | grep -E "^quay.io/wire/(team-settings|account|webapp|namshi-smtp)" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/" -sudo docker image ls | grep -E "^bitnami/redis" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/" -sudo docker image ls | grep -E "^airdock/fake-sqs" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/" -sudo docker image ls | grep -E "^localstack/localstack" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/" +sudo docker image ls | grep -E "^(bitnami/redis|airdock/fake-sqs|localstack/localstack)" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/" sudo docker image rm ``` -and remove SFT images from the Wire Kubernetes. -``` -``` ## Preparing for deployment Verify you have the container images and configuration for the version of wire you are currently running. @@ -129,7 +137,7 @@ https://docs.wire.com/how-to/install/ansible-VMs.html ### updates to the inventory -make sure your inventory sets: +Make sure your inventory sets: # Explicitely specify the restund user id to be "root" to override the default of "997" restund_uid = root @@ -140,7 +148,7 @@ minio_deeplink_domain = prefix- # move the kubeconfig -old versions of the package contained the kubeconfig at ansible/kubeconfig. newer ones create a directory at ansible/inventory/offline/artifacts, and place the kubeconfig there, as 'admin.conf' +Old versions of the package contained the kubeconfig at ansible/kubeconfig. newer ones create a directory at ansible/inventory/offline/artifacts, and place the kubeconfig there, as 'admin.conf' If your deployment package uses the old style, then in the place where you are keeping your new package: ``` @@ -151,11 +159,16 @@ cp ..//ansible/inventory/offline/artifacts/admin.conf ansible/inventory/offline/artifacts/admin.conf ``` ## Preparing to upgrade kubernetes services +Log into the assethost, and verify the 'serve-assets' systemd component is running by looking at `netstat -an`, and checking for `8080`. If it's not: +``` +sudo service serve-assets start +``` + Since docker is already installed on all nodes that need it, push the new container images to the assethost, and seed all container images: ``` @@ -170,37 +183,84 @@ d kubectl get nodes -owide ``` They should all report ready. -If you are worried about disk space, removing unneeded images could be performed here. - ## Upgrading wire-server using helm ### Upgrading non-wire components: -Copy your external service definition values into place. +#### External Service Definitions: + +Compare your external service definition files, and decide whether you need to change them or not. ``` +diff -u ..//values/cassandra-external/values.yaml values/cassandra-external/prod-values.example.yaml +diff -u ..//values/elasticsearch-external/values.yaml values/elasticsearch-external/prod-values.example.yaml +diff -u ..//values/minio-external/values.yaml values/minio-external/prod-values.example.yaml ``` -First, upgrade the external service definitions, as those rarely change. +If there are only IP addresses in the diff output, copy these files into your new tree. +``` +cp ..//values/cassandra-external/values.yaml values/cassandra-external/values.yaml +cp ..//values/elasticsearch-external/values.yaml values/elasticsearch-external/values.yaml +cp ..//values/minio-external/values.yaml values/minio-external/values.yaml +``` + +If not, examine differences between the values files for the old service definitions and the new service definitions + +When you are satisfied with the results of the above, upgrade the external service definitions. ``` d helm upgrade cassandra-external ./charts/cassandra-external/ --values ./values/cassandra-external/values.yaml d helm upgrade elasticsearch-external ./charts/elasticsearch-external/ --values ./values/elasticsearch-external/values.yaml d helm upgrade minio-external ./charts/minio-external/ --values ./values/minio-external/values.yaml ``` +#### Non-Wire Services + +Compare your non-wire service definition files, and decide whether you need to change them or not. +``` +diff -u ..//values/fake-aws/prod-values.example.yaml values/cassandra-external/prod-values.example.yaml +diff -u ..//values/databases-ephemeral/values.yaml values/databases-ephemeral/prod-values.example.yaml +``` + +If there are no differences, copy these files into your new tree. +``` +cp ..//values/fake-aws/prod-values.example.yaml values/cassandra-external/values.yaml +cp ..//values/databases-ephemeral/values.yaml values/databases-ephemeral/values.yaml +``` + Next, upgrade the internal non-wire services. ``` -d helm upgrade fake-aws ./charts/fake-aws/ --values ./values/fake-aws/prod-values.example.yaml +d helm upgrade fake-aws ./charts/fake-aws/ --values ./values/fake-aws/values.yaml d helm upgrade databases-ephemeral ./charts/databases-ephemeral/ --values ./values/databases-ephemeral/values.yaml d helm upgrade reaper ./charts/reaper/ ``` -Finally, upgrade demo-smtp. Note that you may have to look for a `values.yaml`, instead of a `prod-values.example.yaml` file, in the case that demo-smtp was configured. If it's not present, use `prod-values.example.yaml`. +#### Demo-SMTP service + +Compare your demo-smtp configuration files, and decide whether you need to change them or not. +``` +diff -u ..//values/demo-smtp/values.yaml values/demo-smtp/values.yaml +``` + +If there are no differences, copy these files into your new tree. +``` +cp ..//values/demo-smtp/values.yaml values/demo-smtp/values.yaml +``` + ``` d helm upgrade demo-smtp ./charts/demo-smtp/ --values ./values/demo-smtp/values.yaml ``` -### Upgrading the NginX Ingress +#### Upgrading the NginX Ingress + +Compare your demo-smtp configuration files, and decide whether you need to change them or not. +``` +diff -u ..//values/ngin-ingress-services/values.yaml values/nginx-ingress-services/prod-values.example.yaml ``` + +If there are no differences, copy these files into your new tree. +``` +cp ..//values/nginx-ingress-services/values.yaml values/nginx-ingress-services/values.yaml +``` + d helm upgrade nginx-ingress-controller ./charts/nginx-ingress-controller/ d helm upgrade nginx-ingress-services ./charts/nginx-ingress-services/ --values ./values/nginx-ingress-services/values.yaml --values ./values/nginx-ingress-services/secrets.yaml ``` @@ -212,7 +272,7 @@ Inspect your `values.yaml` and `secrets.yaml` files with diff comparing them to Now upgrade `wire-server`: ``` -d helm upgrade wire-server ./charts/wire-server --timeout=15m0s --values ./values/wire-server/values.yaml --values ./values/wire-server/secrets.yaml +d helm upgrade wire-server ./charts/wire-server/ --timeout=15m0s --values ./values/wire-server/values.yaml --values ./values/wire-server/secrets.yaml ``` ### Marking kubenode for calling server (SFT)