Skip to content

Commit

Permalink
Update upgrade docs (#557)
Browse files Browse the repository at this point in the history 
* upgrade upgrade documentation.

* add workaround.

* Capitalize character

* Capitalizing characters and mark a command as a command

Co-authored-by: Arthur Wolf <[email protected]>
  • Loading branch information
@julialongtin @arthurwolf
julialongtin and arthurwolf authored Apr 21, 2022
1 parent daebed4 commit f1c4633
Showing 1 changed file with 77 additions and 17 deletions.
94 changes: 77 additions & 17 deletions offline/upgrading.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ df -h
sudo docker container prune
```

Prune old security update deployment archives:
```
sudo apt clean
```

### Kubernetes hosts:

#### Wire
Remove wire-server images from two releases ago, or from the current release that we know are unused. For instance,

```
Expand All @@ -30,17 +37,18 @@ sudo docker image ls | grep -E "^quay.io/wire/" | grep $VERSION | sed "s/.*[ ]*\
```

If you are not running SFT in your main cluster (for example, do not use SFT, or have SFT in a separate DMZ'd cluster).. then remove SFT images from the Wire Kubernetes.
```
sudo docker image ls | grep -E "^quay.io/wire/sftd" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/"
```

#### SFT
If you are running a DMZ deployment, prune the old wire-server images and their dependencies on the SFT kubernetes hosts...
```
sudo docker image ls | grep -E "^quay.io/wire/(team-settings|account|webapp|namshi-smtp)" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/"
sudo docker image ls | grep -E "^bitnami/redis" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/"
sudo docker image ls | grep -E "^airdock/fake-sqs" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/"
sudo docker image ls | grep -E "^localstack/localstack" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/"
sudo docker image ls | grep -E "^(bitnami/redis|airdock/fake-sqs|localstack/localstack)" | sed "s/.*[ ]*\([0-9a-f]\{12\}\).*/sudo docker image rm \1/"
sudo docker image rm
```
and remove SFT images from the Wire Kubernetes.
```
```

## Preparing for deployment
Verify you have the container images and configuration for the version of wire you are currently running.
Expand Down Expand Up @@ -129,7 +137,7 @@ https://docs.wire.com/how-to/install/ansible-VMs.html

### updates to the inventory

make sure your inventory sets:
Make sure your inventory sets:

# Explicitely specify the restund user id to be "root" to override the default of "997"
restund_uid = root
Expand All @@ -140,7 +148,7 @@ minio_deeplink_domain = prefix-

# move the kubeconfig

old versions of the package contained the kubeconfig at ansible/kubeconfig. newer ones create a directory at ansible/inventory/offline/artifacts, and place the kubeconfig there, as 'admin.conf'
Old versions of the package contained the kubeconfig at ansible/kubeconfig. newer ones create a directory at ansible/inventory/offline/artifacts, and place the kubeconfig there, as 'admin.conf'

If your deployment package uses the old style, then in the place where you are keeping your new package:
```
Expand All @@ -151,11 +159,16 @@ cp ../<OLD_PACKAGE_DIR/ansible/kubeconfig ansible/inventory/offline/artifacts/ad
otherwise:
```
mkdir ansible/inventory/offline/artifacts
cp ../<OLD_PACKAGE_DIR/ansible/inventory/offline/artifacts/admin.conf ansible/inventory/offline/artifacts/admin.conf
sudo cp ../<OLD_PACKAGE_DIR>/ansible/inventory/offline/artifacts/admin.conf ansible/inventory/offline/artifacts/admin.conf
```

## Preparing to upgrade kubernetes services

Log into the assethost, and verify the 'serve-assets' systemd component is running by looking at `netstat -an`, and checking for `8080`. If it's not:
```
sudo service serve-assets start
```

Since docker is already installed on all nodes that need it, push the new container images to the assethost, and seed all container images:

```
Expand All @@ -170,37 +183,84 @@ d kubectl get nodes -owide
```
They should all report ready.

If you are worried about disk space, removing unneeded images could be performed here.

## Upgrading wire-server using helm

### Upgrading non-wire components:

Copy your external service definition values into place.
#### External Service Definitions:

Compare your external service definition files, and decide whether you need to change them or not.
```
diff -u ../<OLD_PACKAGE_DIR>/values/cassandra-external/values.yaml values/cassandra-external/prod-values.example.yaml
diff -u ../<OLD_PACKAGE_DIR>/values/elasticsearch-external/values.yaml values/elasticsearch-external/prod-values.example.yaml
diff -u ../<OLD_PACKAGE_DIR>/values/minio-external/values.yaml values/minio-external/prod-values.example.yaml
```

First, upgrade the external service definitions, as those rarely change.
If there are only IP addresses in the diff output, copy these files into your new tree.
```
cp ../<OLD_PACKAGE_DIR>/values/cassandra-external/values.yaml values/cassandra-external/values.yaml
cp ../<OLD_PACKAGE_DIR>/values/elasticsearch-external/values.yaml values/elasticsearch-external/values.yaml
cp ../<OLD_PACKAGE_DIR>/values/minio-external/values.yaml values/minio-external/values.yaml
```

If not, examine differences between the values files for the old service definitions and the new service definitions

When you are satisfied with the results of the above, upgrade the external service definitions.
```
d helm upgrade cassandra-external ./charts/cassandra-external/ --values ./values/cassandra-external/values.yaml
d helm upgrade elasticsearch-external ./charts/elasticsearch-external/ --values ./values/elasticsearch-external/values.yaml
d helm upgrade minio-external ./charts/minio-external/ --values ./values/minio-external/values.yaml
```

#### Non-Wire Services

Compare your non-wire service definition files, and decide whether you need to change them or not.
```
diff -u ../<OLD_PACKAGE_DIR>/values/fake-aws/prod-values.example.yaml values/cassandra-external/prod-values.example.yaml
diff -u ../<OLD_PACKAGE_DIR>/values/databases-ephemeral/values.yaml values/databases-ephemeral/prod-values.example.yaml
```

If there are no differences, copy these files into your new tree.
```
cp ../<OLD_PACKAGE_DIR>/values/fake-aws/prod-values.example.yaml values/cassandra-external/values.yaml
cp ../<OLD_PACKAGE_DIR>/values/databases-ephemeral/values.yaml values/databases-ephemeral/values.yaml
```

Next, upgrade the internal non-wire services.
```
d helm upgrade fake-aws ./charts/fake-aws/ --values ./values/fake-aws/prod-values.example.yaml
d helm upgrade fake-aws ./charts/fake-aws/ --values ./values/fake-aws/values.yaml
d helm upgrade databases-ephemeral ./charts/databases-ephemeral/ --values ./values/databases-ephemeral/values.yaml
d helm upgrade reaper ./charts/reaper/
```

Finally, upgrade demo-smtp. Note that you may have to look for a `values.yaml`, instead of a `prod-values.example.yaml` file, in the case that demo-smtp was configured. If it's not present, use `prod-values.example.yaml`.
#### Demo-SMTP service

Compare your demo-smtp configuration files, and decide whether you need to change them or not.
```
diff -u ../<OLD_PACKAGE_DIR>/values/demo-smtp/values.yaml values/demo-smtp/values.yaml
```

If there are no differences, copy these files into your new tree.
```
cp ../<OLD_PACKAGE_DIR>/values/demo-smtp/values.yaml values/demo-smtp/values.yaml
```

```
d helm upgrade demo-smtp ./charts/demo-smtp/ --values ./values/demo-smtp/values.yaml
```

### Upgrading the NginX Ingress
#### Upgrading the NginX Ingress

Compare your demo-smtp configuration files, and decide whether you need to change them or not.
```
diff -u ../<OLD_PACKAGE_DIR>/values/ngin-ingress-services/values.yaml values/nginx-ingress-services/prod-values.example.yaml
```

If there are no differences, copy these files into your new tree.
```
cp ../<OLD_PACKAGE_DIR>/values/nginx-ingress-services/values.yaml values/nginx-ingress-services/values.yaml
```

d helm upgrade nginx-ingress-controller ./charts/nginx-ingress-controller/
d helm upgrade nginx-ingress-services ./charts/nginx-ingress-services/ --values ./values/nginx-ingress-services/values.yaml --values ./values/nginx-ingress-services/secrets.yaml
```
Expand All @@ -212,7 +272,7 @@ Inspect your `values.yaml` and `secrets.yaml` files with diff comparing them to
Now upgrade `wire-server`:
```
d helm upgrade wire-server ./charts/wire-server --timeout=15m0s --values ./values/wire-server/values.yaml --values ./values/wire-server/secrets.yaml
d helm upgrade wire-server ./charts/wire-server/ --timeout=15m0s --values ./values/wire-server/values.yaml --values ./values/wire-server/secrets.yaml
```
### Marking kubenode for calling server (SFT)
Expand Down

0 comments on commit f1c4633

Please sign in to comment.