+
+#### Create and restore Elasticsearch backup
+
+}
+desired={}
+/>
+
+
+
+#### Current state
+
+The Camunda web applications are currently not reachable by end-users and will not process any new process instances. This allows creating a backup of Elasticsearch without losing any data.
+
+#### Desired state
+
+You are creating a backup of the main Elasticsearch instance in the surviving region and restore it in the recreated region. This Elasticsearch backup contains all the data and may take some time to be finished. The failover Elasticsearch instance only contains a subset of the data from after the region loss and is not sufficient to restore this in the new region.
+
+#### How to get there
+
+This builds on top of the [AWS Setup](./../../platform-deployment/helm-kubernetes/platforms/amazon-eks/dual-region.md) and assumes that the S3 bucket was automatically created as part of the Terraform execution.
+
+:::info
+
+The procedure works for other cloud providers and bare metal the same. You have to adjust the AWS S3 specific part depending on your chosen backup source for Elasticsearch. Make sure to conduct the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) on snapshot and restore to learn more about it and specifically the [different supported types](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html#ess-repo-types) by Elasticsearch.
+
+:::
+
+1. Determine the S3 bucket name by retrieving it via Terraform. Go to `aws/dual-region/terraform` within the repository and retrieve the bucket name from the Terraform state.
+
+```bash
+export S3_BUCKET_NAME=$(terraform output -raw s3_bucket_name)
+```
+
+2. Configure Elasticsearch backup endpoint in the surviving namespace `CAMUNDA_NAMESPACE_SURVIVING`
+
+```bash
+ELASTIC_POD=$(kubectl --context $CLUSTER_SURVIVING get pod --selector=app\.kubernetes\.io/name=elasticsearch -o jsonpath='{.items[0].metadata.name}' -n $CAMUNDA_NAMESPACE_SURVIVING)
+kubectl --context $CLUSTER_SURVIVING exec -n $CAMUNDA_NAMESPACE_SURVIVING -it $ELASTIC_POD -- curl -XPUT "http://localhost:9200/_snapshot/camunda_backup" -H "Content-Type: application/json" -d'
+{
+ "type": "s3",
+ "settings": {
+ "bucket": "'$S3_BUCKET_NAME'",
+ "client": "camunda",
+ "base_path": "backups"
+ }
+}
+'
+```
+
+3. Create an Elasticsearch backup in the surviving namespace `CAMUNDA_NAMESPACE_SURVIVING`. Depending on the amount of data, this operation will take a while to complete.
+
+```bash
+# The backup will be called failback
+kubectl --context $CLUSTER_SURVIVING exec -n $CAMUNDA_NAMESPACE_SURVIVING -it $ELASTIC_POD -- curl -XPUT "http://localhost:9200/_snapshot/camunda_backup/failback?wait_for_completion=true"
+```
+
+4. Verify that the backup has been completed successfully by checking all backups and ensuring the `state` is `SUCCESS`
+
+```bash
+kubectl --context $CLUSTER_SURVIVING exec -n $CAMUNDA_NAMESPACE_SURVIVING -it $ELASTIC_POD -- curl -XGET "http://localhost:9200/_snapshot/camunda_backup/_all"
+```
+
+
+ Example output
+
+
+```json
+{
+ "snapshots": [
+ {
+ "snapshot": "failback",
+ "uuid": "uTHGdUAYSk-91aAS0sMKFQ",
+ "repository": "camunda_backup",
+ "version_id": 8090299,
+ "version": "8.9.2",
+ "indices": [
+ "operate-web-session-1.1.0_",
+ "tasklist-form-8.4.0_",
+ "operate-process-8.3.0_",
+ "zeebe-record_process-instance-creation_8.4.5_2024-03-28",
+ "operate-batch-operation-1.0.0_",
+ "operate-user-1.2.0_",
+ "operate-incident-8.3.1_",
+ "zeebe-record_job_8.4.5_2024-03-28",
+ "operate-variable-8.3.0_",
+ "tasklist-web-session-1.1.0_",
+ "tasklist-draft-task-variable-8.3.0_",
+ "operate-operation-8.4.0_",
+ "zeebe-record_process_8.4.5_2024-03-28",
+ ".ds-.logs-deprecation.elasticsearch-default-2024.03.28-000001",
+ "tasklist-process-8.4.0_",
+ "operate-metric-8.3.0_",
+ "operate-flownode-instance-8.3.1_",
+ "tasklist-flownode-instance-8.3.0_",
+ "tasklist-variable-8.3.0_",
+ "tasklist-metric-8.3.0_",
+ "operate-post-importer-queue-8.3.0_",
+ "tasklist-task-variable-8.3.0_",
+ "operate-event-8.3.0_",
+ "tasklist-process-instance-8.3.0_",
+ "operate-import-position-8.3.0_",
+ "operate-decision-requirements-8.3.0_",
+ "zeebe-record_command-distribution_8.4.5_2024-03-28",
+ "operate-list-view-8.3.0_",
+ "zeebe-record_process-instance_8.4.5_2024-03-28",
+ "tasklist-import-position-8.2.0_",
+ "tasklist-user-1.4.0_",
+ "operate-decision-instance-8.3.0_",
+ "zeebe-record_deployment_8.4.5_2024-03-28",
+ "operate-migration-steps-repository-1.1.0_",
+ "tasklist-migration-steps-repository-1.1.0_",
+ ".ds-ilm-history-5-2024.03.28-000001",
+ "operate-decision-8.3.0_",
+ "operate-sequence-flow-8.3.0_",
+ "tasklist-task-8.4.0_"
+ ],
+ "data_streams": [
+ "ilm-history-5",
+ ".logs-deprecation.elasticsearch-default"
+ ],
+ "include_global_state": true,
+ "state": "SUCCESS",
+ "start_time": "2024-03-28T03:17:38.340Z",
+ "start_time_in_millis": 1711595858340,
+ "end_time": "2024-03-28T03:17:39.340Z",
+ "end_time_in_millis": 1711595859340,
+ "duration_in_millis": 1000,
+ "failures": [],
+ "shards": {
+ "total": 43,
+ "failed": 0,
+ "successful": 43
+ },
+ "feature_states": []
+ }
+ ],
+ "total": 1,
+ "remaining": 0
+}
+```
+
+
+
+
+5. Configure Elasticsearch backup endpoint in the new region namespace `CAMUNDA_NAMESPACE_RECREATED`. It's essential to only do this step now as otherwise it won't see the backup.
+
+```bash
+ELASTIC_POD=$(kubectl --context $CLUSTER_RECREATED get pod --selector=app\.kubernetes\.io/name=elasticsearch -o jsonpath='{.items[0].metadata.name}' -n $CAMUNDA_NAMESPACE_RECREATED)
+kubectl --context $CLUSTER_RECREATED exec -n $CAMUNDA_NAMESPACE_RECREATED -it $ELASTIC_POD -- curl -XPUT "http://localhost:9200/_snapshot/camunda_backup" -H "Content-Type: application/json" -d'
+{
+ "type": "s3",
+ "settings": {
+ "bucket": "'$S3_BUCKET_NAME'",
+ "client": "camunda",
+ "base_path": "backups"
+ }
+}
+'
+```
+
+6. Verify that the backup can be found in the shared S3 bucket
+
+```bash
+kubectl --context $CLUSTER_RECREATED exec -n $CAMUNDA_NAMESPACE_RECREATED -it $ELASTIC_POD -- curl -XGET "http://localhost:9200/_snapshot/camunda_backup/_all"
+```
+
+The example output above should be the same since it's the same backup.
+
+7. Restore Elasticsearch backup in the new region namespace `CAMUNDA_NAMESPACE_RECREATED`. Depending on the amount of data, this operation will take a while to complete.
+
+```bash
+kubectl --context $CLUSTER_RECREATED exec -n $CAMUNDA_NAMESPACE_RECREATED -it $ELASTIC_POD -- curl -XPOST "http://localhost:9200/_snapshot/camunda_backup/failback/_restore?wait_for_completion=true"
+```
+
+8. Verify that the restore has been completed successfully in the new region.
+
+```bash
+kubectl --context $CLUSTER_RECREATED exec -n $CAMUNDA_NAMESPACE_RECREATED -it $ELASTIC_POD -- curl -XGET "http://localhost:9200/_snapshot/camunda_backup/failback/_status"
+```
+
+
+ Example output
+
+
+The important part being the `state: "SUCCESS"` and that `done` and `total` are equal. This is just an example and the values will differ for you!
+
+```json
+{
+ "snapshots": [
+ {
+ "snapshot": "failback",
+ "repository": "camunda_backup",
+ "uuid": "8AmblqA2Q9WAhuDk-NO5Cg",
+ "state": "SUCCESS",
+ "include_global_state": true,
+ "shards_stats": {
+ "initializing": 0,
+ "started": 0,
+ "finalizing": 0,
+ "done": 43,
+ "failed": 0,
+ "total": 43
+ },
+ "stats": {
+ "incremental": {
+ "file_count": 145,
+ "size_in_bytes": 353953
+ },
+ "total": {
+ "file_count": 145,
+ "size_in_bytes": 353953
+ },
+ "start_time_in_millis": 1712058365525,
+ "time_in_millis": 1005
+ },
+ "indices": {
+ ...
+ }
+ }
+ ]
+}
+```
+
+
+
+
+
+
+
+