Skip to content

Commit

Permalink
feat: document new backup/restore procedure and webapps specific config
Browse files Browse the repository at this point in the history 
relates #4790
  • Loading branch information
@entangled90
entangled90 committed Dec 18, 2024
1 parent 2e28756 commit b18d8db
Show file tree
Hide file tree
Showing 3 changed files with 101 additions and 49 deletions.
26 changes: 11 additions & 15 deletions docs/self-managed/operational-guides/backup-restore/backup-and-restore.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ keywords: ["backup", "backups"]
---

:::note
This release introduces breaking changes for [Operate and Tasklist](./operate-tasklist-backup.md), as well as [Optimize](./optimize-backup.md).
This release introduces breaking changes for [Operate, Tasklist and Optimize](./webapps-backup.md).
:::

You can use the backup feature of Camunda 8 Self-Managed to regularly back up the state of all of its components (Zeebe, Operate, Tasklist, and Optimize) without any downtime (except Web Modeler, see [the Web Modeler backup and restore documentation](./modeler-backup-and-restore.md)).
Expand All @@ -24,22 +24,18 @@ Zeebe stores its backup to an external storage and must be configured before the

### Backup process

The backup of each component and the backup of a Camunda 8 cluster is identified by an id. This means a backup `x` of Camunda 8 consists of backup `x` of Zeebe, backup `x` of Optimize, backup `x` of Operate, and backup `x` of Tasklist. The backup ID must be an integer and greater than the previous backups.
The backup of each component and the backup of a Camunda 8 cluster is identified by an id. This means a backup `x` of Camunda 8 consists of backup `x` of Zeebe, backup `x` of Webapps(which includes Optimize, Operate and Tasklist). The backup ID must be an integer and greater than the previous backups.

:::note
We recommend using the timestamp as the backup id.
:::

To back up a Camunda 8 cluster, execute the following sequential steps:

1. Trigger a backup `x` of Optimize. See [how to take an Optimize backup](/self-managed/operational-guides/backup-restore/optimize-backup.md).
2. Trigger a backup `x` of Operate. See [how to take an Operate backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md).
3. Trigger a backup `x` of Tasklist. See [how to take a Tasklist backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md).
4. Wait until the backup `x` of Optimize is complete. See [how to monitor an Optimize backup](/self-managed/operational-guides/backup-restore/optimize-backup.md).
5. Wait until the backup `x` of Operate is complete. See [how to monitor an Operate backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md).
6. Wait until the backup `x` of Tasklist is complete. See [how to monitor a Tasklist backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md).
7. Soft pause exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md).
8. Take a backup `x` of the exported Zeebe records in Elasticsearch using the Elasticsearch Snapshots API.
1. Soft pause exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md).
2. Trigger a backup `x` of Webapps. See [how to take a Webapp backup](/self-managed/operational-guides/backup-restore/webapps-backup.md).
3. Wait until the backup `x` of Webapps is complete. See [how to monitor an Webapp backup](/self-managed/operational-guides/backup-restore/webapps-backup.md).
4. Take a backup `x` of the exported Zeebe records in Elasticsearch using the Elasticsearch Snapshots API.

```
Expand All @@ -53,10 +49,10 @@ PUT /_snapshot/my_repository/camunda_zeebe_records_backup_x

By default, the indices are prefixed with `zeebe-record`. If you have configured a different prefix when configuring Elasticsearch exporter in Zeebe, use this instead.

9. Wait until the backup `x` of the exported Zeebe records is complete before proceeding.
Take a backup `x` of Zeebe. See [how to take a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md).
10. Wait until the backup `x` of Zeebe is completed before proceeding. See [how to monitor a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md).
Resume exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md).
5. Wait until the backup `x` of the exported Zeebe records is complete before proceeding.
6. Take a backup `x` of Zeebe. See [how to take a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md).
7. Wait until the backup `x` of Zeebe is completed before proceeding. See [how to monitor a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md).
8. Resume exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md).

:::note
If any of the steps above fail, you may have to restart with a new backup id. Ensure exporting is resumed if the backup process force quits in the middle of the process.
Expand All @@ -70,7 +66,7 @@ To restore a Camunda 8 cluster from a backup, all components must be restored fr
2. Confirm proper configuration (such as shards, replicas count, etc.)
3. Stop Operate, Tasklist, and Optimize.
4. Delete all indices.
5. Restore the state of [Operate](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md), [Tasklist](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md), and [Optimize](/self-managed/operational-guides/backup-restore/optimize-backup.md).
5. Restore the state of [Webapps](/self-managed/operational-guides/backup-restore/webapps-backup.md).
6. Restore `zeebe-records*` indices from Elasticsearch snapshot.
7. Restore [Zeebe](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md).
8. Start Zeebe, Operate, Tasklist, and Optimize.
121 changes: 89 additions & 32 deletions ...backup-restore/operate-tasklist-backup.md → ...l-guides/backup-restore/webapps-backup.md
View file
Original file line number Diff line number Diff line change
@@ -1,41 +1,51 @@
---
id: operate-tasklist-backup
title: Backup and restore Operate and Tasklist data
description: "How to perform a backup and restore of Operate and Tasklist data."
id: webapps-backup
title: Backup and restore Webapps (Operate, Tasklist and Optimize) data
description: "How to perform a backup and restore of Webapps (Operate, Tasklist and Optimize) data."
keywords: ["backup", "backups"]
---

import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";

:::warning breaking changes!
As of the Camunda 8.6 release, the `/actuator` endpoints (including `/backups`) now default to port 9600. Ensure your `management.server.port` configuration parameter is correctly set before continuing.
:::warning breaking changes
As of the Camunda 8.7 release, the `/actuator` endpoints for backups have been moved to `/actuator/backupHistory`. The old `/actuator/backups` endpoint is still active only if the applications are deployed standalone (i.e. each application in its own process)
:::

Operate stores its data over multiple indices in Elasticsearch. Backup of Operate data includes several
Elasticsearch snapshots containing sets of Operate indices. Each backup is identified by `backupId`. For example, a backup with an ID of `123` may contain the following Elasticsearch snapshots:
:::note
In this page we will refer to "webapps" as Operate, Tasklist and Optimize. Depending on your deployment configuration, you may not have all of them deployed. If that is the case, you can ignore the configurations for those applications.
:::

Webapps store their data over multiple indices in Elasticsearch. Backup of Webapps data includes several
Elasticsearch snapshots containing sets of different indices. Each backup is identified by `backupId`. For example, a backup with an ID of `123` may contain the following Elasticsearch snapshots:

```
camunda_operate_123_8.1.0_part_1_of_6
camunda_operate_123_8.1.0_part_2_of_6
camunda_operate_123_8.1.0_part_3_of_6
camunda_operate_123_8.1.0_part_4_of_6
camunda_operate_123_8.1.0_part_5_of_6
camunda_operate_123_8.1.0_part_6_of_6
camunda_webapps_123_8.1.0_part_1_of_6
camunda_webapps_123_8.1.0_part_2_of_6
camunda_webapps_123_8.1.0_part_3_of_6
camunda_webapps_123_8.1.0_part_4_of_6
camunda_webapps_123_8.1.0_part_5_of_6
camunda_webapps_123_8.1.0_part_6_of_6
```

Operate provides an API to perform a backup and manage backups (list, check state, delete). Restore a backup using the standard Elasticsearch API.
All webapps provides an API to perform a backup and manage backups (list, check state, delete). Restore a backup using the standard Elasticsearch API.

:::note
The backup API can be reached via the Actuator management port, which by default is the same as application HTTP port (and in turn defaults to 9600). The port may be reconfigured with the help of `management.server.port` configuration parameter.
:::

## Prerequisites

### Snapshot repository

Before you can use the backup and restore feature:

1. The [Elasticsearch snapshot repository](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) must be configured.
2. Operate and Tasklist must be configured with the repository name using one of the following configuration options:
2. All deployed webapps must be configured with the **same** repository name using one of the following configuration options.

:::warning breaking change
Configuring different webapps with different repository names will potentially create multiple backups in different repositories
:::

<Tabs groupId="config" defaultValue="yaml" values={
[
Expand Down Expand Up @@ -97,17 +107,56 @@ CAMUNDA_TASKLIST_BACKUP_REPOSITORYNAME=<es snapshot repository name>
</TabItem>
</Tabs>

#### Optimize

<Tabs groupId="config" className="tabs-hidden" defaultValue="yaml" values={
[
{label: 'YAML file', value: 'yaml', },
{label: 'Environment variables', value: 'env', },
]
}>

<TabItem value='yaml'>

In file `environment-config.yaml`:

```yaml
backup:
repositoryName: <repository name>
```
</TabItem>
<TabItem value='env'>
```
CAMUNDA_OPTIMIZE_BACKUP_REPOSITORY_NAME=<es snapshot repository name>
```

</TabItem>
</Tabs>

### Index prefix

::: note breaking change
Since version 8.7 the `indexPrefix` of all webapps must match. By default it's set to `""`, but if it was overriden, then it must done consistently across Operate, Tasklist and Optimize.

:::

## Create backup API

During backup creation Operate can continue running. To create the backup, call the following endpoint:
During backup creation webapps can continue running. To create the backup, call the following endpoint:

```
POST actuator/backups
POST actuator/backupHistory
{
"backupId": <backupId>
}
```

:::note
For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone
:::

Response:

| Code | Description |
Expand All @@ -120,7 +169,7 @@ Response:
Example request:

```shell
curl --request POST 'http://localhost:9600/actuator/backups' \
curl --request POST 'http://localhost:9600/actuator/backupHistory' \
-H 'Content-Type: application/json' \
-d '{ "backupId": 123 }'
```
Expand All @@ -145,9 +194,13 @@ Example response:
As a backup is created asynchronously, call the following endpoint to check the state of the backup:

```
GET actuator/backups/{backupId}
GET actuator/backupHistory/{backupId}
```

:::note
For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone
:::

Response:

| Code | Description |
Expand All @@ -160,7 +213,7 @@ Response:
For example, the request could look like this:

```shell
curl 'http://localhost:9600/actuator/backups/123'
curl 'http://localhost:9600/actuator/backupHistory/123'
```

Example response:
Expand Down Expand Up @@ -198,9 +251,13 @@ Possible **states** of the backup:
To get the list of existing backups, the following endpoint can be used:

```
GET actuator/backups
GET actuator/backupHistory
```

:::note
For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone
:::

Response:

| Code | Description |
Expand All @@ -213,7 +270,7 @@ Response:
For example, the request could look like this:

```shell
curl 'http://localhost:9600/actuator/backups'
curl 'http://localhost:9600/actuator/backupHistory'
```

Response will contain JSON with array of objects representing state of each backup (see [get backup state API endpoint](#get-backup-state-api)).
Expand All @@ -224,10 +281,14 @@ To delete all the Elasticsearch snapshots associated with the specific backup id

```
DELETE actuator/backups/123
DELETE actuator/backupHistory/123
```

:::note
For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone
:::

Response:

| Code | Description |
Expand All @@ -239,10 +300,10 @@ Response:

## Restore backup

There is no Operate API to preform the backup restore. Instead, use the [Elasticsearch restore snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html).
There is no Webapp API to preform the backup restore. Instead, use the [Elasticsearch restore snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html).

:::note
Operate must **not** be running while a backup restore is taking place.
Operate, Tasklist and Optimize must **not** be running while a backup restore is taking place.
:::

To restore the backup with a known backup id, you must restore all the snapshots this backup contains (check the response of the [create backup API](#create-backup-api)).
Expand All @@ -255,11 +316,7 @@ curl --request POST `http://localhost:9200/_snapshot/test/camunda_operate_123_8.

To summarize, the process may look as follows:

1. Stop Operate.
2. Ensure there are no Operate indices present in Elasticsearch (otherwise the restore process will fail).
1. Stop all webapps.
2. Ensure there are no webapp indices present in Elasticsearch (otherwise the restore process will fail).
3. Iterate over all Elasticsearch snapshots included in the desired backup and restore them using the Elasticsearch restore snapshot API.
4. Start Operate.

## Backup and restore of Tasklist data

Backup and restore of Tasklist may be performed in exactly the same way as [Operate data](#).
4. Start all webapps.
3 changes: 1 addition & 2 deletions sidebars.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1044,8 +1044,7 @@ module.exports = {
id: "self-managed/operational-guides/backup-restore/backup-and-restore",
},
items: [
"self-managed/operational-guides/backup-restore/optimize-backup",
"self-managed/operational-guides/backup-restore/operate-tasklist-backup",
"self-managed/operational-guides/backup-restore/webapps-backup",
"self-managed/operational-guides/backup-restore/zeebe-backup-and-restore",
"self-managed/operational-guides/backup-restore/modeler-backup-and-restore",
],
Expand Down

0 comments on commit b18d8db

Please sign in to comment.