From 4bd252d707bf4b67444fb81f3d7ab129704426e0 Mon Sep 17 00:00:00 2001 From: Carlo Sana Date: Wed, 18 Dec 2024 14:09:45 +0100 Subject: [PATCH] feat: document new backup/restore procedure and webapps specific config relates #4790 --- .../backup-restore/backup-and-restore.md | 26 +-- .../backup-restore/optimize-backup.md | 189 ------------------ ...e-tasklist-backup.md => webapps-backup.md} | 120 ++++++++--- sidebars.js | 3 +- 4 files changed, 100 insertions(+), 238 deletions(-) delete mode 100644 docs/self-managed/operational-guides/backup-restore/optimize-backup.md rename docs/self-managed/operational-guides/backup-restore/{operate-tasklist-backup.md => webapps-backup.md} (69%) diff --git a/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md b/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md index c5d68a21004..3353b7936a9 100644 --- a/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md +++ b/docs/self-managed/operational-guides/backup-restore/backup-and-restore.md @@ -6,7 +6,7 @@ keywords: ["backup", "backups"] --- :::note -This release introduces breaking changes for [Operate and Tasklist](./operate-tasklist-backup.md), as well as [Optimize](./optimize-backup.md). +This release introduces breaking changes for [Operate, Tasklist and Optimize](./webapps-backup.md). ::: You can use the backup feature of Camunda 8 Self-Managed to regularly back up the state of all of its components (Zeebe, Operate, Tasklist, and Optimize) without any downtime (except Web Modeler, see [the Web Modeler backup and restore documentation](./modeler-backup-and-restore.md)). @@ -24,7 +24,7 @@ Zeebe stores its backup to an external storage and must be configured before the ### Backup process -The backup of each component and the backup of a Camunda 8 cluster is identified by an id. This means a backup `x` of Camunda 8 consists of backup `x` of Zeebe, backup `x` of Optimize, backup `x` of Operate, and backup `x` of Tasklist. The backup ID must be an integer and greater than the previous backups. +The backup of each component and the backup of a Camunda 8 cluster is identified by an id. This means a backup `x` of Camunda 8 consists of backup `x` of Zeebe, backup `x` of Webapps(which includes Optimize, Operate and Tasklist). The backup ID must be an integer and greater than the previous backups. :::note We recommend using the timestamp as the backup id. @@ -32,14 +32,10 @@ We recommend using the timestamp as the backup id. To back up a Camunda 8 cluster, execute the following sequential steps: -1. Trigger a backup `x` of Optimize. See [how to take an Optimize backup](/self-managed/operational-guides/backup-restore/optimize-backup.md). -2. Trigger a backup `x` of Operate. See [how to take an Operate backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md). -3. Trigger a backup `x` of Tasklist. See [how to take a Tasklist backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md). -4. Wait until the backup `x` of Optimize is complete. See [how to monitor an Optimize backup](/self-managed/operational-guides/backup-restore/optimize-backup.md). -5. Wait until the backup `x` of Operate is complete. See [how to monitor an Operate backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md). -6. Wait until the backup `x` of Tasklist is complete. See [how to monitor a Tasklist backup](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md). -7. Soft pause exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md). -8. Take a backup `x` of the exported Zeebe records in Elasticsearch using the Elasticsearch Snapshots API. +1. Soft pause exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md). +2. Trigger a backup `x` of Webapps. See [how to take a Webapp backup](/self-managed/operational-guides/backup-restore/webapps-backup.md). +3. Wait until the backup `x` of Webapps is complete. See [how to monitor an Webapp backup](/self-managed/operational-guides/backup-restore/webapps-backup.md). +4. Take a backup `x` of the exported Zeebe records in Elasticsearch using the Elasticsearch Snapshots API. ``` @@ -53,10 +49,10 @@ PUT /_snapshot/my_repository/camunda_zeebe_records_backup_x By default, the indices are prefixed with `zeebe-record`. If you have configured a different prefix when configuring Elasticsearch exporter in Zeebe, use this instead. -9. Wait until the backup `x` of the exported Zeebe records is complete before proceeding. - Take a backup `x` of Zeebe. See [how to take a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). -10. Wait until the backup `x` of Zeebe is completed before proceeding. See [how to monitor a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). - Resume exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md). +5. Wait until the backup `x` of the exported Zeebe records is complete before proceeding. +6. Take a backup `x` of Zeebe. See [how to take a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). +7. Wait until the backup `x` of Zeebe is completed before proceeding. See [how to monitor a Zeebe backup](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). +8. Resume exporting in Zeebe. See [Zeebe management API](/self-managed/zeebe-deployment/operations/management-api.md). :::note If any of the steps above fail, you may have to restart with a new backup id. Ensure exporting is resumed if the backup process force quits in the middle of the process. @@ -70,7 +66,7 @@ To restore a Camunda 8 cluster from a backup, all components must be restored fr 2. Confirm proper configuration (such as shards, replicas count, etc.) 3. Stop Operate, Tasklist, and Optimize. 4. Delete all indices. -5. Restore the state of [Operate](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md), [Tasklist](/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md), and [Optimize](/self-managed/operational-guides/backup-restore/optimize-backup.md). +5. Restore the state of [Webapps](/self-managed/operational-guides/backup-restore/webapps-backup.md). 6. Restore `zeebe-records*` indices from Elasticsearch snapshot. 7. Restore [Zeebe](/self-managed/operational-guides/backup-restore/zeebe-backup-and-restore.md). 8. Start Zeebe, Operate, Tasklist, and Optimize. diff --git a/docs/self-managed/operational-guides/backup-restore/optimize-backup.md b/docs/self-managed/operational-guides/backup-restore/optimize-backup.md deleted file mode 100644 index 2215873da0d..00000000000 --- a/docs/self-managed/operational-guides/backup-restore/optimize-backup.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -id: optimize-backup -title: Backup and restore Optimize data -description: "How to perform a backup of Optimize data and restore the backup." -keywords: ["backup", "backups"] ---- - -:::note -This release introduces breaking changes, including the utilized URL. - -For example, `curl 'http://localhost:8092/actuator/backups'` rather than the previously used `backup`. -::: - -Optimize stores its data over multiple indices in the database. To ensure data integrity across indices, a backup of Optimize data consists of two ElasticSearch/OpenSearch snapshots, each containing a different set of Optimize indices. Each backup is identified by a positive integer backup ID. For example, a backup with ID `123456` consists of the following snapshots: - -``` -camunda_optimize_123456_3.9.0_part_1_of_2 -camunda_optimize_123456_3.9.0_part_2_of_2 -``` - -Optimize provides an API to trigger a backup and retrieve information about a given backup's state. During backup creation Optimize can continue running. The backed up data can later be restored using the standard ElasticSearch/OpenSearch snapshot restore API. - -## Prerequisites - -The following prerequisites must be set up before using the backup API: - -1. A snapshot repository of your choice must be registered with ElasticSearch/OpenSearch. -2. The repository name must be specified using the `CAMUNDA_OPTIMIZE_BACKUP_REPOSITORY_NAME` environment variable, or by adding it to your Optimize [`environment-config.yaml`]($optimize$/self-managed/optimize-deployment/configuration/system-configuration/): - -```yaml -backup: - repositoryName: -``` - -## Create backup API - -Note that the backup API can be reached via the `/actuator` management port, which by default is `8092`. -The configured context path does not apply to the management port. - -The following endpoint can be used to trigger the backup process: - -``` -POST actuator/backups -{ - "backupId": -} -``` - -### Response - -| Code | Description | -| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -| 202 Accepted | Backup process was successfully initiated. To determine whether backup process was completed refer to the GET API. | -| 400 Bad Request | Indicates issues with the request, for example when the `backupId` contains invalid characters. | -| 409 Conflict | Indicates that a backup with the same `backupId` already exists. | -| 500 Server Error | All other errors, e.g. issues communicating with the database for snapshot creation. Refer to the returned error message for more details. | -| 502 Bad Gateway | Optimize has encountered issues while trying to connect to the database. | - -### Example request - -```shell -curl --request POST 'http://localhost:8092/actuator/backups' \ --H 'Content-Type: application/json' \ --d '{ "backupId": 123456 }' -``` - -### Example response - -```json -{ - "message": "Backup creation for ID 123456 has been scheduled. Use the GET API to monitor completion of backup process" -} -``` - -## Get backup info API - -Note that the backup API can be reached via the `/actuator` management port, which by default is `8092`. -The configured context path does not apply to the management port. - -Information about a specific backup can be retrieved using the following request: - -``` -GET actuator/backups/{backupId} -``` - -Information about all existing Optimize backups can be retrieved by omitting the optional `backupId` parameter: - -``` -GET actuator/backup -``` - -### Response - -| Code | Description | -| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 200 OK | Backup state could be determined and is returned in the response body (see example below). | -| 400 Bad Request | There is an issue with the request, for example the repository name specified in the Optimize configuration does not exist. Refer to returned error message for details. | -| 404 Not Found | If a backup ID was specified, no backup with that ID exists. | -| 500 Server Error | All other errors, e.g. issues communicating with the database for snapshot state retrieval. Refer to the returned error message for more details. | -| 502 Bad Gateway | Optimize has encountered issues while trying to connect to the database. | - -### Example request - -```shell -curl --request GET 'http://localhost:8092/actuator/backups/123456' -``` - -### Example response - -```json - { - "backupId": 123456, - "failureReason": null, - "state": "COMPLETE", - “details”: [ - { - "snapshotName": "camunda_optimize_123456_3.10.0_part_1_of_2", - "state": "SUCCESS", - "startTime": "2022-11-09T10:11:36.978+0100", - "failures": [] - }, - { - "snapshotName": "camunda_optimize_123456_3.10.0_part_2_of_2", - "state": "SUCCESS", - "startTime": "2022-11-09T10:11:37.178+0100", - "failures": [] - } - ] - } -``` - -Note that the endpoint will return a single item when called with a `backupId` and a list of items when called without specifying a `backupId`. - -Possible states of the backup: - -- `COMPLETE`: The backup can be used for restoring data. -- `IN_PROGRESS`: The backup process for this backup ID is still in progress. -- `FAILED`: Something went wrong when creating this backup. To find out the exact problem, use the [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-snapshot-status-api.html) / [OpenSearch](https://opensearch.org/docs/latest/api-reference/snapshots/get-snapshot-status/) get snapshot status API for each of the snapshots included in the given backup. -- `INCOMPATIBLE`: The backup is incompatible with the current ElasticSearch/OpenSearch version. -- `INCOMPLETE`: The backup is incomplete (this could occur when the backup process was interrupted or individual snapshots were deleted). - -## Delete backup API - -Note that the backup API can be reached via the `/actuator` management port, which by default is `8092`. -The configured context path does not apply to the management port. - -An existing backup can be deleted using the below API which deletes all Optimize snapshots associated with the supplied backupID. - -``` -DELETE actuator/backups/{backupId} -``` - -### Response - -| Code | Description | -| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 204 No Content | The delete request for the associated snapshots was submitted to the database successfully. | -| 400 Bad Request | There is an issue with the request, for example the repository name specified in the Optimize configuration does not exist. Refer to returned error message for details. | -| 500 Server Error | An error occurred, for example the snapshot repository does not exist. Refer to the returned error message for details. | -| 502 Bad Gateway | Optimize has encountered issues while trying to connect to ElasticSearch/OpenSearch. | - -### Example request - -```shell -curl --request DELETE 'http://localhost:8092/actuator/backups/123456' -``` - -## Restore backup - -There is no Optimize API to perform the backup restore. Instead, the standard [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html) / [OpenSearch](https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot) restore snapshot API can be used. Note that the Optimize versions of your backup snapshots must match the currently running version of Optimize. You can identify the version at which the backup was taken by the version tag included in respective snapshot names; for example, a snapshot with the name`camunda_optimize_123456_3.9.0_part_1_of_2` was taken of Optimize version `3.9.0`. - -:::note -Optimize must NOT be running while a backup is being restored. -::: - -To restore an existing backup, all the snapshots this backup contains (as listed in the response of the [create backup API request](#example-response)) must be restored using the restore API. - -To restore a given backup, the following steps must be performed: - -1. Stop Optimize. -2. Ensure no Optimize indices are present in the database (or the restore process will fail). -3. Iterate over all ElasticSearch/OpenSearch snapshots included in the desired backup and restore them using the restore snapshot API mentioned above. -4. Start Optimize. - -Example request: - -```shell -curl --request POST `http://localhost:9200/_snapshot/repository_name/camunda_optimize_123456_3.9.0_part_1_of_2/_restore?wait_for_completion=true` -``` diff --git a/docs/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md b/docs/self-managed/operational-guides/backup-restore/webapps-backup.md similarity index 69% rename from docs/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md rename to docs/self-managed/operational-guides/backup-restore/webapps-backup.md index eb3d1a568f0..ed028b3fbbc 100644 --- a/docs/self-managed/operational-guides/backup-restore/operate-tasklist-backup.md +++ b/docs/self-managed/operational-guides/backup-restore/webapps-backup.md @@ -1,30 +1,34 @@ --- -id: operate-tasklist-backup -title: Backup and restore Operate and Tasklist data -description: "How to perform a backup and restore of Operate and Tasklist data." +id: webapps-backup +title: Backup and restore Webapps (Operate, Tasklist and Optimize) data +description: "How to perform a backup and restore of Webapps (Operate, Tasklist and Optimize) data." keywords: ["backup", "backups"] --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; -:::warning breaking changes! -As of the Camunda 8.6 release, the `/actuator` endpoints (including `/backups`) now default to port 9600. Ensure your `management.server.port` configuration parameter is correctly set before continuing. +:::warning breaking changes +As of the Camunda 8.7 release, the `/actuator` endpoints for backups have been moved to `/actuator/backupHistory`. The old `/actuator/backups` endpoint is still active only if the applications are deployed standalone (i.e. each application in its own process) ::: -Operate stores its data over multiple indices in Elasticsearch. Backup of Operate data includes several -Elasticsearch snapshots containing sets of Operate indices. Each backup is identified by `backupId`. For example, a backup with an ID of `123` may contain the following Elasticsearch snapshots: +:::note +In this page we will refer to "webapps" as Operate, Tasklist and Optimize. Depending on your deployment configuration, you may not have all of them deployed. If that is the case, you can safely ignore the configurations for those applications. +::: + +Webapps store their data over multiple indices in Elasticsearch. Backup of Webapps data includes several +Elasticsearch snapshots containing sets of different indices. Each backup is identified by `backupId`. For example, a backup with an ID of `123` may contain the following Elasticsearch snapshots: ``` -camunda_operate_123_8.1.0_part_1_of_6 -camunda_operate_123_8.1.0_part_2_of_6 -camunda_operate_123_8.1.0_part_3_of_6 -camunda_operate_123_8.1.0_part_4_of_6 -camunda_operate_123_8.1.0_part_5_of_6 -camunda_operate_123_8.1.0_part_6_of_6 +camunda_webapps_123_8.1.0_part_1_of_6 +camunda_webapps_123_8.1.0_part_2_of_6 +camunda_webapps_123_8.1.0_part_3_of_6 +camunda_webapps_123_8.1.0_part_4_of_6 +camunda_webapps_123_8.1.0_part_5_of_6 +camunda_webapps_123_8.1.0_part_6_of_6 ``` -Operate provides an API to perform a backup and manage backups (list, check state, delete). Restore a backup using the standard Elasticsearch API. +All webapps provides the same API to perform a backup and manage backups (list, check state, delete). Restore a backup using the standard Elasticsearch API. :::note The backup API can be reached via the Actuator management port, which by default is the same as application HTTP port (and in turn defaults to 9600). The port may be reconfigured with the help of `management.server.port` configuration parameter. @@ -32,10 +36,16 @@ The backup API can be reached via the Actuator management port, which by default ## Prerequisites +### Snapshot repository + Before you can use the backup and restore feature: 1. The [Elasticsearch snapshot repository](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) must be configured. -2. Operate and Tasklist must be configured with the repository name using one of the following configuration options: +2. All deployed webapps must be configured with the **same** repository name using one of the following configuration options. + +:::warning breaking change +Configuring different webapps with different repository names will potentially create multiple backups in different repositories +::: +#### Optimize + + + + + +In file `environment-config.yaml`: + +```yaml +backup: + repositoryName: +``` + + + + +``` +CAMUNDA_OPTIMIZE_BACKUP_REPOSITORY_NAME= +``` + + + + +### Index prefix + +:::warning breaking change +Since version 8.7 the `indexPrefix` of all webapps must match. By default it's set to `""`, but if it was overriden, then it must done consistently across Operate, Tasklist and Optimize. +::: + ## Create backup API -During backup creation Operate can continue running. To create the backup, call the following endpoint: +During backup creation webapps can continue running. To create the backup, call the following endpoint: ``` -POST actuator/backups +POST actuator/backupHistory { "backupId": } ``` +:::note +For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone +::: + Response: | Code | Description | @@ -120,7 +168,7 @@ Response: Example request: ```shell -curl --request POST 'http://localhost:9600/actuator/backups' \ +curl --request POST 'http://localhost:9600/actuator/backupHistory' \ -H 'Content-Type: application/json' \ -d '{ "backupId": 123 }' ``` @@ -145,9 +193,13 @@ Example response: As a backup is created asynchronously, call the following endpoint to check the state of the backup: ``` -GET actuator/backups/{backupId} +GET actuator/backupHistory/{backupId} ``` +:::note +For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone +::: + Response: | Code | Description | @@ -160,7 +212,7 @@ Response: For example, the request could look like this: ```shell -curl 'http://localhost:9600/actuator/backups/123' +curl 'http://localhost:9600/actuator/backupHistory/123' ``` Example response: @@ -198,9 +250,13 @@ Possible **states** of the backup: To get the list of existing backups, the following endpoint can be used: ``` -GET actuator/backups +GET actuator/backupHistory ``` +:::note +For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone +::: + Response: | Code | Description | @@ -213,7 +269,7 @@ Response: For example, the request could look like this: ```shell -curl 'http://localhost:9600/actuator/backups' +curl 'http://localhost:9600/actuator/backupHistory' ``` Response will contain JSON with array of objects representing state of each backup (see [get backup state API endpoint](#get-backup-state-api)). @@ -224,10 +280,14 @@ To delete all the Elasticsearch snapshots associated with the specific backup id ``` -DELETE actuator/backups/123 +DELETE actuator/backupHistory/123 ``` +:::note +For backward compatibility, the endpoint `actuator/backups` is available if the app is running standalone +::: + Response: | Code | Description | @@ -239,10 +299,10 @@ Response: ## Restore backup -There is no Operate API to preform the backup restore. Instead, use the [Elasticsearch restore snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html). +There is no Webapp API to preform the backup restore. Instead, use the [Elasticsearch restore snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html). :::note -Operate must **not** be running while a backup restore is taking place. +Operate, Tasklist and Optimize must **not** be running while a backup restore is taking place. ::: To restore the backup with a known backup id, you must restore all the snapshots this backup contains (check the response of the [create backup API](#create-backup-api)). @@ -255,11 +315,7 @@ curl --request POST `http://localhost:9200/_snapshot/test/camunda_operate_123_8. To summarize, the process may look as follows: -1. Stop Operate. -2. Ensure there are no Operate indices present in Elasticsearch (otherwise the restore process will fail). +1. Stop all webapps. +2. Ensure there are no webapp indices present in Elasticsearch (otherwise the restore process will fail). 3. Iterate over all Elasticsearch snapshots included in the desired backup and restore them using the Elasticsearch restore snapshot API. -4. Start Operate. - -## Backup and restore of Tasklist data - -Backup and restore of Tasklist may be performed in exactly the same way as [Operate data](#). +4. Start all webapps. diff --git a/sidebars.js b/sidebars.js index 34f89302958..b0e9bbc4461 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1044,8 +1044,7 @@ module.exports = { id: "self-managed/operational-guides/backup-restore/backup-and-restore", }, items: [ - "self-managed/operational-guides/backup-restore/optimize-backup", - "self-managed/operational-guides/backup-restore/operate-tasklist-backup", + "self-managed/operational-guides/backup-restore/webapps-backup", "self-managed/operational-guides/backup-restore/zeebe-backup-and-restore", "self-managed/operational-guides/backup-restore/modeler-backup-and-restore", ],