docs(self-managed): external-elasticsearch configuration

docs/self-managed/setup/deploy/amazon/amazon-eks/irsa.md

@@ -234,6 +234,12 @@ Don't forget to set the `serviceAccountName` of the deployment/statefulset to th
 As of the 8.4 release, Zeebe, Operate, and Tasklist are now compatible with [Amazon OpenSearch](https://aws.amazon.com/de/opensearch-service/) 2.5.x. Note that using Amazon OpenSearch requires [setting up a new Camunda installation](/self-managed/setup/overview.md). A migration from previous versions or Elasticsearch environments is currently not supported.
 :::
 
+:::caution
+
+Optimize is not supported using the IRSA method. However, Optimize can be utilized by supplying a username and password. The migration step must also be disabled. For more information, refer to [using AWS managed OpenSearch](/self-managed/setup/guides/using-existing-opensearch.md).
+
+:::
+
 ### Setup
 
 For OpenSearch, the most common use case is the use of `fine-grained access control`.
@@ -345,79 +351,28 @@ There are different ways to configure the mapping within OpenSearch:
 
 The important part is assigning the `iam_role_arn` of the previously created `opensearch_role` to an internal role within OpenSearch. For example, `all_access` on the OpenSearch side is a good candidate, or if required, extra roles can be created with more restrictive access.
 
-### Operate
-
-Configure Operate to use the feature set of IRSA for the OpenSearch Exporter. Check the [Operate OpenSearch configuration](../../../../operate-deployment/operate-configuration.md#elasticsearch-or-opensearch).
-
-#### Kubernetes configuration
-
-As an example, configure the following environment variables:
-
-```
-- name: CAMUNDA_OPERATE_OPENSEARCH_URL
-  value: https://test-domain.region.es.amazonaws.com
-- name: CAMUNDA_OPERATE_ZEEBEOPENSEARCH_URL
-  value: https://test-domain.region.es.amazonaws.com
-- name: CAMUNDA_OPERATE_DATABASE
-  value: opensearch
-```
-
-Where the value is whatever the endpoint of your OpenSearch cluster is.
-
-:::note
-AWS OpenSearch listens on port 443 opposed to the usual port 9200.
-:::
-
-:::note
-Don't forget to set the `serviceAccountName` of the deployment/statefulset to the created service account with the IRSA annotation.
-:::
-
-### Tasklist
+### Camunda 8 Self-Managed Helm chart configuration
 
-Configure Tasklist to use the feature set of IRSA for the OpenSearch Exporter. Check the [Tasklist OpenSearch configuration](../../../../tasklist-deployment/tasklist-configuration.md#elasticsearch-or-opensearch).
-
-#### Kubernetes configuration
-
-As an example, configure the following environment variables:
-
-```
-- name: CAMUNDA_TASKLIST_OPENSEARCH_URL
-  value: https://test-domain.region.es.amazonaws.com
-- name: CAMUNDA_TASKLIST_ZEEBEOPENSEARCH_URL
-  value: https://test-domain.region.es.amazonaws.com
-- name: CAMUNDA_TASKLIST_DATABASE
-  value: opensearch
-```
-
-Where the value is whatever the endpoint of your OpenSearch cluster is.
-
-:::note
-AWS OpenSearch listens on port 443 opposed to the usual port 9200.
-:::
-
-:::note
-Don't forget to set the `serviceAccountName` of the deployment/statefulset to the created service account with the IRSA annotation.
-:::
-
-### Zeebe
-
-Configure Zeebe to use the feature set of IRSA for the OpenSearch Exporter. Check the [Zeebe OpenSearch exporter configuration](../../../../zeebe-deployment/configuration/broker.md#zeebebrokerexportersopensearch-opensearch-exporter).
-
-#### Kubernetes configuration
-
-As an example, configure the following environment variables:
+The following is an example configuration that can be used to configure the Camunda 8 Self-Managed Helm chart to use the feature set of IRSA for the OpenSearch Exporter:
 
 ```yaml
-- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_AWS_ENABLED
-  value: "true"
-- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_CLASSNAME
-  value: io.camunda.zeebe.exporter.opensearch.OpensearchExporter
-- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_URL
-  value: https://test-domain.region.es.amazonaws.com
-- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_BULK_SIZE
-  value: "1"
-- name: ZEEBE_BROKER_EXPORTERS_OPENSEARCH_ARGS_INDEX_PROCESSMESSAGESUBSCRIPTION
-  value: "true"
+global:
+  elasticsearch:
+    enabled: false
+  opensearch:
+    enabled: true
+    aws:
+      enabled: true
+    url:
+      protocol: https
+      host: aws.opensearch.example.com
+      port: 443
+
+elasticsearch:
+  enabled: false
+
+optimize:
+  enabled: false
 ```
 
 :::note

docs/self-managed/setup/guides/using-existing-elasticsearch.md

@@ -0,0 +1,109 @@
+---
+id: using-existing-elasticsearch
+title: "Using existing Elasticsearch"
+description: "Learn how to use an existing Elasticsearch instance in Camunda 8 Self-Managed Helm chart deployment."
+---
+
+By default, the [Helm chart deployment](/self-managed/setup/install.md) creates a new Elasticsearch instance, but it's possible to use an existing Elasticsearch instance either inside the same Kubernetes cluster or outside of it. This guide steps through using an existing Elasticsearch instance.
+
+## Connecting to existing Elasticsearch without a certificate
+
+By default, `global.elasticsearch.url.protocol` is set to `http`. This makes it possible to connect to Elasticsearch through `http`.
+
+The following information must be known relating to the Self-Managed Elasticsearch cluster:
+
+- Protocol, host, port
+- Username and password
+
+The Camunda 8 Self-Managed Helm chart can then be configured as follows:
+
+```yaml
+global:
+  elasticsearch:
+    enabled: true
+    external: true
+    auth:
+      username: elastic
+      password: pass
+    url:
+      protocol: http
+      host: elastic.example.com
+      port: 443
+
+elasticsearch:
+  enabled: false
+```
+
+## Connecting to existing Elasticsearch with a self-signed certificate
+
+If a self-signed certificate is used and only `https` requests are accepted in the Elasticsearch cluster, then the following steps can be applied:
+
+1. Create an `externaldb.jks` file from your Elasticsearch certificate file. Here is an example of that, using the `keytool` CLI:
+
+```yaml
+keytool -import -alias elasticsearch -keystore externaldb.jks -storetype jks -file elastic.crt -storepass changeit -noprompt
+```
+
+2. Create a Kubernetes secret from the `externaldb.jks` file before installing Camunda. This is how you can create the secret:
+
+```yaml
+kubectl  create secret -n camunda generic elastic-jks --from-file=externaldb.jks
+```
+
+The Camunda 8 Self-Managed Helm chart can then be configured as follows:
+
+```yaml
+global:
+  elasticsearch:
+    enabled: true
+    external: true
+    tls:
+      enabled: true
+      existingSecret: elastic-jks
+    auth:
+      username: elastic
+      password: pass
+    url:
+      protocol: https
+      host: elastic.example.com
+      port: 443
+
+elasticsearch:
+  enabled: false
+```
+
+## Connecting to existing Elasticsearch with a publicly trusted certificate
+
+This configuration should work with any managed Elasticsearch. We have specifically tested this configuration using Elastic Cloud on Google Cloud.
+
+The following information must be known relating to the Elasticsearch cluster:
+
+- Protocol, host, port
+- Username and password
+
+The Camunda 8 Self-Managed Helm chart can then be configured as follows:
+
+```yaml
+global:
+  elasticsearch:
+    enabled: true
+    external: true
+    auth:
+      username: elastic
+      password: pass
+    url:
+      protocol: https
+      host: elastic.example.com
+      port: 443
+
+elasticsearch:
+  enabled: false
+```
+
+## Next steps
+
+Use the custom values file to [deploy Camunda 8](/self-managed/setup/overview.md) as usual:
+
+```sh
+helm install camunda camunda/camunda-platform -f existing-elasticsearch-values.yaml
+```

docs/self-managed/setup/guides/using-existing-opensearch.md

@@ -0,0 +1,71 @@
+---
+id: using-existing-opensearch
+title: "Using AWS managed OpenSearch"
+description: "Learn how to use an AWS managed OpenSearch instance in Camunda 8 Self-Managed deployment."
+---
+
+Camunda 8 Self-Managed has two different types of components: Camunda components (Operate, Optimize, Tasklist, etc.) and non-Camunda dependency components (such as Keycloak and Elasticsearch). For more details, review the [architecture](/self-managed/platform-architecture/overview.md) documentation for more information on the different types of applications.
+
+This guide steps through using an existing AWS managed OpenSearch instance. By default, [Helm chart deployment](/self-managed/setup/overview.md) creates a new Elasticsearch instance, but it's possible to use AWS managed OpenSearch instead.
+
+## Preparation
+
+### Authentication
+
+There are two layers of permissions with OpenSearch: AWS IAM and OpenSearch internal. If you would like to connect to OpenSearch using AWS IAM roles for service accounts (IRSA) then please also refer to the [IAM roles for service accounts documentation](/self-managed/setup/deploy/amazon/amazon-eks/irsa.md#OpenSearch).
+
+Otherwise, if it is intended to connect to AWS managed OpenSearch with basic auth, then the example below can be followed:
+
+## Values file
+
+:::caution
+
+The migration step within Optimize is currently not supported with OpenSearch. Disable the migration as shown in the example below.
+
+:::
+
+The following values can be configured in the Camunda 8 Self-Managed Helm chart in order to use AWS managed OpenSearch:
+
+### Connecting to AWS managed OpenSearch with basic auth
+
+```yaml
+global:
+  elasticsearch:
+    enabled: false
+  opensearch:
+    enabled: true
+    auth:
+      username: user
+      password: pass
+    url:
+      protocol: https
+      host: opensearch.example.com
+      port: 443
+
+optimize:
+  migration:
+    enabled: false
+
+elasticsearch:
+  enabled: false
+```
+
+From the above configuration, the internal Elasticsearch component and the Elasticsearch configuration for all components are disabled. This is required to use AWS managed OpenSearch.
+
+If you do not wish to specify the username and password in plaintext within the `values.yaml` file, then the following configuration can be used:
+
+```yaml
+global:
+  opensearcn:
+    auth:
+      existingSecret: secretName
+      existingSecretKey: secretKey
+```
+
+## Next steps
+
+Use the custom values file to [deploy Camunda 8](/self-managed/setup/overview.md) as usual:
+
+```sh
+helm install camunda camunda/camunda-platform -f existing-elasticsearch-values.yaml
+```

docs/self-managed/setup/upgrade.md

@@ -98,7 +98,10 @@ For more details on the Keycloak upgrade path, you can also read the [Bitnami Ke
 Camunda Release Cycle: 8.5
 
 :::caution Breaking changes
-The Camunda Helm chart v10.0.0 has major changes in the values file structure. Follow the upgrade steps for each component before starting the chart upgrade.
+
+- The Camunda Helm chart v10.0.0 has major changes in the values file structure. Follow the upgrade steps for each component before starting the chart upgrade.
+- The Elasticsearch configuration has changed to support external Elasticsearch.
+
 :::
 
 #### Identity
@@ -181,6 +184,35 @@ New:
 zeebeGateway:
 ```
 
+#### Enabling external Elasticsearch
+
+It is possible to use external Elasticsearch. For more information on how to set up external Elasticsearch, refer to [using existing Elasticsearch](./guides/using-existing-elasticsearch.md).
+
+##### Elasticsearch - values file
+
+The `global.elasticsearch.disableExporter` field has been deprecated in favor of `global.elasticsearch.enabled`. When `global.elasticsearch.enabled` is set to false, all configurations for Elasticsearch in all components are removed.
+
+The `global.elasticsearch.url` field has changed. If you are using the default `values.yaml` and have not configured the URL, no change is required. However, if the URL value is used, then instead of specifying a single URL, you must now explicitly specify the protocol, host, and port separately like so:
+
+```yaml
+global:
+  elasticsearch:
+    url:
+      protocol: https
+      host: example.elasticsearch.com
+      port: 443
+```
+
+Because of this change to the `global.elasticsearch.url` value, the following values have been removed:
+
+1. `global.elasticsearch.protocol`
+2. `global.elasticsearch.host`
+3. `global.elasticsearch.port`
+
+#### Enabling external AWS managed OpenSearch
+
+It is possible to use external AWS managed OpenSearch. For more information on how to set up external AWS managed OpenSearch, refer to [using AWS managed OpenSearch](./guides/using-existing-opensearch.md).
+
 ### v9.3.0
 
 Camunda Release Cycle: 8.4

optimize_sidebars.js

@@ -1834,6 +1834,14 @@ module.exports = {
               "Using existing Keycloak",
               "self-managed/setup/guides/using-existing-keycloak/"
             ),
+            docsLink(
+              "Using existing Elasticsearch",
+              "self-managed/setup/guides/using-existing-elasticsearch/"
+            ),
+            docsLink(
+              "Using AWS managed OpenSearch",
+              "self-managed/setup/guides/using-existing-opensearch/"
+            ),
             docsLink(
               "Connect to an OpenID Connect provider",
               "self-managed/setup/guides/connect-to-an-oidc-provider/"

sidebars.js

@@ -874,6 +874,8 @@ module.exports = {
             "self-managed/setup/guides/accessing-components-without-ingress",
             "self-managed/setup/guides/ingress-setup",
             "self-managed/setup/guides/using-existing-keycloak",
+            "self-managed/setup/guides/using-existing-elasticsearch",
+            "self-managed/setup/guides/using-existing-opensearch",
             "self-managed/setup/guides/connect-to-an-oidc-provider",
             "self-managed/setup/guides/air-gapped-installation",
             "self-managed/setup/guides/running-custom-connectors",