Skip to content

Commit

Permalink
docs(Optimize): Technical documentation for Optimize with OpenSearch (#…
Browse files Browse the repository at this point in the history 
…3564)

* Technical documentation for Optimize with OpenSearch

* doc(optimize): Corrected one finding from review

* style(formatting): technical review

* doc(optimize): Implementing suggestions from Review

* doc(optimize): Implementing suggestions from technical review

---------

Co-authored-by: Christina Ausley <[email protected]>
  • Loading branch information
@grlimacan @christinaausley
grlimacan and christinaausley authored Apr 5, 2024
1 parent 36bf1e8 commit 5252446
Show file tree
Hide file tree
Showing 6 changed files with 187 additions and 34 deletions.
2 changes: 1 addition & 1 deletion docs/reference/supported-environments.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ Requirements for the components can be seen below:
| Operate | OpenJDK 17+ | Elasticsearch 8.9+<br/>Amazon OpenSearch 2.5.x |
| Tasklist | OpenJDK 17+ | Elasticsearch 8.9+<br/>Amazon OpenSearch 2.5.x |
| Identity | OpenJDK 17+ | Keycloak 22.x, 23.x<br/>PostgreSQL 14.x, 15.x or Amazon Aurora PostgreSQL 13.x, 14.x, 15.x (required for [certain features](/self-managed/identity/deployment/configuration-variables.md#database-configuration)) |
| Optimize | OpenJDK 17+ | Elasticsearch 8.9+ |
| Optimize | OpenJDK 17+ | Elasticsearch 8.9+<br/>Amazon OpenSearch 2.5.x |
| Connectors | OpenJDK 21+ | |
| Web Modeler | - | PostgreSQL 13.x, 14.x, 15.x, 16.x or Amazon Aurora PostgreSQL 13.x, 14.x, 15.x, 16.x |

Expand Down
24 changes: 18 additions & 6 deletions docs/self-managed/platform-deployment/docker.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -131,16 +131,21 @@ Some configuration properties are optional and have default values. See a descri
| Name | Description | Default value |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- |
| SPRING_PROFILES_ACTIVE | Determines the mode Optimize is to be run in. For Self-Managed, set to `ccsm`. |
| CAMUNDA_OPTIMIZE_DATABASE | Determines the database Optimize will use. Allowed values: `elasticsearch` or `opensearch` | elasticsearch |
| CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_URL | The URL at which Identity can be accessed by Optimize. |
| CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_BACKEND_URL | The URL at which the Identity auth provider can be accessed by Optimize. This should match the configured provider in Identity and is to be used for container to container communication. |
| CAMUNDA_OPTIMIZE_IDENTITY_CLIENTID | The Client ID used to register Optimize with Identity. |
| CAMUNDA_OPTIMIZE_IDENTITY_CLIENTSECRET | The secret used when registering Optimize with Identity. |
| CAMUNDA_OPTIMIZE_IDENTITY_AUDIENCE | The audience used when registering Optimize with Identity. |
| OPTIMIZE_ELASTICSEARCH_HOST | The address/hostname under which the Elasticsearch node is available. | localhost |
| OPTIMIZE_ELASTICSEARCH_HTTP_PORT | The port number used by Elasticsearch to accept HTTP connections. | 9200 |
| OPTIMIZE_ELASTICSEARCH_HOST\* | The address/hostname under which the Elasticsearch node is available. | localhost |
| OPTIMIZE_ELASTICSEARCH_HTTP_PORT\* | The port number used by Elasticsearch to accept HTTP connections. | 9200 |
| CAMUNDA_OPTIMIZE_OPENSEARCH_HOST\*\* | The address/hostname under which the OpenSearch node is available. | localhost |
| CAMUNDA_OPTIMIZE_OPENSEARCH_HTTP_PORT \*\* | The port number used by OpenSearch to accept HTTP connections. | 9205 |
| CAMUNDA_OPTIMIZE_SECURITY_AUTH_COOKIE_SAME_SITE_ENABLED | Determines if `same-site` is enabled for Optimize cookies. This must be set to `false`. | true |
| CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_USERNAME | The username for authentication in environments where a secured Elasticsearch connection is configured. |
| CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_PASSWORD | The password for authentication in environments where a secured Elasticsearch connection is configured. |
| CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_USERNAME \* | The username for authentication in environments where a secured Elasticsearch connection is configured. |
| CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_PASSWORD \* | The password for authentication in environments where a secured Elasticsearch connection is configured. |
| CAMUNDA_OPTIMIZE_OPENSEARCH_SECURITY_USERNAME\*\* | The username for authentication in environments where a secured OpenSearch connection is configured. |
| CAMUNDA_OPTIMIZE_OPENSEARCH_SECURITY_PASSWORD\*\* | The password for authentication in environments where a secured OpenSearch connection is configured. |
| CAMUNDA_OPTIMIZE_ENTERPRISE | This should only be set to `true` if an Enterprise License has been acquired. | true |
| CAMUNDA_OPTIMIZE_ZEEBE_ENABLED | Enables import of Zeebe data in Optimize. | false |
| CAMUNDA_OPTIMIZE_ZEEBE_NAME | The record prefix for exported Zeebe records. | zeebe-record |
Expand All @@ -149,6 +154,13 @@ Some configuration properties are optional and have default values. See a descri
| SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI | Authentication for the Public REST API using a resource server to validate the JWT token. Complete URI to get public keys for JWT validation. | null |
| OPTIMIZE_API_ACCESS_TOKEN | Authentication for the Public REST API using a static shared token. Will be ignored if SPRING_SECURITY_OAUTH2_RESOURCESERVER_JWT_JWK_SET_URI is also set. | null |

\* Only relevant when `CAMUNDA_OPTIMIZE_DATABASE` is either undefined or has the value `elasticsearch`.<br/>
\*\* Only relevant when `CAMUNDA_OPTIMIZE_DATABASE` has the value `opensearch`.

:::note
OpenSearch support in Optimize is limited to data import and the raw data report. The remaining functionality will be delivered with upcoming patches.
:::

Like for example this `docker-compose` configuration:

```
Expand Down Expand Up @@ -176,12 +188,12 @@ optimize:
- OPTIMIZE_API_ACCESS_TOKEN=secret
```

Self-Managed Optimize must be able to connect to Elasticsearch to write and read data. In addition, Optimize needs to connect to Identity for authentication purposes. Both of these requirements can be configured with the options described above.
Self-Managed Optimize must be able to connect to the configured database to write and read data. In addition, Optimize needs to connect to Identity for authentication purposes. Both of these requirements can be configured with the options described above.

Optimize must also be configured as a client in Identity, and users will only be granted access to Optimize if they have a role
that has `write:*` permission for Optimize.

For Optimize to import Zeebe data, Optimize must also be configured to be aware of the record prefix used when the records are exported to Elasticsearch. This can also be configured per the example above.
For Optimize to import Zeebe data, Optimize must also be configured to be aware of the record prefix used when the records are exported to the database. This can also be configured per the example above.

### Connectors

Expand Down
4 changes: 2 additions & 2 deletions optimize/self-managed/optimize-deployment/configuration/getting-started.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ You can see all supported values and read about logging configuration [here](./s

Refer to the [configuration section on container settings](./system-configuration.md) for more information on how to adjust the Optimize web container configuration.

## Elasticsearch configuration
## Elasticsearch/OpenSearch configuration

You can customize the [Elasticsearch connection settings](./system-configuration.md#connection-settings) as well as the [index settings](./system-configuration.md#index-settings).
You can customize the [Elasticsearch/OpenSearch connection settings](./system-configuration.md#connection-settings) as well as the [index settings](./system-configuration.md#index-settings).

## Camunda 7 configuration

Expand Down
73 changes: 73 additions & 0 deletions optimize/self-managed/optimize-deployment/configuration/service-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,79 @@ es:
# process instance can contain. This limit helps to prevent out of memory errors and should be used with care.
nested_documents_limit: 10000

# everything that is related with configuring OpenSearch or creating
# a connection to it.
opensearch:
connection:
# Maximum time without connection to OpenSearch, Optimize should
# wait until a timeout triggers.
timeout: 10000
# Maximum size of the OpenSearch response consumer heap buffer.
responseConsumerBufferLimitInMb: 100
# The path prefix under which OpenSearch is available
pathPrefix: ""
# a list of OpenSearch nodes Optimize can connect to. If you have built
# an OpenSearch cluster with several nodes it is recommended to define
# several connection points in case one node fails.
nodes:
# the address/hostname under which the OpenSearch node is available.
- host: "localhost"
# A port number used by OpenSearch to accept HTTP connections.
httpPort: 9200
# Determines whether the hostname verification should be skipped
skipHostnameVerification: false
# Configuration relating to OS backup
backup:
# The repository name in which the backups should be stored
repositoryName: ""
# OpenSearch security settings
security:
# the basic auth (x-pack) username
username: null
# the basic auth (x-pack) password
password: null
# SSL/HTTPS secured connection settings
ssl:
# path to a PEM encoded file containing the certificate (or certificate chain)
# that will be presented to clients when they connect.
certificate: null
# A list of paths to PEM encoded CA certificate files that should be trusted, e.g. ['/path/to/ca.crt'].
# Note: if you are using a public CA that is already trusted by the Java runtime,
# you do not need to set the certificate_authorities.
certificate_authorities: []
# used to enable or disable TLS/SSL for the HTTP connection
enabled: false
# used to specify that the certificate was self-signed
selfSigned: false
# Maximum time in seconds a request to opensearch should last, before a timeout
# triggers.
scrollTimeoutInSeconds: 60
settings:
# the maximum number of buckets returned for an aggregation
aggregationBucketLimit: 1000
index:
# the prefix prepended to all Optimize index and alias names
# NOTE: Changing this after Optimize was already run before, will create new empty indexes
prefix: "optimize"
# How often should the data replicated in case of node failure.
number_of_replicas: 1
# How many shards should be used in the cluster for process instance and decision instance indices.
# All other indices will be made up of a single shard
# NOTE: this property only applies the first time Optimize is started and
# the schema/mapping is deployed on OpenSearch. If you want to take
# this property to take effect again, you need to delete all indexes (with it all data)
# and restart Optimize. This configuration will also only be applied to the current write instance indices. Archive
# indices will have a single shard regardless
number_of_shards: 1
# How long OpenSearch waits until the documents are available
# for search. A positive value defines the duration in seconds.
# A value of -1 means that a refresh needs to be done manually.
refresh_interval: 2s
# Optimize uses nested documents to store list information such as activities or variables belonging to a
# process instance. So this setting defines the maximum number of activities/variables that a single
# process instance can contain. This limit helps to prevent out of memory errors and should be used with care.
nested_documents_limit: 10000

plugin:
# Defines the directory path in the local Optimize file system which should be checked for plugins
directory: "./plugin"
Expand Down
20 changes: 13 additions & 7 deletions .../self-managed/optimize-deployment/configuration/shared-elasticsearch-cluster.md
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,27 @@
---
id: shared-elasticsearch-cluster
title: "Shared Elasticsearch cluster"
description: "Operate multiple Optimize instances on a shared Elasticsearch cluster."
title: "Shared Elasticsearch/OpenSearch cluster"
description: "Operate multiple Optimize instances on a shared Elasticsearch/OpenSearch cluster."
---

In case you have a large shared Elasticsearch cluster that you want to operate multiple Optimize instances on that are intended to run in complete isolation from each other, it is required to change the [`es.settings.index.prefix`](./system-configuration.md#index-settings) setting for each Optimize instance.
In case you have a large shared Elasticsearch/OpenSearch cluster that you want to operate multiple Optimize instances on that are intended to run in complete isolation from each other, it is required to change the [`*.settings.index.prefix`](./system-configuration.md#index-settings) setting for each Optimize instance.

:::note Heads Up!
Although a shared Elasticsearch cluster setup is possible, it's recommended to operate a dedicated Elasticsearch cluster per Optimize instance.
Although a shared Elasticsearch/OpenSearch cluster setup is possible, it's recommended to operate a dedicated Elasticsearch/OpenSearch cluster per Optimize instance.

This is due to the fact that a dedicated cluster provides the highest reliability (no resource sharing and no breaking side effects due to misconfiguration) and flexibility (e.g. Elasticsearch and/or Optimize updates can be performed independently between different Optimize setups).
This is due to the fact that a dedicated cluster provides the highest reliability (no resource sharing and no breaking side effects due to misconfiguration) and flexibility (e.g. Elasticsearch/OpenSearch and/or Optimize updates can be performed independently between different Optimize setups).
:::

The following illustration demonstrates this use case with two Optimize instances that connect to the same Elasticsearch cluster but are configured with different `es.settings.index.prefix` values. This results in different indexes and aliases created on the cluster, strictly isolating the data of both Optimize instances, so no instance accesses the data of the other instance.
The following illustration demonstrates this use case with two Optimize instances that connect to the same Elasticsearch/OpenSearch cluster but are configured with different `*.settings.index.prefix` values. This results in different indexes and aliases created on the cluster, strictly isolating the data of both Optimize instances, so no instance accesses the data of the other instance.

:::note Warning
Changing the value of `es.settings.index.prefix` after an instance was already running results in new indexes being created with the new prefix value. There is no support in migrating data between indexes based on different prefixes.
Changing the value of `*.settings.index.prefix` after an instance was already running results in new indexes being created with the new prefix value. There is no support in migrating data between indexes based on different prefixes.
:::

:::note
OpenSearch support is currently only available for `ccsm` mode. Moreover, OpenSearch support in Optimize is limited to data import and the raw data report. The remaining functionality will be delivered with upcoming patches.
:::

\* Elasticsearch index prefix settings path: `es.settings.index.prefix`<br/> \* OpenSearch index prefix settings path: `opensearch.settings.index.prefix`

![Shared Elasticsearch Cluster Setup](img/shared-elasticsearch-cluster.png)
Loading

0 comments on commit 5252446

Please sign in to comment.