Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ES exporter to application.yaml #4031

Merged
merged 9 commits into from
Jul 23, 2024
169 changes: 165 additions & 4 deletions docs/guides/react-components/install-plain-java.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,169 @@ For this installation, you must have:
- OpenJDK 21+ locally installed
- Camunda `8.6.0-alpha2` or later
akeller marked this conversation as resolved.
Show resolved Hide resolved

### Download and configure Elasticsearch

1. Download [Elasticsearch 8.9.2](https://www.elastic.co/downloads/past-releases/elasticsearch-8-9-2).
akeller marked this conversation as resolved.
Show resolved Hide resolved
2. Disable Elasticsearch's security packages by setting the `xpack.security.*` configuration options to `false` in `ELASTICSEARCH_HOME/config/elasticsearch.yml`.
akeller marked this conversation as resolved.
Show resolved Hide resolved
akeller marked this conversation as resolved.
Show resolved Hide resolved
3. Start Elasticsearch by running `ELASTICSEARCH_HOME/bin/elasticsearch` (or `ELASTICSEARCH_HOME\bin\elasticsearch.bat` on Windows).
akeller marked this conversation as resolved.
Show resolved Hide resolved
akeller marked this conversation as resolved.
Show resolved Hide resolved

### Download and configure Camunda

1. Download the [latest release artifact](https://github.com/camunda/camunda/releases), starting with [8.6.0-alpha2](https://github.com/camunda/camunda/releases/tag/8.6.0-alpha2).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running alpha2 seemed to result in a version mismatch for me, but running alpha3 went fine:

io.camunda.operate.schema.SchemaStartup - Schema startup failed: Error(s) in index schema: Newer version(s) for operation (8.4.0) already exists: [8.4.1]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romansmirnov thoughts on this?

Copy link
Member

@romansmirnov romansmirnov Jul 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assumption: The same ES instance has been used, but first, alpha3 was started, creating an index in version 8.4.1, and then alpha2 was started, expecting the index to be in version 8.4.0.

Can you please confirm my understanding? If so, I don't recommend using the same ES instance and switching from a newer C8 version to an older C8 version.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@conceptualshark I don't know if we should expect people using the same ES instance (purposefully or accidentally). Can I give you a follow up here to maybe add a small troubleshooting section or think more about this experience for new users?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing I can think of that would have impacted this is having started C8SM Run on this machine (it was not running at the time). When I tested this PR, I ran ES + alpha2, failed with the above error, stopped both, then reinstalled and ran ES + alpha3, so I am not sure why an index (or even ES instance) would have been maintained.

Unfortunately, I can't add anything for resolving it, because it still doesn't work on my machine (I had to run it in a VM).

akeller marked this conversation as resolved.
Show resolved Hide resolved
2. Download [Elasticsearch 8.9.2](https://www.elastic.co/downloads/past-releases/elasticsearch-8-9-2).
3. For non-production cases, disable Elasticsearch's security packages by setting the `xpack.security.*` configuration options to `false` in `ELASTICSEARCH_HOME/config/elasticsearch.yml`.
4. Start Elasticsearch by running `ELASTICSEARCH_HOME/bin/elasticsearch` (or `ELASTICSEARCH_HOME\bin\elasticsearch.bat` on Windows).
5. To start Camunda, run `bin/camunda` (or `bin\camunda.bat` on Windows).
2. Modify the `application.yaml` to include the Elasticsearch exporter by adding the following to line 91:
akeller marked this conversation as resolved.
Show resolved Hide resolved

```
exporters:
elasticsearch:
className: io.camunda.zeebe.exporter.ElasticsearchExporter
args:
url: http://localhost:9200
index:
prefix: zeebe-record
```
akeller marked this conversation as resolved.
Show resolved Hide resolved

:::note
Spacing is important! Ensure four spaces and proper nesting of configuration.
akeller marked this conversation as resolved.
Show resolved Hide resolved

<details>
<summary>Still need help?</summary>
akeller marked this conversation as resolved.
Show resolved Hide resolved

Here is the full `application.yaml` file:

```
zeebe:
broker:
gateway:
# Enable the embedded gateway to start on broker startup.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_ENABLE.
enable: true

network:
# Sets the port the embedded gateway binds to.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_NETWORK_PORT.
port: 26500

security:
# Enables TLS authentication between clients and the gateway
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED.
enabled: false
authentication:
# Controls which authentication mode is active, supported modes are 'none' and 'identity'.
# If 'identity' is set, authentication will be done using camunda-identity, which needs to
# be configured in the corresponding subsection. See also https://docs.camunda.io/docs/self-managed/identity/what-is-identity/ .
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_SECURITY_AUTHENTICATION_MODE.
mode: none

network:
# Controls the default host the broker should bind to. Can be overwritten on a
# per-binding basis for client, management and replication
# This setting can also be overridden using the environment variable ZEEBE_BROKER_NETWORK_HOST.
host: 0.0.0.0

data:
# Specify a directory in which data is stored.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_DIRECTORY.
directory: data
# The size of data log segment files.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_LOGSEGMENTSIZE.
logSegmentSize: 128MB
# How often we take snapshots of streams (time unit)
# This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_SNAPSHOTPERIOD.
snapshotPeriod: 15m

cluster:
# Specifies the Zeebe cluster size.
# This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_CLUSTERSIZE.
clusterSize: 1
# Controls the replication factor, which defines the count of replicas per partition.
# This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR.
replicationFactor: 1
# Controls the number of partitions, which should exist in the cluster.
# This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT.
partitionsCount: 1

threads:
# Controls the number of non-blocking CPU threads to be used.
# WARNING: You should never specify a value that is larger than the number of physical cores
# available. Good practice is to leave 1-2 cores for ioThreads and the operating
# system (it has to run somewhere). For example, when running Zeebe on a machine
# which has 4 cores, a good value would be 2.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
cpuThreadCount: 2
# Controls the number of io threads to be used.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_IOTHREADCOUNT
ioThreadCount: 2

exporters:
elasticsearch:
className: io.camunda.zeebe.exporter.ElasticsearchExporter
args:
url: http://localhost:9200
index:
prefix: zeebe-record

camunda:
# Operate configuration properties
operate:
# Set operate username and password.
# If user with <username> does not exists it will be created.
# Default: demo/demo
#username:
#password:
# ELS instance to store Operate data
elasticsearch:
# Cluster name
clusterName: elasticsearch
# URL
url: http://localhost:9200
# Zeebe instance
zeebe:
# Gateway address
gatewayAddress: localhost:26500
# ELS instance to export Zeebe data to
zeebeElasticsearch:
# Cluster name
clusterName: elasticsearch
# URL
url: http://localhost:9200
# Index prefix, configured in Zeebe Elasticsearch exporter
prefix: zeebe-record
# Tasklist configuration properties
tasklist:
# Set Tasklist username and password.
# If user with <username> does not exists it will be created.
# Default: demo/demo
#username:
#password:
# ELS instance to store Tasklist data
elasticsearch:
# Cluster name
clusterName: elasticsearch
# URL
url: http://localhost:9200
# Zeebe instance
zeebe:
# Gateway address
gatewayAdress: localhost:26500
# ELS instance to export Zeebe data to
zeebeElasticsearch:
# Cluster name
clusterName: elasticsearch
# Url
url: http://localhost:9200
# Index prefix, configured in Zeebe Elasticsearch exporter
prefix: zeebe-record

```

</details>

:::

Save the file. Without performing this step, no data will be visible in Operate or Tasklist.

3. To start Camunda, run `bin/camunda` (or `bin\camunda.bat` on Windows).

akeller marked this conversation as resolved.
Show resolved Hide resolved
:::tip
Operate can be found at `http://localhost:8080/` and Tasklist can be found at `http://localhost:8080/tasklist`.
:::
akeller marked this conversation as resolved.
Show resolved Hide resolved
Loading