Skip to content

Commit

Permalink
add additional source information to backpressure docs
Browse files Browse the repository at this point in the history
  • Loading branch information
conceptualshark committed Aug 9, 2024
1 parent bfcc222 commit d809299
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 5 deletions.
6 changes: 6 additions & 0 deletions docs/components/react-components/backpressure-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
---

:::note
A backlog may also occur when Elasticsearch (or other seconday storage instance) is under high load, reducing the rate at which records can be exported.
:::
16 changes: 11 additions & 5 deletions docs/components/zeebe/technical-concepts/internal-processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,23 @@ This command can in turn be processed, completing the service task and driving t

## Handling backpressure

import BackpressureWarning from '../../../components/react-components/backpressure-warning.md'
import BackpressureExport from '../../../components/react-components/backpressure-export.md'

When a broker receives a client request, it is written to the **event stream** first, and processed later by the stream processor. If the processing is slow or if there are many client requests in the stream, it might take too long for the processor to start processing the command. If the broker keeps accepting new requests from the client, the backlog increases and the processing latency can grow beyond an acceptable time.

To avoid such problems, Zeebe employs a [backpressure](/self-managed/zeebe-deployment/operations/backpressure.md) mechanism.
When the broker receives more requests than it can process with an acceptable latency, it rejects some requests.
<BackpressureExport/>

To avoid such problems, Zeebe employs a [backpressure](/self-managed/zeebe-deployment/operations/backpressure.md) mechanism. When the broker receives more requests than it can process with an acceptable latency, it rejects some requests.

### Resource exhausted

Backpressure is indicated to the client by throwing a **resource exhausted** exception. If a client sees this exception, it can retry the requests with an appropriate retry strategy. If the rejection rate is high, it indicates the broker is constantly under high load and you need to reduce the rate of requests. Alternatively, you can also increase broker resources to adjust to your needs. In high-load scenarios, it is recommended to [benchmark](https://camunda.com/blog/2022/05/how-to-benchmark-your-camunda-platform-8-cluster/) your Zeebe broker up front to size it correctly.
Backpressure is indicated to the client by throwing a **resource exhausted** exception, which indicates whether the write limit or request limit has been reached.

If a client sees this exception, it can retry the requests with an appropriate retry strategy. If the rejection rate is high, it indicates the broker is constantly under high load and you need to reduce the rate of requests. Alternatively, you can also increase broker resources to adjust to your needs. In high-load scenarios, it is recommended to [benchmark](https://camunda.com/blog/2022/05/how-to-benchmark-your-camunda-platform-8-cluster/) your Zeebe broker up front to size it correctly.

The maximum rate of requests that can be processed by a broker depends on the processing capacity of the machine, the network latency, current load of the system, etc. There is no fixed limit configured in Zeebe for the maximum rate of requests it accepts. Instead, Zeebe uses an adaptive algorithm to dynamically determine the limit of the number of in-flight requests (the requests that are accepted by the broker, but not yet processed).

The in-flight request count is incremented when a request is accepted, and decremented when a response is sent back to the client. The broker rejects requests when the in-flight request count reaches the limit.

import BackpressureWarning from '../../../components/react-components/backpressure-warning.md'

<BackpressureWarning/>
4 changes: 4 additions & 0 deletions docs/self-managed/zeebe-deployment/operations/backpressure.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,15 @@ description: "This document outlines an overview of backpressure and its accompa
keywords: [back-pressure, backpressure, back pressure]
---

import BackpressureExport from '../../../components/react-components/backpressure-export.md'

When a broker receives a client request, it is written to the **event stream** first (see section [internal processing](/components/zeebe/technical-concepts/internal-processing.md) for details), and processed later by the stream processor.

If the processing is slow or if there are many client requests in the stream, it might take too long for the processor to start processing the command.
If the broker keeps accepting new requests from the client, the backlog increases and the processing latency can grow beyond an acceptable time.

<BackpressureExport/>

To avoid such problems, Zeebe employs a backpressure mechanism. When the broker receives more requests than it can process with an acceptable latency, it rejects some requests (see [technical error handling](/apis-tools/zeebe-api/technical-error-handling.md)).

### Terminology
Expand Down

0 comments on commit d809299

Please sign in to comment.