diff --git a/docs/components/react-components/backpressure-export.md b/docs/components/react-components/backpressure-export.md new file mode 100644 index 00000000000..234546e726f --- /dev/null +++ b/docs/components/react-components/backpressure-export.md @@ -0,0 +1,6 @@ +--- +--- + +:::note +A backlog may also occur when Elasticsearch (or other seconday storage instance) is under high load, reducing the rate at which records can be exported. +::: diff --git a/docs/components/zeebe/technical-concepts/internal-processing.md b/docs/components/zeebe/technical-concepts/internal-processing.md index 34582e34b41..2c77abfa68c 100644 --- a/docs/components/zeebe/technical-concepts/internal-processing.md +++ b/docs/components/zeebe/technical-concepts/internal-processing.md @@ -57,17 +57,23 @@ This command can in turn be processed, completing the service task and driving t ## Handling backpressure +import BackpressureWarning from '../../../components/react-components/backpressure-warning.md' +import BackpressureExport from '../../../components/react-components/backpressure-export.md' + When a broker receives a client request, it is written to the **event stream** first, and processed later by the stream processor. If the processing is slow or if there are many client requests in the stream, it might take too long for the processor to start processing the command. If the broker keeps accepting new requests from the client, the backlog increases and the processing latency can grow beyond an acceptable time. -To avoid such problems, Zeebe employs a [backpressure](/self-managed/zeebe-deployment/operations/backpressure.md) mechanism. -When the broker receives more requests than it can process with an acceptable latency, it rejects some requests. + + +To avoid such problems, Zeebe employs a [backpressure](/self-managed/zeebe-deployment/operations/backpressure.md) mechanism. When the broker receives more requests than it can process with an acceptable latency, it rejects some requests. + +### Resource exhausted -Backpressure is indicated to the client by throwing a **resource exhausted** exception. If a client sees this exception, it can retry the requests with an appropriate retry strategy. If the rejection rate is high, it indicates the broker is constantly under high load and you need to reduce the rate of requests. Alternatively, you can also increase broker resources to adjust to your needs. In high-load scenarios, it is recommended to [benchmark](https://camunda.com/blog/2022/05/how-to-benchmark-your-camunda-platform-8-cluster/) your Zeebe broker up front to size it correctly. +Backpressure is indicated to the client by throwing a **resource exhausted** exception, which indicates whether the write limit or request limit has been reached. + +If a client sees this exception, it can retry the requests with an appropriate retry strategy. If the rejection rate is high, it indicates the broker is constantly under high load and you need to reduce the rate of requests. Alternatively, you can also increase broker resources to adjust to your needs. In high-load scenarios, it is recommended to [benchmark](https://camunda.com/blog/2022/05/how-to-benchmark-your-camunda-platform-8-cluster/) your Zeebe broker up front to size it correctly. The maximum rate of requests that can be processed by a broker depends on the processing capacity of the machine, the network latency, current load of the system, etc. There is no fixed limit configured in Zeebe for the maximum rate of requests it accepts. Instead, Zeebe uses an adaptive algorithm to dynamically determine the limit of the number of in-flight requests (the requests that are accepted by the broker, but not yet processed). The in-flight request count is incremented when a request is accepted, and decremented when a response is sent back to the client. The broker rejects requests when the in-flight request count reaches the limit. -import BackpressureWarning from '../../../components/react-components/backpressure-warning.md' - diff --git a/docs/self-managed/zeebe-deployment/operations/backpressure.md b/docs/self-managed/zeebe-deployment/operations/backpressure.md index 7d07a60ed75..2bfd9670c3d 100644 --- a/docs/self-managed/zeebe-deployment/operations/backpressure.md +++ b/docs/self-managed/zeebe-deployment/operations/backpressure.md @@ -5,11 +5,15 @@ description: "This document outlines an overview of backpressure and its accompa keywords: [back-pressure, backpressure, back pressure] --- +import BackpressureExport from '../../../components/react-components/backpressure-export.md' + When a broker receives a client request, it is written to the **event stream** first (see section [internal processing](/components/zeebe/technical-concepts/internal-processing.md) for details), and processed later by the stream processor. If the processing is slow or if there are many client requests in the stream, it might take too long for the processor to start processing the command. If the broker keeps accepting new requests from the client, the backlog increases and the processing latency can grow beyond an acceptable time. + + To avoid such problems, Zeebe employs a backpressure mechanism. When the broker receives more requests than it can process with an acceptable latency, it rejects some requests (see [technical error handling](/apis-tools/zeebe-api/technical-error-handling.md)). ### Terminology