docs(zeebe): document job worker metrics concept (#2508)

* docs(zeebe): document job worker metrics concept

* style(formatting): technical review

---------

Co-authored-by: Christina Ausley <[email protected]>
Co-authored-by: christinaausley <[email protected]>

docs/apis-tools/java-client/job-worker.md

@@ -41,3 +41,106 @@ By backing off, the job worker helps Zeebe by reducing the load.
 :::note
 Zeebe's [backpressure mechanism](../../../self-managed/zeebe-deployment/operations/backpressure) can also be configured.
 :::
+
+## Metrics
+
+The job worker exposes metrics through a custom interface: [JobWorkerMetrics](https://github.com/camunda/zeebe/blob/main/clients/java/src/main/java/io/camunda/zeebe/client/api/worker/JobWorkerMetrics.java). These represent specific callbacks used by the job worker to keep track of various internals, e.g. count of jobs activated, count of jobs handled, etc.
+
+:::note
+By default, job workers will not track any metrics, and it's up to the caller to specify an implementation if they wish to make use of this feature.
+:::
+
+### Available metrics
+
+The API currently supports two metrics: the count of jobs activated, and the count of jobs handled.
+
+- **The count of jobs activated** is incremented every time a worker activates new jobs. This is done by calling `JobWorkerMetrics#jobActivated(int)`, with the first argument being the count of jobs newly activated. The method is called before the job is passed to the job handler.
+- **The count of jobs handled** is incremented every time a worker's `JobHandler` (passed to the builder via `JobWorkerBuilderStep2#handler(JobHandler)`) returns (regardless of whether it was successful). This is done by calling `JobWorkerMetrics#jobHandled(int)`, with the first argument being the count of jobs newly handled.
+
+For both counters, the expectation is that implementations will simply increment an underlying counter, and track the rate or increase of this counter to derive the speed at which jobs are activated/handled by a given worker.
+
+Additionally, by subtracting both counters, you can derive the count of queued or buffered jobs - jobs which have yet to be handled by the worker. This can help you tune your workers, e.g. scaling in or out, tuning the amount of jobs activated, etc.
+
+### Usage
+
+To use job worker metrics, create a new instance of a `JobWorkerMetrics` implementation, and pass it along to the builder:
+
+```java
+public final JobWorker openWorker(final ZeebeClient client, final JobHandler handler) {
+  final JobWorkerMetrics metrics = new MyCustomJobWorkerMetrics();
+  return client.newJobWorker()
+    .jobType("foo")
+    .handler(handler)
+    .metrics(metrics)
+    .open();
+}
+```
+
+#### Micrometer implementation
+
+The Java client comes with an optional, built-in [Micrometer](https://micrometer.io/) implementation of `JobWorkerMetrics`.
+
+:::note
+[Micrometer](https://micrometer.io/) is a popular metrics facade in the Java ecosystem - what SLF4J is to logging. It can be configured to export metrics to many other systems, such as OpenTelemetry, Prometheus, StatsD, Datadog, etc.
+:::
+
+If your project does not yet use Micrometer, you need to add it to your dependencies, and wire it up to your metrics backend, [as described in the Micrometer docs](https://micrometer.io/docs).
+
+Once Micrometer is set up in your project, you can start using the implementation. For example:
+
+```java
+public final JobWorker openWorker(final ZeebeClient client, final JobHandler handler) {
+  final MeterRegistry meterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
+  final JobWorkerMetrics metrics = JobWorkerMetrics
+    .micrometer()
+    .withMeterRegistry(meterRegistry)
+    .withTags(Tags.of("zeebe.client.worker.jobType", "foo", "zeebe.client.worker.name", "bee"))
+    .build();
+  return client.newJobWorker()
+    .jobType("foo")
+    .handler(handler)
+    .metrics(metrics)
+    .name("bee")
+    .open();
+}
+```
+
+:::note
+There are currently no built-in tags, primarily because these are likely to be high cardinality, which can become an issue with some metric registries. If you want per-worker tags, create a different `JobWorkerMetrics` instance per worker.
+:::
+
+This implementation creates two metrics:
+
+- `zeebe.client.worker.job.activated`: A counter tracking the count of jobs activated.
+- `zeebe.client.worker.job.handled`: A counter tracking the count of jobs handled.
+
+### Workarounds for additional metrics
+
+The decision to track a small set of metrics directly in the client is a conscious one. The idea is we should only be tracking what is not possible for users to track themselves. If you believe a specific metric should be tracked by us, do open a feature request for it. In the meantime, here is a list of workarounds to help you track additional job worker-related metrics that you can already use:
+
+#### Job polling count
+
+You can use a gRPC [ClientInterceptor](https://grpc.github.io/grpc-java/javadoc/io/grpc/ClientInterceptor.html) to track any client calls, including the `ActivateJobsCommand` call that is sent every time a worker polls for more jobs.
+
+Here's an example using [Micrometer](https://javadoc.io/doc/io.micrometer/micrometer-core/1.7.2/io/micrometer/core/instrument/binder/grpc/MetricCollectingServerInterceptor.html):
+
+```java
+public ZeebeClientBuilder configureClientMetrics(final ZeebeClientBuilder builder, final MeterRegistry meterRegistry) {
+  final ClientInterceptor monitoringInterceptor = new MetricCollectingClientInterceptor(meterRegistry);
+  return builder.withInterceptors(monitoringInterceptor);
+}
+```
+
+#### Executor metrics
+
+If you wish to tune your job worker executor, you can pass a custom, instrumented executor to the client builder. For example, if we use Micrometer:
+
+```java
+public ZeebeClientBuilder configureClientMetrics(
+    final ZeebeClientBuilder builder,
+    final ScheduledExecutorService executor,
+    final MeterRegistry meterRegistry) {
+  final ScheduledExecutorService instrumentedExecutor = ExecutorServiceMetrics.monitor(meterRegistry, executor, "job-worker-executor");
+  return builder.jobWorkerExecutor(instrumentedExecutor);
+}
+```