Skip to content

Commit

Permalink
update(MirrorMaker): improve configuration concepts
Browse files Browse the repository at this point in the history
Explained the different configuration layers and moved monitoring
instructions to howto section.

[DOC-1169]
  • Loading branch information
jclarysse committed Dec 3, 2024
1 parent 0b66f99 commit 56e7196
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 34 deletions.
Original file line number Diff line number Diff line change
@@ -1,12 +1,34 @@
---
title: MirrorMaker 2 common parameters
title: Configuration parameters
---

MirrorMaker 2 (MM2) offers a suite of parameters to help with data replication and monitoring within Apache Kafka® ecosystems.
This topic outlines common parameters you can adjust, along with tips for
validating MM2's performance.
Apache Kafka® MirrorMaker 2 provides a suite of configuration parameters
to help with data replication within Apache Kafka® ecosystems.

1. Increase the value of `kafka_mirrormaker.tasks_max_per_cpu` in the
## Configuration layers

1. **Service** configurations apply to the nodes and workers of Apache Kafka® MirrorMaker 2 cluster.
- They are documented under [Advanced parameters for Aiven for Apache Kafka® MirrorMaker 2](/products/kafka/kafka-mirrormaker/reference/advanced-params).
- An example of service configuration is `kafka_mirrormaker.emit_checkpoints_enabled`: Whether to emit consumer group offset checkpoints to target cluster periodically.
- Changing the value of such parameter leads to a restart of the workers (along with their connectors and tasks).
1. **Replication-flow** configurations apply to the connectors (Source, Sink, Checkpoint, Heartbeat).
- They are documented under [Aiven Terraform provider mirrormaker_replication_flow resource documentation](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/mirrormaker_replication_flow).
- An example of replication-flow configuration is `topics`: List of topics and/or regular expressions to replicate (see [topics included in a replication flow](/products/kafka/kafka-mirrormaker/concepts/replication-flow-topics-regex)).
- Chaging the value of such parameter leads to the restart of impacted connectors (along with their tasks).
1. **Integration** configurations apply to the consumers and producers of the connectors.
- They are documented under [Aiven Terraform provider service_integration resource documentation](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/service_integration#nested-schema-for-kafka_mirrormaker_user_configkafka_mirrormaker).
- An example of integration configuration is `consumer_fetch_min_bytes`: The minimum amount of data the server should return for a fetch request.
- Changing the value of such parameter leads to a restart of the workers (along with their connectors and tasks).

:::note
Most configurations are directly inherited from the upstream [KIP-382: MirrorMaker 2.0 - Configuration Properties](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95650722#KIP382:MirrorMaker2.0-ConnectorConfigurationProperties).
:::

## Common parameters

This section outlines common parameters which you can adjust.

1. Increase the value of `kafka_mirrormaker.tasks_max_per_cpu` _in the
advanced options. Setting this to match the number of partitions can
enhance performance.
1. Ensure the interval seconds for the following settings match. You
Expand All @@ -19,32 +41,9 @@ validating MM2's performance.
1. To exclude internal topics, add these patterns to your topic
blacklist:
- `.*[\-\.]internal` `.*\.replica` `__.*` `connect.*`
1. Depending on your use case, consider adjusting these parameters:
- `kafka_mirrormaker.consumer_fetch_min_bytes`
- `kafka_mirrormaker.producer_batch_size`
- `kafka_mirrormaker.producer_buffer_memory`
- `kafka_mirrormaker.producer_linger_ms`
- `kafka_mirrormaker.producer_max_request_size`

## MirrorMaker 2 validation tips

To ensure MirrorMaker 2 is up-to-date with message processing, monitor
these:

1. **Consumer lag metric**: Monitor the `kafka.consumer_lag` metric.

1. **Dashboard metrics**: If MirrorMaker 2 stops adding records to a
topic, the `jmx.kafka.connect.mirror.record_count` metric stops
increasing, showing a flat line on the dashboard.

1. **Retrieve latest messages with \`kt\`**: Use
[kt](https://github.com/fgeller/kt) to retrieve the latest messages
from all partitions with the following command:

```
kt consume -auth ./mykafka.conf \
-brokers SERVICE-PROJECT.aivencloud.com:PORT \
-topic topicname -offsets all=newest:newest | \
jq -c -s 'sort_by(.partition) | .[] | \
{partition: .partition, value: .value, timestamp: .timestamp}'
```
1. Depending on your use case, consider adjusting these integration parameters:
- `consumer_fetch_min_bytes`
- `producer_batch_size`
- `producer_buffer_memory`
- `producer_linger_ms`
- `producer_max_request_size`
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: Monitor replication execution
---

Apache Kafka® MirrorMaker 2 leverages Kafka Connect to help with state management
and monitoring.

## Tips

To ensure that the replication is up-to-date with message processing, check this:

1. **Consumer lag metric**: Monitor the `kafka.consumer_lag` metric.

1. **Dashboard metrics**: If MirrorMaker 2 stops adding records to a
topic, the `jmx.kafka.connect.mirror.record_count` metric stops
increasing, showing a flat line on the dashboard.

1. **Retrieve latest messages with \`kt\`**: Use
[kt](https://github.com/fgeller/kt) to retrieve the latest messages
from all partitions with the following command:

```
kt consume -auth ./mykafka.conf \
-brokers SERVICE-PROJECT.aivencloud.com:PORT \
-topic topicname -offsets all=newest:newest | \
jq -c -s 'sort_by(.partition) | .[] | \
{partition: .partition, value: .value, timestamp: .timestamp}'
```
1 change: 1 addition & 0 deletions sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1058,6 +1058,7 @@ const sidebars: SidebarsConfig = {
items: [
'products/kafka/kafka-mirrormaker/howto/integrate-external-kafka-cluster',
'products/kafka/kafka-mirrormaker/howto/setup-replication-flow',
'products/kafka/kafka-mirrormaker/howto/monitor-replication-execution',
'products/kafka/kafka-mirrormaker/howto/remove-mirrormaker-prefix',
'products/kafka/kafka-mirrormaker/howto/datadog-customised-metrics',
'products/kafka/kafka-mirrormaker/howto/log-analysis-offset-sync-tool',
Expand Down

0 comments on commit 56e7196

Please sign in to comment.