diff --git a/docs/content/maintenance/write-performance.md b/docs/content/maintenance/write-performance.md index 02362b9096e7..ade2c3353e3c 100644 --- a/docs/content/maintenance/write-performance.md +++ b/docs/content/maintenance/write-performance.md @@ -160,12 +160,3 @@ You can use fine-grained-resource-management of Flink to increase committer heap 1. Configure Flink Configuration `cluster.fine-grained-resource-management.enabled: true`. (This is default after Flink 1.18) 2. Configure Paimon Table Options: `sink.committer-memory`, for example 300 MB, depends on your `TaskManager`. (`sink.committer-cpu` is also supported) - -## Changelog Compaction - -If Flink's checkpoint interval is short (for example, 30 seconds) and the number of buckets is large, -each snapshot may produce lots of small changelog files. -Too many files may put a burden on the distributed storage cluster. - -In order to compact small changelog files into large ones, you can set the table option `changelog.precommit-compact = true`. -Default value of this option is false, if true, it will add a compact coordinator and worker operator after the writer operator, which copies changelog files into large ones. diff --git a/docs/content/primary-key-table/changelog-producer.md b/docs/content/primary-key-table/changelog-producer.md index 011f7b6f27a7..a9364ee9f07c 100644 --- a/docs/content/primary-key-table/changelog-producer.md +++ b/docs/content/primary-key-table/changelog-producer.md @@ -130,3 +130,14 @@ efficient as the input changelog producer and the latency to produce changelog m Full-compaction changelog-producer supports `changelog-producer.row-deduplicate` to avoid generating -U, +U changelog for the same record. + +## Changelog Merging + +For `input`, `lookup`, `full-compaction` 'changelog-producer'. + +If Flink's checkpoint interval is short (for example, 30 seconds) and the number of buckets is large, each snapshot may +produce lots of small changelog files. Too many files may put a burden on the distributed storage cluster. + +In order to compact small changelog files into large ones, you can set the table option `changelog.precommit-compact = true`. +Default value of this option is false, if true, it will add a compact coordinator and worker operator after the writer +operator, which copies changelog files into large ones.