Skip to content

Commit

Permalink
[doc] Introduce how the read.batch-size option can impact memory cons…
Browse files Browse the repository at this point in the history
…umption when compaction in write-performance page (apache#1175)
  • Loading branch information
wg1026688210 authored May 19, 2023
1 parent ffff75b commit 864b5fa
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions docs/content/maintenance/write-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,4 +213,5 @@ There are three main places in Paimon writer that takes up memory:
* Writer's memory buffer, shared and preempted by all writers of a single task. This memory value can be adjusted by the `write-buffer-size` table property.
* Memory consumed when merging several sorted runs for compaction. Can be adjusted by the `num-sorted-run.compaction-trigger` option to change the number of sorted runs to be merged.
* If the row is very large, reading too many lines of data at once can consume a lot of memory when making a compaction. Reducing the `read.batch-size` option can alleviate the impact of this case.
* The memory consumed by writing columnar (ORC, Parquet, etc.) file, which is not adjustable.

0 comments on commit 864b5fa

Please sign in to comment.