Skip to content

Commit

Permalink
[doc] Introduce Full Compaction in compaction doc
Browse files Browse the repository at this point in the history
  • Loading branch information
JingsongLi committed Aug 7, 2024
1 parent b118e63 commit 37bf58b
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 4 deletions.
13 changes: 13 additions & 0 deletions docs/content/primary-key-table/compaction.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,19 @@ In compaction, you can configure record-Level expire time to expire records, you

Expiration happens in compaction, and there is no strong guarantee to expire records in time.

## Full Compaction

Paimon Compaction uses [Universal-Compaction](https://github.com/facebook/rocksdb/wiki/Universal-Compaction).
By default, when there is too much incremental data, Full Compaction will be automatically performed. You don't usually
have to worry about it.

Paimon also provides a configuration that allows for regular execution of Full Compaction.

1. 'compaction.optimization-interval': Implying how often to perform an optimization full compaction, this
configuration is used to ensure the query timeliness of the read-optimized system table.
2. 'full-compaction.delta-commits': Full compaction will be constantly triggered after delta commits. its disadvantage
is that it can only perform compaction synchronously, which will affect writing efficiency.

## Compaction Options

### Number of Sorted Runs to Pause Writing
Expand Down
5 changes: 1 addition & 4 deletions docs/content/primary-key-table/table-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,10 +109,7 @@ So by default, compaction is synchronous, and if asynchronous is turned on, ther
If you don't want to use Deletion Vectors mode, you want to query fast enough in MOR mode, but can only find
older data, you can also:

1. Configure 'compaction.optimization-interval' when writing data. For streaming jobs, optimized compaction will then
be performed periodically; For batch jobs, optimized compaction will be carried out when the job ends. (Or configure
`'full-compaction.delta-commits'`, its disadvantage is that it can only perform compaction synchronously, which will
affect writing efficiency)
1. Configure 'compaction.optimization-interval' when writing data.
2. Query from [read-optimized system table]({{< ref "maintenance/system-tables#read-optimized-table" >}}). Reading from
results of optimized files avoids merging records with the same key, thus improving reading performance.

Expand Down

0 comments on commit 37bf58b

Please sign in to comment.