Skip to content

Commit

Permalink
[doc] Add write and write buffer metrics (#2474)
Browse files Browse the repository at this point in the history
  • Loading branch information
schnappi17 authored Dec 11, 2023
1 parent 2a8fd89 commit ac328f1
Showing 1 changed file with 78 additions and 2 deletions.
80 changes: 78 additions & 2 deletions docs/content/maintenance/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,11 @@ There are three types of metrics provided in the Paimon metric system, `Gauge`,
- `Counter`: Used to count values by incrementing and decrementing.
- `Histogram`: Measure the statistical distribution of a set of values including the min, max, mean, standard deviation and percentile.

Paimon has supported built-in metrics to measure operations of **commits**, **scans** and **compactions**, which can be bridged to any computing engine that supports, like Flink, Spark etc.
Paimon has supported built-in metrics to measure operations of **commits**, **scans**, **writes** and **compactions**, which can be bridged to any computing engine that supports, like Flink, Spark etc.

## Metrics List

Below is lists of Paimon built-in metrics. They are summarized into three types of metrics, scan metrics, commit metrics and compaction metrics.
Below is lists of Paimon built-in metrics. They are summarized into types of scan metrics, commit metrics, write metrics, write buffer metrics and compaction metrics.

### Scan Metrics

Expand Down Expand Up @@ -215,6 +215,72 @@ Below is lists of Paimon built-in metrics. They are summarized into three types
</tbody>
</table>

### Write Metrics

<table class="table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 225pt">Metrics Name</th>
<th class="text-left" style="width: 65pt">Level</th>
<th class="text-left" style="width: 70pt">Type</th>
<th class="text-left" style="width: 300pt">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>writeRecordCount</td>
<td>Bucket</td>
<td>Counter</td>
<td>Total number of records written into the bucket.</td>
</tr>
<tr>
<td>flushCostMillis</td>
<td>Bucket</td>
<td>Histogram</td>
<td>Distributions of the time taken by the last few write buffer flushing.</td>
</tr>
<tr>
<td>prepareCommitCostMillis</td>
<td>Bucket</td>
<td>Histogram</td>
<td>Distributions of the time taken by the last few call of `prepareCommit`.</td>
</tr>
</tbody>
</table>

### Write Buffer Metrics

<table class="table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 225pt">Metrics Name</th>
<th class="text-left" style="width: 65pt">Level</th>
<th class="text-left" style="width: 70pt">Type</th>
<th class="text-left" style="width: 300pt">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>bufferPreemptCount</td>
<td>Table</td>
<td>Gauge</td>
<td>The total number of memory preempted.</td>
</tr>
<tr>
<td>usedWriteBufferSizeByte</td>
<td>Table</td>
<td>Gauge</td>
<td>Current used write buffer size in byte.</td>
</tr>
<tr>
<td>totalWriteBufferSizeByte</td>
<td>Table</td>
<td>Gauge</td>
<td>The total write buffer size configured in byte.</td>
</tr>
</tbody>
</table>

### Compaction Metrics

<table class="table table-bordered">
Expand Down Expand Up @@ -316,6 +382,16 @@ From Flink Web-UI, go to the committer operator's metrics, it's shown as:
<td>&lt;host&gt;.taskmanager.&lt;tm_id&gt;.&lt;job_name&gt;.&lt;committer_operator_name&gt;.&lt;subtask_index&gt;</td>
<td>paimon.table.&lt;table_name&gt;.commit</td>
</tr>
<tr>
<td>Write Metrics</td>
<td>&lt;host&gt;.taskmanager.&lt;tm_id&gt;.&lt;job_name&gt;.&lt;writer_operator_name&gt;.&lt;subtask_index&gt;</td>
<td>paimon.table.&lt;table_name&gt;.partition.&lt;partition_string&gt;.bucket.&lt;bucket_index&gt;.writer</td>
</tr>
<tr>
<td>Write Buffer Metrics</td>
<td>&lt;host&gt;.taskmanager.&lt;tm_id&gt;.&lt;job_name&gt;.&lt;writer_operator_name&gt;.&lt;subtask_index&gt;</td>
<td>paimon.table.&lt;table_name&gt;.writeBuffer</td>
</tr>
<tr>
<td>Compaction Metrics</td>
<td>&lt;host&gt;.taskmanager.&lt;tm_id&gt;.&lt;job_name&gt;.&lt;writer_operator_name&gt;.&lt;subtask_index&gt;</td>
Expand Down

0 comments on commit ac328f1

Please sign in to comment.