[core] Add parquet write page limit parameter

[Aiden-Dong]

Purpose

Linked issue: close #4586

Added the parquet.page.row.count.limit parameter for Parquet file writing. The writing of Parquet files is influenced by both parquet.page.size and parquet.page.row.count.limit. If only parquet.page.size is set, it may not have an effect or could lead to misalignment of pages, impacting performance.

新增parquet 文件写入 parquet.page.row.count.limit 参数传递，parquet 文件写入时是通过 parquet.page.size 与 parquet.page.row.count.limit 共同影响的， 如果单一设置 parquet.page.size 可能没有产生作用，或者导致 page 不对齐影响性能

// org.apache.parquet.column.impl.ColumnWriteStoreBase
private void sizeCheck() {
    ...
    int pageRowCountLimit = props.getPageRowCountLimit();
    ...
    for (ColumnWriterBase writer : columns.values()) {
      long usedMem = writer.getCurrentPageBufferedSize();
      long rows = rowCount - writer.getRowsWrittenSoFar();
      long remainingMem = props.getPageSizeThreshold() - usedMem;
       if (remainingMem <= thresholdTolerance || rows >= pageRowCountLimit) {
        writer.writePage();
        remainingMem = props.getPageSizeThreshold();
      }
    }
    ... 
}

Tests

API and Format

Documentation

[JingsongLi]

Hi @Aiden-Dong , please keep English version in your description.

[JingsongLi]

+1