[Core] Append-Only table use zstd compression default reduce storage in parquet format

[xuzifu666]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When we test append only table in stream scences, we found the condition paimon table storage is 1.3x size of hive table，this is a litter high，then we change compression from snappy(wich is default) to zstd, storage from 1.3x to 1.04x which is expected to us.
From other lake engine we also do the change and other lake engine also reduce storage and stream job is stable as the same.
So should we make the default compression in parquet to zstd in append only sences?

snappy : 499GB
hive: 372GB
snappy / hive = 1.34

zstd: 390GB
hive: 372GB

zstd / hive = 1.04

Solution

Set compression with zstd in append only table

Anything else?

none

Are you willing to submit a PR?

• I'm willing to submit a PR!

[xuzifu666]

close it first due to doc had remind it