[core] Introduce metadata.stats-dense-store to reduce meta size for multiple columns table #4322

JingsongLi · 2024-10-14T12:30:54Z

Purpose

At present, Paimon uses BinaryRow to store statistical information. Generally, it is not a problem, but some businesses have fields with over 3000 columns.

The BinaryRow structure has a characteristic that each field occupies a fixed 8 bytes, and for a 3000 column table, one BinaryRow has 25kb of storage. SimpleStats has 3 BinaryRow, then it will be 100 kb storage. 100000 files have GB level storage. This is unacceptable.

This PR：

Introduce metadata.stats-dense-store to a dense mode to store SimpleStats and valueStatsCols in DataFileMeta.
You can set metadata.stats-mode = none, then valueStatsCols will be empty list, and SimpleStats is empty.
You can also set fields.b.stats-mode = full to enable stats for specific columns to enable data skipping, the meta storage will only contain b column.

Tests

AppendOnlyFileStoreTableTest
PrimaryKeyFileStoreTableTest

API and Format

Documentation

…ultiple columns table fix

wwj6591812 · 2024-10-14T15:23:19Z

Good，we just encountered a similar problem today!

wwj6591812 · 2024-10-14T15:52:40Z

paimon-common/src/main/java/org/apache/paimon/CoreOptions.java

+                                                    + " none statistic mode is set.")
+                                    .linebreak()
+                                    .text(
+                                            "Note, When this mode is enabled, the sdk in reading engine requires at least"


1、Change the "When" to "when".
2、the Paimon sdk in reading engine requires at least version 0.9.1 or 1.0.0 or higher?

wwj6591812 · 2024-10-14T15:58:10Z

paimon-common/src/main/java/org/apache/paimon/table/SystemFields.java

@@ -55,4 +55,8 @@ public class SystemFields {
    public static boolean isSystemField(int fieldId) {
        return fieldId >= SYSTEM_FIELD_ID_START;
    }
+
+    public static boolean isSystemField(String field) {


Why not add KEY_FIELD_PREFIX to SYSTEM_FIELD_NAMES?

What to solve?

If KEY_FIELD_PREFIX is not in SYSTEM_FIELD_NAMES, then the funtion name "isSystemField" is inappropriate.

So add KEY_FIELD_PREFIX to SYSTEM_FIELD_NAMES, what problem can be solved? Can you write a ut demo?

A solution is using a SYSTEM_FIELD_PREFIXS, and always using starWith. But it is not good for performance.

Let it go now.

wwj6591812 · 2024-10-14T16:01:26Z

paimon-common/src/main/java/org/apache/paimon/utils/ProjectedArray.java

+ */
+public class ProjectedArray implements InternalArray {
+
+    protected final int[] indexMapping;


The indexMapping、array and ProjectedArray should be private?

wwj6591812 · 2024-10-15T00:25:01Z

paimon-core/src/main/java/org/apache/paimon/io/KeyValueDataFileWriter.java

@@ -101,8 +103,8 @@ public KeyValueDataFileWriter(
        this.schemaId = schemaId;
        this.level = level;

-        this.keyStatsConverter = new SimpleStatsConverter(keyType);
-        this.valueStatsConverter = new SimpleStatsConverter(valueType);
+        this.keyStatsConverter = new SimpleStatsConverter(keyType, false);


this.keyStatsConverter = new SimpleStatsConverter(keyType);

wwj6591812 · 2024-10-15T01:07:08Z

paimon-core/src/main/java/org/apache/paimon/stats/SimpleStatsEvolution.java

+        private final InternalArray array;
+        private final long notFoundValue;
+
+        protected NullCountsEvoArray(int[] indexMapping, InternalArray array, long notFoundValue) {


Why protected?

wwj6591812 · 2024-10-15T01:14:59Z

paimon-core/src/main/java/org/apache/paimon/stats/SimpleStatsEvolution.java

+        private final InternalArray array;
+        private final long notFoundValue;
+
+        protected NullCountsEvoArray(int[] indexMapping, InternalArray array, long notFoundValue) {


Why protected?

wwj6591812 · 2024-10-15T01:17:06Z

paimon-core/src/main/java/org/apache/paimon/table/sink/CommitMessageSerializer.java

-                            dataFileSerializer.deserializeList(view),
-                            dataFileSerializer.deserializeList(view),
-                            dataFileSerializer.deserializeList(view)),
+                            fileDeserializer.get(), fileDeserializer.get(), fileDeserializer.get()),
                    new IndexIncrement(
                            indexEntrySerializer.deserializeList(view),
                            indexEntrySerializer.deserializeList(view)));


Condition 'version <= 2' is always 'true'

wwj6591812 · 2024-10-15T01:25:33Z

paimon-core/src/main/java/org/apache/paimon/table/source/DataSplit.java

@@ -316,6 +317,9 @@ private static FunctionWithIOException<DataInputView, DataFileMeta> getFileMetaS
            DataFileMeta08Serializer serializer = new DataFileMeta08Serializer();
            return serializer::deserialize;
        } else if (version == 2) {
+            DataFileMeta09Serializer serializer = new DataFileMeta09Serializer();
+            return serializer::deserialize;
+        } else if (version == 3) {


public static final int DATA_FILE_META_VERSION_1 = 1;
public static final int DATA_FILE_META_VERSION_2= 2;
public static final int DATA_FILE_META_VERSION_3= 3;

No need to do this.

wwj6591812 · 2024-10-15T01:26:15Z

paimon-common/src/main/java/org/apache/paimon/CoreOptions.java

@@ -1026,7 +1026,7 @@ public class CoreOptions implements Serializable {
    public static final String STATS_MODE_SUFFIX = "stats-mode";

    public static final ConfigOption<String> METADATA_STATS_MODE =
-            key("metadata." + STATS_MODE_SUFFIX)
+            key("metadata.stats-mode")


Why do this?

Can become more intuitive in the code.

wwj6591812 · 2024-10-15T01:29:12Z

...src/test/java/org/apache/paimon/manifest/ManifestCommittableSerializerCompatibilityTest.java

+                        null);
+        List<DataFileMeta> dataFiles = Collections.singletonList(dataFile);
+
+        LinkedHashMap<String, Pair<Integer, Integer>> dvRanges = new LinkedHashMap<>();


Map<String, Pair<Integer, Integer>> dvRanges = new LinkedHashMap<>();

IndexFileMeta requires a LinkedHashMap

wwj6591812 · 2024-10-15T01:34:08Z

Please modify this doc : https://paimon.apache.org/docs/master/flink/sql-ddl/#specify-statistics-mode

wwj6591812 · 2024-10-15T01:39:57Z

A question :
If I have a paimon table with 1000 columns, and I just want field B to use dense stat, other columns close stats.
Is my configuration correct？

metadata.stats-dense-store=true
metadata.stats-mode=none
fields.b.stats-mode=truncate(16)

wwj6591812 · 2024-10-15T01:47:54Z

paimon-common/src/main/java/org/apache/paimon/CoreOptions.java

+    public static final ConfigOption<Boolean> METADATA_STATS_DENSE_STORE =
+            key("metadata.stats-dense-store")
+                    .booleanType()
+                    .defaultValue(false)


Why the default value is not true?
You are worry about that many users are using old versions of Paimon sdk in their reading engine?

Yes, it is true

JingsongLi · 2024-10-15T02:51:24Z

A question : If I have a paimon table with 1000 columns, and I just want field B to use dense stat, other columns close stats. Is my configuration correct？

metadata.stats-dense-store=true metadata.stats-mode=none fields.b.stats-mode=truncate(16)

Correct!

wwj6591812 · 2024-10-15T03:37:58Z

Looks good to me!
I can't wait to use this feature quickly to solve our online produce problems!

+1

JingsongLi added 4 commits October 14, 2024 20:08

[core] Introduce metadata.stats-dense-store to reduce meta size for m…

a2f7893

…ultiple columns table fix

fix

b72d17e

fix

3d2e2f0

fix

547fa56

wwj6591812 reviewed Oct 15, 2024

View reviewed changes

Fix comments

7939203

JingsongLi merged commit 11636ac into apache:master Oct 15, 2024
13 checks passed

This was referenced Nov 1, 2024

[doc]update fileSource and valueStatsCols in manifest.md #4430

Closed

[doc]Update fileSource and valueStatsCols in manifest.md #4431

Merged

JingsongLi mentioned this pull request Dec 22, 2024

[Core]Add data file root location in DataFileMeta #4751

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] Introduce metadata.stats-dense-store to reduce meta size for multiple columns table #4322

[core] Introduce metadata.stats-dense-store to reduce meta size for multiple columns table #4322

JingsongLi commented Oct 14, 2024 •

edited

Loading

wwj6591812 commented Oct 14, 2024

wwj6591812 Oct 14, 2024

wwj6591812 Oct 14, 2024

JingsongLi Oct 15, 2024

wwj6591812 Oct 15, 2024

JingsongLi Oct 15, 2024

JingsongLi Oct 15, 2024

wwj6591812 Oct 14, 2024

wwj6591812 Oct 15, 2024

wwj6591812 Oct 15, 2024

wwj6591812 Oct 15, 2024

wwj6591812 Oct 15, 2024

wwj6591812 Oct 15, 2024

JingsongLi Oct 15, 2024

wwj6591812 Oct 15, 2024

wwj6591812 Oct 15, 2024

JingsongLi Oct 15, 2024

wwj6591812 Oct 15, 2024

JingsongLi Oct 15, 2024

wwj6591812 commented Oct 15, 2024

wwj6591812 commented Oct 15, 2024

wwj6591812 Oct 15, 2024

JingsongLi Oct 15, 2024

JingsongLi commented Oct 15, 2024

wwj6591812 commented Oct 15, 2024

[core] Introduce metadata.stats-dense-store to reduce meta size for multiple columns table #4322

[core] Introduce metadata.stats-dense-store to reduce meta size for multiple columns table #4322

Conversation

JingsongLi commented Oct 14, 2024 • edited Loading

Purpose

Tests

API and Format

Documentation

wwj6591812 commented Oct 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwj6591812 commented Oct 15, 2024

wwj6591812 commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingsongLi commented Oct 15, 2024

wwj6591812 commented Oct 15, 2024

JingsongLi commented Oct 14, 2024 •

edited

Loading