New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] Optizime IN filter pushdown to snapshot/tag/schema system tables #4436

Merged

JingsongLi merged 10 commits into apache:master from xuzifu666:system_opt_in

Nov 6, 2024

Member

xuzifu666 commented Nov 4, 2024

Purpose

Linked issue: close #xxx

Currently IN filter cannot pushdown to snapshot/tag/schema system tables and IN would tramsform to OR(contians all Equal LeafPredicate), if user query with IN filter would cost more unnessary IO in a large number of snapshots/tag/schema;
TagTable predicate specified as LeafPredicate which cannot resolve other kind of Predicate, this pr had improved it.
Add APIs for get snapshots/tags/schemas which get multiple numbers for a list argument.

such like sql: select snapshot_id, schema_id, commit_user from T$snapshots where snapshot_id in (1, 3);
before the pr would query all snapshots files firstly, then filter with 1 and 3;
after the pr would query only snapshots file with 1 and 3, not query all snapshots file, which can reduce unnessary IO.
note: if filter contains other fields would handle regress as before( like: select snapshot_id, schema_id, commit_user from T$snapshots where snapshot_id in (1, 3) or/and commit_user='xxx'; ), just work only snapshot_id IN filter.

Tests

API and Format

Documentation

xuzifu666 and others added 7 commits

November 3, 2024 10:56


          [core] Support IN filter pushdown for snapshot and schema tables

da9010a


          fix imports

f8105e1

fix

5cb9b69


          support tag

116c04f

fix

e105876


          fix imports

de1c84d


          fix name

d1bfef2

xuzifu666 changed the title ~~System opt in~~ [core] Optizeme IN filter pushdown to snapshot/tag/schema system tables

xuzifu666 changed the title ~~[core] Optizeme IN filter pushdown to snapshot/tag/schema system tables~~ [core] Optizime IN filter pushdown to snapshot/tag/schema system tables

xuzifu666 closed this

xuzifu666 reopened this

xuzifu666 closed this

xuzifu666 reopened this

fix

42ec226

xuzifu666 closed this

xuzifu666 reopened this

wwj6591812 reviewed

View reviewed changes

paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java Outdated

@@ @@ -128,6 +128,10 @@ public List<TableSchema> listAll() { @@
                       return listAllIds().stream().map(this::schema).collect(Collectors.toList());
                   }
+                  public List<TableSchema> schemasWithIds(List<Long> schemaIds) {

Contributor

wwj6591812 Nov 4, 2024

schemasWithId

paimon-core/src/main/java/org/apache/paimon/table/system/SchemasTable.java Outdated

@@ @@ -203,6 +205,7 @@ private class SchemasRead implements InnerTableRead { @@
                       private Optional<Long> optionalFilterSchemaIdMax = Optional.empty();
                       private Optional<Long> optionalFilterSchemaIdMin = Optional.empty();
+                      private List<Long> schemaIds = new ArrayList<>();

Contributor

wwj6591812 Nov 4, 2024

final

paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java Outdated

                               .collect(Collectors.toList());
                   }
+                  public Iterator<Snapshot> snapshotsWithIds(List<Long> snapshotIds) {

Contributor

wwj6591812 Nov 5, 2024

snapshotsWithId

paimon-core/src/main/java/org/apache/paimon/table/system/SnapshotsTable.java Outdated

@@ @@ -206,6 +208,7 @@ private class SnapshotsRead implements InnerTableRead { @@
                       private RowType readType;
                       private Optional<Long> optionalFilterSnapshotIdMax = Optional.empty();
                       private Optional<Long> optionalFilterSnapshotIdMin = Optional.empty();
+                      private List<Long> snapshotIds = new ArrayList<>();

Contributor

wwj6591812 Nov 5, 2024

final

paimon-core/src/main/java/org/apache/paimon/table/system/SnapshotsTable.java

@@ @@ -226,6 +229,22 @@ public InnerTableRead withFilter(Predicate predicate) { @@
                                       handleLeafPredicate(leaf, leafName);
                                   }
                               }

Contributor

wwj6591812 Nov 5, 2024

List children = compoundPredicate.children();
if ((compoundPredicate.function()) instanceof And) {
xxx;
} else if (if ((compoundPredicate.function()) instanceof And) {) {
xxx;
} else {
throw xxx;
}

paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java Outdated

		TagManager tagManager = new TagManager(fileIO, location, branch);
		String leafName = "tag_name";

Contributor

wwj6591812 Nov 5, 2024

first : private static final TAG_NAME = "tag_name";
second : replace all "tag_name" to TAG_NAME in this class

paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java Outdated

+                                                          != null) {
+                                              String equalValue =
+                                                      ((LeafPredicate) leaf).literals().get(0).toString();
+                                              predicateMap.put(equalValue, tagManager.tag(equalValue));

Contributor

wwj6591812 Nov 5, 2024

if (tagManager.tagExists(equalValue)) {
predicateMap.put(equalValue, tagManager.tag(equalValue));
}

paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java Outdated

		TagManager tagManager = new TagManager(fileIO, location, branch);
		String leafName = "tag_name";

Contributor

wwj6591812 Nov 5, 2024

first : private static final TAG_NAME = "tag_name";
second : replace all "tag_name" to TAG_NAME in this class

paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java Outdated

+                                      List<Predicate> children = compoundPredicate.children();
+                                      for (Predicate leaf : children) {
+                                          if (leaf instanceof LeafPredicate
+                                                  && (((LeafPredicate) leaf).function() instanceof Equal)

Contributor

wwj6591812 Nov 5, 2024

Why here not judge:
((LeafPredicate) predicate).literals().get(0) instanceof BinaryString
?

paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java Outdated

+                                                          != null) {
+                                              String equalValue =
+                                                      ((LeafPredicate) leaf).literals().get(0).toString();
+                                              predicateMap.put(equalValue, tagManager.tag(equalValue));

Contributor

wwj6591812 Nov 5, 2024

if (tagManager.tagExists(equalValue)) {
predicateMap.put(equalValue, tagManager.tag(equalValue));
}


          address

2bc72cb

Member Author

xuzifu666 commented Nov 5, 2024

@wwj6591812 Thanks review，had addressed.

fix

1dcf28f

Contributor

wwj6591812 commented Nov 5, 2024

+1

1 similar comment

Contributor

LinMingQiang commented Nov 6, 2024

+1

JingsongLi reviewed

View reviewed changes

paimon-core/src/main/java/org/apache/paimon/table/system/SchemasTable.java

@@ @@ -223,6 +226,22 @@ public InnerTableRead withFilter(Predicate predicate) { @@
                                       handleLeafPredicate(leaf, leafName);
                                   }
                               }
+                              // optimize for IN filter
+                              if ((compoundPredicate.function()) instanceof Or) {

Contributor

JingsongLi Nov 6, 2024

We should extract a class to reduce duplicate codes, this can be done in a new PR.

Member Author

xuzifu666 Nov 6, 2024

OK，would refactor it as suggest，Thanks @JingsongLi

JingsongLi approved these changes

View reviewed changes

Contributor

JingsongLi left a comment

+1

JingsongLi merged commit 321c8ef into apache:master

12 checks passed

xuzifu666 mentioned this pull request

[core] Introduce PredicateUtils to refactor Predicate handle #4469

Closed

hang8929201 pushed a commit to hang8929201/paimon that referenced this pull request


          [core] Optizime IN filter pushdown to snapshot/tag/schema system tabl…

83a18d0

…es (apache#4436)

xuzifu666 mentioned this pull request

[core] Introduce InPredicateVisitor to refactor Predicate handle #4486

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet