[fix](catalog) opt the count pushdown rule for iceberg/paimon/hive scan node (#44038) #45564
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…an node (apache#44038) 1. Opt the parallelism when doing count push down optimization Count push down optimization is used to optimize queries such as `select count(*) from table`. In this scenario, we can directly obtain the number of rows through the row count statistics of the external table, or the metadata of the Parquet/ORC file, without reading the actual file content, thereby speeding up such queries. Currently, we support count push down optimization for Hive, Iceberg, and Paimon tables. There are two ways to obtain the number of rows: 1. Obtain directly from statistics For Iceberg tables, we can obtain the number of rows directly from statistics. However, due to the historical issues of Iceberg, if there is position/equality delete in the table, this method cannot be used to prevent incorrect row count. In this case, it will degenerate to obtaining from the metadata of the file. 2. Obtain from the metadata of the file For Hive, Paimon, and some of Iceberg tables, the number of rows can be obtained directly from the metadata of the Parquet/ORC file. For Text format tables, efficiency can also be improved by only performing row separation, without column separation. In the task splitting logic, for Count push-down optimization, the number of split tasks should comprehensively consider the file format, number of files, parallelism, number of BE nodes, and the Local Shuffle: 1. Count push-down optimization should avoid Local Shuffle, so the number of split tasks should be greater than or equal to `parallelism * number of BE nodes`. 2. Fix the incorrect logic of Count push-down optimization In the previous code, for Iceberg and Paimon tables, Count push-down optimization did not take effect because we did not push CountPushDown information to FileFormatReader inside TableForamtReader. This PR fixes this problem. 3. Store SessionVaraible variables in FileQueryScanNode. SessionVaraible is a variable in ConnectionContext. And ConnectionContext is a ThreadLocal variable. In FileQueryScanNode, SessionVaraible may be accessed in other threads in some cases, so ThreadLocal variables may not be obtained. Therefore, the SessionVaraible reference is stored in FileQueryScanNode to prevent illegal access. 4. Independent FileSplitter class. The FileSplitter class is a tool class that allows users to split `Split` according to different strategies. This PR does not modify the splitting strategy, but only extracts this part of the logic separately, to be able to perform logic optimization later.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
bp #44038