branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 #44438
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…cate (#43255) ### What problem does this PR solve? Problem Summary: This issue addresses a limitation in Apache Doris where only predicates joined by AND are pushed down to the ORC reader, leaving OR-connected predicates unoptimized. By extending pushdown functionality to handle these OR conditions, the aim is to better leverage ORC’s predicate pushdown capabilities, reducing data reads and improving query performance.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
| @@ -461,8 +467,10 @@ static std::unordered_map<orc::TypeKind, orc::PredicateDataType> TYPEKIND_TO_PRE | |||
| {orc::TypeKind::BOOLEAN, orc::PredicateDataType::BOOLEAN}}; | |||
|
|
|||
| template <PrimitiveType primitive_type> | |||
| std::tuple<bool, orc::Literal> convert_to_orc_literal(const orc::Type* type, const void* value, | |||
| int precision, int scale) { | |||
| std::tuple<bool, orc::Literal> convert_to_orc_literal(const orc::Type* type, | |||
There was a problem hiding this comment.
warning: function 'convert_to_orc_literal' exceeds recommended size/complexity thresholds [readability-function-size]
std::tuple<bool, orc::Literal> convert_to_orc_literal(const orc::Type* type,
^Additional context
be/src/vec/exec/format/orc/vorc_reader.cpp:469: 94 lines including whitespace and comments (threshold 80)
std::tuple<bool, orc::Literal> convert_to_orc_literal(const orc::Type* type,
^| // check if there are rest children of expr can be pushed down to orc reader | ||
| bool OrcReader::_check_rest_children_can_push_down(const VExprSPtr& expr) { | ||
| if (expr->children().size() < 2) { | ||
| return false; |
There was a problem hiding this comment.
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/vec/exec/format/orc/vorc_reader.cpp:639:
- if (expr->children().size() < 2) {
- return false;
- }
-
- for (size_t i = 1; i < expr->children().size(); ++i) {
- if (!expr->children()[i]->is_literal()) {
- return false;
- }
- }
- return true;
+ return !expr->children().size() < 2;| return true; | ||
| } | ||
|
|
||
| bool OrcReader::_build_search_argument(const VExprSPtr& expr, |
There was a problem hiding this comment.
warning: function '_build_search_argument' exceeds recommended size/complexity thresholds [readability-function-size]
bool OrcReader::_build_search_argument(const VExprSPtr& expr,
^Additional context
be/src/vec/exec/format/orc/vorc_reader.cpp:772: 115 lines including whitespace and comments (threshold 80)
bool OrcReader::_build_search_argument(const VExprSPtr& expr,
^| @@ -18,9 +18,9 @@ | |||
| #pragma once | |||
|
|
|||
| #include <cctz/time_zone.h> | |||
There was a problem hiding this comment.
warning: 'cctz/time_zone.h' file not found [clang-diagnostic-error]
#include <cctz/time_zone.h>
^| @@ -17,20 +17,9 @@ | |||
|
|
|||
| #include "testutil/desc_tbl_builder.h" | |||
There was a problem hiding this comment.
warning: 'testutil/desc_tbl_builder.h' file not found [clang-diagnostic-error]
#include "testutil/desc_tbl_builder.h"
^| @@ -20,15 +20,16 @@ | |||
|
|
|||
| #include <gen_cpp/Descriptors_types.h> | |||
There was a problem hiding this comment.
warning: 'gen_cpp/Descriptors_types.h' file not found [clang-diagnostic-error]
#include <gen_cpp/Descriptors_types.h>
^| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #include <glog/logging.h> |
There was a problem hiding this comment.
warning: 'glog/logging.h' file not found [clang-diagnostic-error]
#include <glog/logging.h>
^
|
TeamCity be ut coverage result: |
…r OR-connected Predicate apache#43255 (apache#44438)" This reverts commit dceaf97.
Cherry-picked from #43255