Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[format][orc] open orc switch useSelected,allowSARGToFilter to make sure pushdown works #4231

Merged
merged 9 commits into from
Nov 12, 2024

Conversation

ranxianglei
Copy link
Contributor

Open orc switch useSelected,allowSARGToFilter to make sure pushdown works,
To reduce query response time.

Purpose

Linked issue: close #xxx

Tests

API and Format

Documentation

@JingsongLi
Copy link
Contributor

Tests failed in OrcReaderFactoryTest.

@ranxianglei
Copy link
Contributor Author

@JingsongLi Wait for me fix up. Need time to test more !

@ranxianglei
Copy link
Contributor Author

Attention :
If open OrcConf.READER_USE_SELECTED and OrcConf.ALLOW_SARG_TO_FILTER,then query rt may be reduced several times, but the deletion vectors future would not work , cased by getRowNumber would be changed .

@ranxianglei
Copy link
Contributor Author

hold for while .

@ranxianglei ranxianglei reopened this Oct 15, 2024
@ranxianglei
Copy link
Contributor Author

ranxianglei commented Oct 15, 2024

Chinese:保持这些参数打开,或许可以获得几倍到几十倍的查询效率提升,尤其是针对大orc文件小数据量查询的场景。注意,由于删除向量功能强依赖于orc文件的row顺序,开启此参数后不保证row顺序不变,从而建议禁用删除向量功能,以在olap场景获得巨大性能提升。
English by google translate:By keeping these parameters turned on, you may be able to improve query efficiency by several to dozens of times, especially for scenarios where you query large orc files with small amounts of data. Note that since the deletion vector function strongly depends on the row order of the orc file, turning this parameter on does not guarantee that the row order remains unchanged. Therefore, it is recommended to disable the deletion vector function to obtain huge performance improvements in OLA scenarios.

'orc.reader.filter.use.selected'='true',
'orc.reader.sarg.to.filter'='true'

@ranxianglei
Copy link
Contributor Author

close this, see #4497

@ranxianglei ranxianglei reopened this Nov 12, 2024
@ranxianglei ranxianglei reopened this Nov 12, 2024
Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants