Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[common][core][format] Support using file index result for RowGroup f… #4470

Closed
wants to merge 59 commits into from
Closed

[common][core][format] Support using file index result for RowGroup f… #4470

wants to merge 59 commits into from

Conversation

hang8929201
Copy link
Contributor

Purpose

Use file index result(bitmap) for more fine-grained IO clipping in the reader.

Tests

This is a query in our real business scenario.

  • no index:
    image
  • index for parquet block filtering:
    image
  • index for orc rowgroup filtering:
    image

API and Format

Documentation

guanshi and others added 30 commits November 7, 2024 11:03
[common][core][format] Support using file index result for RowGroup f…
c41fafa 
…iltering.
@FangYongs
[core] Support guava cache in CacheManager (apache#4389)
039ecea
@yunfengzhou-hub
[flink] fix false assumption on catalog table passing logic (apache#4381
3cea7f0 
)
@LinMingQiang
[flink] Procedure sorting compact a table without partitions filter. (a…
b527310 
…pache#4405)
@xuzifu666
[core] Check snapshot when reset consumer (apache#4373)
8a87c1c
@zhongyujiang
[core] support rename primary key columns and bucket key columns (apa…
6ade4de 
…che#3809)
@askwang
Change variable name in RecordLevelExpire (apache#4411)
150030c
@askwang
[core] Space allowed between multiple deleted tags (apache#4412)
6ea4063
@LinMingQiang
[cdc] Complementary flink exec mode check for mongodb and postgres cd…
948b9a0 
…c table sync. (apache#4408)
@askwang
[spark] Replace create existing tag semantic with replace_tag (apache…
2b31960 
…#4346)
@JingsongLi
[hive] Refactor HiveCatalog to reduce io when getTable (apache#4409)
6b389e5
@JingsongLi
[doc] Explain level upgrade in understand-files
37c9360
@LsomeYeah @tsreaper
[flink] add coordinate and worker operator for small changelog files …
3c4f7a1 
…compaction (apache#4380)

This closes apache#4380.

---------

Co-authored-by: tsreaper <[email protected]>
@JingsongLi
[doc] Move clone-tables to flink from migration
c44ef1d
@LsomeYeah
[fix] Fix error in FlinkSink for changelog files merge (apache#4415)
8697c7a 
This closes apache#4415.
@JingsongLi
[doc] Move filesystems doc to maintenance
25c8915
@YanZhangN
[hive] Pass 'partition.expiration-time' prop to hms. (apache#4416)
fa2f8b1
@reswqa
[core] Remove unused snapshot option for materialized table (apache#4421
80d2384 
)
@xuzifu666
[core] Introduce RollbackToTimestamp Procedure and Action (apache#4410)
eb1ade6
@wwj6591812
[flink] support speculative execution when batch read paimon table (a…
e88bc52 
…pache#4395)
@Aitozi
[core] Introduce BlockCache for SortLookupStoreReader (apache#4419)
ce925cc
@LsomeYeah
[core] Try to validate table for checking table options when creating…
8c286b0 
… table (apache#4404)
@JingsongLi
[core] Validate table before commit in SchemaManager
d548411
@JingsongLi
[core] Validate FileStoreTableFactory create fallback branch
5e72497
@herefree
[core] Support rename view (apache#4429)
fc94fe6
@JingsongLi
[flink] Adjust renameView after renameTable failed
6161388
@Edward-Gavin
[doc] update fileSource and valueStatsCols in manifest.md (apache#4431)
7b36096
@leaves12138
[core] Enable file index for DV table (apache#4310)
5841756
@JingsongLi
[hotfix] Minor refactor for datafile filter in KeyValueFileStoreScan
b2e4fe5
@JingsongLi
[doc] Add query performance to primary key table
34872b2
liming30 and others added 29 commits November 7, 2024 14:57
@liming30
[flink] support setting uid suffix for source/sink to improve state …
c0f4406 
…compatibility when jobGraph changes. (apache#4425)

* [flink] support setting uid suffix for source/sink to improve state compatibility when jobGraph changes.

* resolve comment
@askwang
[core] Add file compression type in file names (apache#4420)
c66922e
@Moonlight-CL
[cdc] Add AWS DMS CDC format support (apache#4433)
79e9333
@JingsongLi
[doc] Update cdc ingestion doc
de6eb42
@discivigour
[flink][spark] Add more branch tests (apache#3797)
fda9701
@BsoBird
[typo] fix typo (apache#4448)
9e77792 
rest => reset
@BsoBird
[doc] fix clone procedure doc style (apache#4449)
8ab44d5 
fix doc style
@mxdzs0612
[core] Allow cache and refresh partitions for CachingCatalog (apache#…
17b775f 
…4427)
@JingsongLi
[core] Introduce listPartitions interface to Catalog
f4a52c7
@zhuyaogai
[cdc] Fix ambiguous naming in CdcRecord. (apache#4450)
2630d78
@Aitozi
[core] Show fileSource in the FilesTable (apache#4443)
ec7643a
@liming30
[core] fix the issue where the historical schemaId could not be found…
269884b 
… when reading the branch table. (apache#4454)
@xiangyuf
[spark] Support table options via SQL conf for Spark Engine and unify…
b41b9c9 
… user experience with Flink engine (apache#4393)
@mxdzs0612
[core] Increase the default value of cache.expiration-interval to 10 …
d6b14ad 
…minute (apache#4456)
@askwang
[core] Support timestamp field type in record level expire (apache#4417)
0b4631f
@zhuangchong
[core] Refactor the Field Aggregator factory to validate data types w…
e4fe63f 
…hen creating function (apache#4446)
@Edward-Gavin
[doc] Update logOffsets in snapshot.md (apache#4444)
4dbf85e
@mxdzs0612
[core] Introduce refreshPartitions interface to CachingCatalog (apach…
a130feb 
…e#4455)
@JingsongLi
[hotfix] Make refreshPartitions in CachingCatalog public
ca7d343
@yuzelin
[flink] Lookup join supports time periods black list (apache#4413)
ee6f5f3
@JingsongLi
[flink] Extract RefreshBlacklist from FileStoreLookupFunction
cab2078
@JingsongLi
[flink] Make FileStoreLookupFunction.refreshBlacklist nullable to avo…
13e08b8 
…id performance regression
@askwang
[spark] Add replace tag ddl syntax (apache#4457)
09d1a5b
@zhuangchong
[core] The record level expired time field type is automatically reco…
1eb6a69 
…gnized. (apache#4458)
@xuzifu666
[core] Optizime IN filter pushdown to snapshot/tag/schema system tabl…
83a18d0 
…es (apache#4436)
@yuzelin
[doc] Add python sdk predicate doc (apache#4463)
c65ff11
@JingsongLi
[core] Delete useless methods in Catalog (apache#4468)
c9ecf27
@LsomeYeah
[core] throw exception if table is recreated when it still being read (…
dd166f4 
…apache#4445)

This closes apache#4445.
[flink] Add fileIndex method in anonymous implementation class of For…
d73f5e4 
…matReaderFactory.Context.
@hang8929201 hang8929201 closed this Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.