Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Introduce file index read/write framework. #3177

Merged
merged 29 commits into from
Apr 15, 2024

Conversation

leaves12138
Copy link
Contributor

@leaves12138 leaves12138 commented Apr 9, 2024

  • API:
CREATE TABLE <PAIMON_TABLE> (<COLUMN> <COLUMN_TYPE> , ...) WITH
(
"file-index.bloom-filter.columns" = "c1,c2",
"file-index.bloom-filter.c1.items" = "200",
"file-index.bloom-filter.c1.fpp" = "0.1",
"file-index.in-manifest-threshold" = "500 B" --default 500 B
)
  • GOAL

This pull request, introduce the read and write framework to paimon. If the file index bytes size is smaller than file.index.size-in-meta, WRITE PART will put them in DataFileMeta. Otherwise, put them in index file. see #3068.

  • Configurations
  1. file-index.<index-Type>.columns: column names those should be indexed.
  2. file-index.in-manifest-threshold: define the bytes size to store in DataFileMeta
  3. file-index.read.enabled: true or false. Default true. Set this to false, read stage will skip file index check.
  4. file-index.<index-Type>.<columnName>.<opKey>: set the property for specified index column.

@leaves12138 leaves12138 marked this pull request as draft April 9, 2024 02:21
@leaves12138 leaves12138 marked this pull request as ready for review April 9, 2024 02:31
@leaves12138 leaves12138 requested a review from JingsongLi April 9, 2024 02:31
@leaves12138 leaves12138 reopened this Apr 12, 2024
@leaves12138 leaves12138 requested a review from JingsongLi April 12, 2024 06:18
@leaves12138 leaves12138 requested a review from JingsongLi April 12, 2024 10:57
@leaves12138 leaves12138 requested a review from JingsongLi April 15, 2024 07:52
@leaves12138 leaves12138 requested a review from JingsongLi April 15, 2024 09:37
@JingsongLi
Copy link
Contributor

+1

@JingsongLi JingsongLi merged commit b3eeea9 into apache:master Apr 15, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants