Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[flink] Support minor compact strategy for dedicated compaction action. #4589

Merged
merged 4 commits into from
Nov 27, 2024
Merged

[flink] Support minor compact strategy for dedicated compaction action. #4589

merged 4 commits into from
Nov 27, 2024

Conversation

LinMingQiang
Copy link
Contributor

@LinMingQiang LinMingQiang commented Nov 26, 2024 •
edited
Loading

Purpose

Linked issue: (#4566)

Tests


- `--compact_strategy` Determines how to pick files to be merged, the default is determined by the runtime execution mode, streaming-mode use minor strategy and batch-mode use full strategy.

- `full `: Only supports batch mode. All files will be selected for merging.

- `minor `: Pick the set of files that need to be merged based on specified conditions.

API and Format

Documentation

JingsongLi
JingsongLi reviewed Nov 26, 2024
View reviewed changes
Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also modify Flink & Spark procedures?

@LinMingQiang
Copy link
Contributor Author

Can you also modify Flink & Spark procedures?

I plan to complete this in another PR.

Plan to add a new procedure minor_compact, and provide some new parameters like
num-sorted-run.compaction-trigger / compaction.size-ratio.

@LinMingQiang LinMingQiang force-pushed the master-feat-4566-step-2 branch from ea69e54 to ee89c4c Compare November 27, 2024 02:24
@LinMingQiang LinMingQiang force-pushed the master-feat-4566-step-2 branch 4 times, most recently from 95df419 to e0c24be Compare November 27, 2024 08:28
@LinMingQiang LinMingQiang reopened this Nov 27, 2024
@LinMingQiang LinMingQiang force-pushed the master-feat-4566-step-2 branch from e0c24be to 8735d54 Compare November 27, 2024 10:11
JingsongLi
JingsongLi approved these changes Nov 27, 2024
View reviewed changes
Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 4bcf857 into apache:master Nov 27, 2024
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wwj6591812 wwj6591812 wwj6591812 left review comments

@JingsongLi JingsongLi JingsongLi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@LinMingQiang @JingsongLi @wwj6591812