Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] Support default aggregate function for partial update and aggregate merge function #3374

Merged
merged 2 commits into from
May 24, 2024
Merged

[core] Support default aggregate function for partial update and aggregate merge function #3374

merged 2 commits into from
May 24, 2024

Conversation

FangYongs
Copy link
Contributor

@FangYongs FangYongs commented May 22, 2024 •
edited
Loading

Fix update null value for partial update with sequence-group

Purpose

Linked issue: close #3373

Tests

  1. Added test unit PartialUpdateMergeFunctionTest#testSequenceGroupDefaultAggFunc
  2. Added test unit PartialUpdateITCase#testSequenceGroupWithDefaultAggFunc and PartialUpdateITCase#testPartialUpdateWithDefaultAndFieldAggregation

API and Format

no

Documentation

no

@JingsongLi
Copy link
Contributor

See https://paimon.apache.org/docs/master/primary-key-table/merge-engine/#sequence-group

So we introduce sequence group mechanism for partial-update tables. It can solve:

  • Disorder during multi-stream update. Each stream defines its own sequence-groups.
  • A true partial-update, not just a non-null update.

True partial-update, means null can update column.

@FangYongs FangYongs changed the title [partial-update] Dont partial old field to null when the field value is null for sequence-group [partial-update] Support default aggregate function for all fields in partial update May 22, 2024
@FangYongs FangYongs changed the title [partial-update] Support default aggregate function for all fields in partial update [core] Support default aggregate function for all fields in partial update May 22, 2024
@FangYongs FangYongs force-pushed the do-not-partial-update-null-field branch from b10a9f4 to eba13c4 Compare May 22, 2024 11:59
@FangYongs
Copy link
Contributor Author

@JingsongLi As we discussed, I add fields.default-aggregate-function for partial update with sequence group, please help to review this PR, thanks cc @liming30

@FangYongs FangYongs changed the title [core] Support default aggregate function for all fields in partial update [core] Support default aggregate function for all fields in partial update and aggregate merge function May 23, 2024
@FangYongs FangYongs changed the title [core] Support default aggregate function for all fields in partial update and aggregate merge function [core] Support default aggregate function for partial update and aggregate merge function May 23, 2024
liming30
liming30 approved these changes May 24, 2024
View reviewed changes
Copy link
Contributor

@liming30 liming30 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1，after adding this, users can avoid manually configuring agg functions one by one.

JingsongLi
JingsongLi approved these changes May 24, 2024
View reviewed changes
Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 264180d into apache:master May 24, 2024
10 checks passed
sunxiaojian pushed a commit to sunxiaojian/paimon that referenced this pull request May 28, 2024
@FangYongs @sunxiaojian
[core] Support default aggregate function for partial update and aggr…
1dacbb1 
…egate merge function (apache#3374)
joyCurry30 pushed a commit to joyCurry30/paimon that referenced this pull request May 30, 2024
@FangYongs @joyCurry30
[core] Support default aggregate function for partial update and aggr…
bef29ae 
…egate merge function (apache#3374)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liming30 liming30 liming30 approved these changes

@JingsongLi JingsongLi JingsongLi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Support default aggregate function for partial update
3 participants
@FangYongs @JingsongLi @liming30