[connector] Fluss sink supports ignore_delete. #272
Conversation
loserwang1024
force-pushed
the
delete_strategy
branch
from
dbf95db to
2dc0437
Compare
...-connector-flink/src/main/java/com/alibaba/fluss/connector/flink/options/DeleteStrategy.java
Outdated
Show resolved
Hide resolved
...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java
Show resolved
Hide resolved
...nnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSinkITCase.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Thanks @luoyuxia and @loserwang1024 , I left some comments about the config.
...-connector-flink/src/main/java/com/alibaba/fluss/connector/flink/sink/FlinkSinkFunction.java
Outdated
Show resolved
Hide resolved
| flussConfig, | ||
| tableRowType, | ||
| targetColumnIndexes, | ||
| deleteStrategy) |
There was a problem hiding this comment.
Add a test for primary key table as well.
...uss-connector-flink/src/main/java/com/alibaba/fluss/connector/flink/sink/FlinkTableSink.java
Outdated
Show resolved
Hide resolved
| .withDescription( | ||
| "This field is used to decide what to do when data of type -D/-U is received. " | ||
| + "`IGNORE_DELETE` means ignoring the `-D` and `-U` type message. " | ||
| + "`CHANGELOG_STANDARD` means neither `-U` nor `-D` is ignored, they both cause the corresponding row in fluss to be deleted"); |
There was a problem hiding this comment.
Personally, I prefer sink.ignore-delete=true/false. If there are only 2 values and CHANGELOG_STANDARD is the default, I think sink.ignore-delete=true/false is much simpler for users to understand and use.
It seems sink.delete-strategy comes from here that contains additional values NON_PK_FIELD_TO_NULL and DELETE_ROW_ON_PK. However, Fluss already handles partial-updates and partial-delete with leveraging the INSERT INTO (columns) grammar and I don't see this config will extend additional values in the near future.
There was a problem hiding this comment.
It seems sink.delete-strategy comes from here that contains additional values NON_PK_FIELD_TO_NULL and DELETE_ROW_ON_PK.
Yes, I don't know whether user want to partial update(-D) not delete the whole row but also not ignore , just set the non pk columns as null.
loserwang1024
force-pushed
the
delete_strategy
branch
from
0c4b74d to
51cb5e6
Compare
|
rebase into main branch |
|
@loserwang1024 rebase but not squash commits in the future, otherwise, reviewer is hard to track what is changed since last time and have to review all the code again. |
|
|
||
| /** The strategy when fluss sink receives delete data. */ | ||
| @PublicEvolving | ||
| public enum DeleteStrategy implements Serializable { |
loserwang1024
force-pushed
the
delete_strategy
branch
from
078ed2c to
60931e0
Compare
|
Let me merge it to reduce the gap between my test branch and main bach... |
Purpose
Linked issue: close #251