[Flink] Support speculative execution when batch read paimon table. #4395
Conversation
wwj6591812
changed the title
| @@ -41,7 +42,8 @@ | |||
|
|
|||
| /** A {@link SplitEnumerator} implementation for {@link StaticFileStoreSource} input. */ | |||
| public class StaticFileStoreSplitEnumerator | |||
| implements SplitEnumerator<FileStoreSourceSplit, PendingSplitsCheckpoint> { | |||
| implements SplitEnumerator<FileStoreSourceSplit, PendingSplitsCheckpoint>, | |||
| SupportsHandleExecutionAttemptSourceEvent { | |||
There was a problem hiding this comment.
You may need to introduce classes for Flink 1.15.
There was a problem hiding this comment.
No, it is good to keep Flink 1.15 support.
wwj6591812
force-pushed
the
support_speculative_interface_1029
branch
from
2e799fe to
766d8d2
Compare
wwj6591812
changed the title
There was a problem hiding this comment.
A simpler solution for 1.15, you can just create a class org.apache.flink.api.connector.source.SupportsHandleExecutionAttemptSourceEvent in paimon-flink-1.15.
wwj6591812
force-pushed
the
support_speculative_interface_1029
branch
from
766d8d2 to
35b595c
Compare
Good idea, done. |
wwj6591812
changed the title
| // Only recognize events that don't care attemptNumber | ||
| handleSourceEvent(subtaskId, sourceEvent); | ||
| } | ||
|
|
||
| @Override | ||
| public void handleSourceEvent(int subtaskId, SourceEvent sourceEvent) { | ||
| if (sourceEvent instanceof ReaderConsumeProgressEvent) { |
There was a problem hiding this comment.
Maybe add a comment to note that when to support a new kind of event, one needs to pay attention that whether the event can be sent multiple times from different attempts of one subtask. See the description of SupportsHandleExecutionAttemptSourceEvent for more details.
There was a problem hiding this comment.
In case someone later adds a new kind of event directly to this method, ignoring the method with attemptNumber.
wwj6591812
force-pushed
the
support_speculative_interface_1029
branch
2 times, most recently
from
c60a5b5 to
97f7845
Compare
wwj6591812
force-pushed
the
support_speculative_interface_1029
branch
from
97f7845 to
e00ebc7
Compare
|
+1 Thanks @wwj6591812 and @zhuzhurk |
Purpose
In my company's production job, when batch read a paimon table, I receive the exception.
After read flink and paimon code, I know it means paimon use custom SourceEvent ReaderConsumeProgressEvent.
I have come up with two solutions:
1、Implement the interface SupportsHandleExecutionAttemptSourceEvent according to the exception suggestion;
2、Delete ReaderConsumeProgressEvent in StaticFileStoreSplitEnumerator.
Due to the possibility of new other SourceEvent being added in the future, so I chose the solutions-1.
Linked issue: close #xxx
Tests
API and Format
Documentation