[Bug] This is a bug! The partition org.apache.paimon.data.BinaryRow@741eefde and bucket 6 is filtered!

[yuanzhangjun]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.8.1

Compute Engine

Flink 1.15.1

Minimal reproduce step

The issue occurs when the Flink parallelism is adjusted from 1 to 4.

What doesn't meet your expectations?

Flink write failure.

Anything else?

2024-07-23 06:41:39.929 [Writer : paimon_order_binlog_append (4/4)#0] INFO org.apache.paimon.disk.FileChannelManagerImpl - FileChannelManager removed spill file directory /tmp/flink-io-d69031ff-26f2-46bb-bdb4-ec1a6674c892/paimon-io-2aaf502a-cdd6-4088-ac9a-26492cfa9c90
2024-07-23 06:41:39.932 [Writer : paimon_order_binlog_append (4/4)#0] WARN org.apache.flink.runtime.taskmanager.Task - Writer : paimon_order_binlog_append (4/4)#0 (350343d9845c1af762d14b3905800c80) switched from RUNNING to FAILED with failure cause: java.io.IOException: java.lang.IllegalArgumentException: This is a bug! The partition org.apache.paimon.data.BinaryRow@741eefde and bucket 6 is filtered!
at org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.processElement(CdcRecordStoreWriteOperator.java:92)
2024-07-23T06:41:39.932342753Z at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
2024-07-23T06:41:39.932346033Z at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
2024-07-23T06:41:39.932354030Z at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
2024-07-23T06:41:39.932360475Z at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
2024-07-23T06:41:39.932364156Z at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
2024-07-23T06:41:39.932367544Z at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
2024-07-23T06:41:39.932370840Z at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
2024-07-23T06:41:39.932374338Z at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
2024-07-23T06:41:39.932377493Z at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
2024-07-23T06:41:39.932380984Z at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
2024-07-23T06:41:39.932386449Z at java.lang.Thread.run(Thread.java:748)
2024-07-23T06:41:39.932390399Z Caused by: java.lang.IllegalArgumentException: This is a bug! The partition org.apache.paimon.data.BinaryRow@741eefde and bucket 6 is filtered!
at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:127)
2024-07-23T06:41:39.932397205Z at org.apache.paimon.operation.AbstractFileStoreScan.withPartitionBucket(AbstractFileStoreScan.java:154)
at org.apache.paimon.operation.AbstractFileStoreWrite.scanExistingFileMetas(AbstractFileStoreWrite.java:416)
2024-07-23T06:41:39.932403779Z at org.apache.paimon.operation.AbstractFileStoreWrite.createWriterContainer(AbstractFileStoreWrite.java:375)
2024-07-23T06:41:39.932407334Z at org.apache.paimon.operation.AbstractFileStoreWrite.lambda$getWriterWrapper$2(AbstractFileStoreWrite.java:350)
2024-07-23T06:41:39.932410784Z at java.util.HashMap.computeIfAbsent(HashMap.java:1128)
2024-07-23T06:41:39.932414024Z at org.apache.paimon.operation.AbstractFileStoreWrite.getWriterWrapper(AbstractFileStoreWrite.java:349)
2024-07-23T06:41:39.932417621Z at org.apache.paimon.operation.AbstractFileStoreWrite.write(AbstractFileStoreWrite.java:127)
2024-07-23T06:41:39.932431557Z at org.apache.paimon.table.sink.TableWriteImpl.writeAndReturn(TableWriteImpl.java:140)
2024-07-23T06:41:39.932434431Z at org.apache.paimon.flink.sink.StoreSinkWriteImpl.write(StoreSinkWriteImpl.java:175)
2024-07-23T06:41:39.932437324Z at org.apache.paimon.flink.sink.cdc.CdcRecordStoreWriteOperator.processElement(CdcRecordStoreWriteOperator.java:90)
2024-07-23T06:41:39.932440299Z ... 13 more
2024-07-23T06:41:39.932443055Z
2024-07-23T06:41:39.933006574Z 2024-07-23 06:41:39.932 [Writer : paimon_order_binlog_append (4/4)#0] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Writer : paimon_order_binlog_append (4/4)#0 (350343d9845c1af762d14b3905800c80).
2024-07-23 06:41:39.942 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FAILED to JobManager for task Writer : paimon_order_binlog_append (4/4)#0 350343d9845c1af762d14b3905800c80.
2024-07-23 06:41:40.003 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task Source: kafka source (1/4)#0 (2cad2e18342d7809ff3550a6c207cfff).
2024-07-23T06:41:40.003360093Z 2024-07-23 06:41:40.003 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskmanager.Task - Source: kafka source (1/4)#0 (2cad2e18342d7809ff3550a6c207cfff) switched from RUNNING to CANCELING.
2024-07-23 06:41:40.003 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: kafka source (1/4)#0 (2cad2e18342d7809ff3550a6c207cfff).
2024-07-23 06:41:40.007 [Source: kafka source (1/4)#0] INFO o.apache.flink.connector.base.source.reader.SourceReaderBase - Closing Source Reader.
2024-07-23 06:41:40.023 [Source: kafka source (1/4)#0] INFO o.a.flink.connector.base.source.reader.fetcher.SplitFetcher - Shutting down split fetcher 0
2024-07-23 06:41:40.023 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task Source: kafka source (3/4)#0 (ccf665e14ca5cae849b8748da6e9f671).
2024-07-23 06:41:40.024 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskmanager.Task - Source: kafka source (3/4)#0 (ccf665e14ca5cae849b8748da6e9f671) switched from RUNNING to CANCELING.
2024-07-23T06:41:40.024196257Z 2024-07-23 06:41:40.024 [flink-akka.actor.default-dispatcher-18] INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: kafka source (3/4)#0 (ccf665e14ca5cae849b8748da6e9f671).

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

Fixed in latest master.