[Bug] When using Flink for incremental queries（incremental-between tag）, the return result contains historical data with non incremental intervals

[Mr-j-yangyu]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.7.0

Compute Engine

Flink 1.17.2

Minimal reproduce step

一. Flink SQL Script for write data

flink-sql.txt

二. Write Step

1. start flink sql script, run for 2-3 hours（run from 2024-05-08 16:.. to 2024-05-08 19:..）
2. stop job and then restart（for simulate some data update scenarios）
3. Running all night（run from 2024-05-08 20:.. to 2024-05-09 10:..）

三. Query Step

query sql（return result: 172271）
select count(1) from tmp_order_tag /*+ OPTIONS('scan.parallelism' = '1','incremental-between'='2024-05-09 01,2024-05-09 02') */;

What doesn't meet your expectations?

in flink-sql.txt ，data generation speed: 25 rows/s， data increment：90000 / h

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

@Mr-j-yangyu Can you show the DDL?

[JingsongLi]

I found the SQL now.