[Bug] spark deletes dynamic bucket table, the result is incorrect

[hekaifei]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.7

Compute Engine

spark

Minimal reproduce step

What doesn't meet your expectations?

For dynamic bucket tables, flink does not support delet. But spark can do it, but the result is incorrect.

I think the error should be reported in advance or it can be repaired and the result will be correct.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Zouxxyy]

@hekaifei Thank you for pointing it out, can you provide more detailed steps to reproduce it, such as table creation and data insert statement

[hekaifei]

@Zouxxyy

1. create table , I use the hive client

CREATE TABLE dynamic_bucket_tests (
id BIGINT,
job_group BIGINT,
job_id BIGINT ,
ums_ts_ BIGINT)  COMMENT ''  
STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'  
TBLPROPERTIES(
 'bucket' = '-1',
 'consumer.expiration-time' = '1h', 
'file.format' = 'parquet', 
'metastore.partitioned-table' = 'true', 
'primary-key' = 'id', 
'sequence.field' = 'ums_ts_');

2. insert data , I use spark sql

spark.read.json("dynamic_bucket_tests.json").repartition(10).createOrReplaceTempView("t1") 
spark.sql("""insert into dynamic_bucket_tests select * from t1""")

dynamic_bucket_tests.json

3. try to delete

spark-sql> delete from dynamic_bucket_tests  where id  =55233981; Time taken: 11.076 seconds    
spark-sql>  > select * from dynamic_bucket_tests  where id  =55233981; Time taken: 3.362 seconds 
spark-sql> delete from dynamic_bucket_tests  where id  =55235391; Time taken: 7.803 seconds
spark-sql> select * from dynamic_bucket_tests  where id  =55235391; 
                   55235391        65      59662   16995456000000 
                 Time taken: 2.226 seconds, Fetched 1 row(s)

[hekaifei]

@Zouxxyy Maybe it's a paimon version issue. I used paimon0.6-spark3.2 to write data and paimon0.7-spark3.2 for deletion.

[Zouxxyy]

@hekaifei Thanks, there are some bug between 'sequence.field' and row lever op, I'll take it

[hekaifei]

@Zouxxyy This problem also occurs when update is found