[Bug] spark integrates paimon, and the write empty array becomes an array containing null

[xyk0930]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.9

Compute Engine

spark3.5.1

Minimal reproduce step

1. Spark generates a dataset where the chronic_list column is of type List
2. Call dataset.show(), empty array display []
3. But when I call dataset.write().mode(SaveMode.Append).format("paimon").save(path) to a piamon table, the query shows [null]

What doesn't meet your expectations?

空数组应该是[]而不是[null]

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Aiden-Dong]

Could you provide me with a sample test code? I want to reproduce this situation.

[Aiden-Dong]

your table store format is orc or parquet ?

[xyk0930]

parquet and use zstd file compression

[Aiden-Dong]

Provide a sample test?

[xyk0930]

1.create a paimon table
CREATE TABLE paimon_default.default_array_test
(
id INT COMMENT '唯一标识',
name STRING COMMENT '姓名',
tags ARRAY COMMENT '标签'
)
USING paimon
COMMENT '数组test'
TBLPROPERTIES (
'bucket' = '-1',
'changelog-producer' = 'none',
'deletion-vectors.enabled' = 'false',
'dynamic-bucket.initial-buckets' = '10',
'dynamic-bucket.target-row-num' = '2000000',
'file.compression' = 'zstd',
'file.compression.zstd-level' = '1',
'file.format' = 'parquet',
'full-compaction.delta-commits' = '1',
'ignore-delete' = 'false',
'merge-engine' = 'deduplicate',
'path' = 'hdfs://hadoop105:8020/paimon/warehouse/paimon_default.db/default_array_test',
'primary-key' = 'id',
'snapshot.expire.limit' = '10',
'snapshot.num-retained.max' = '10',
'snapshot.num-retained.min' = '3',
'snapshot.time-retained' = '1 h',
'tag.num-retained-max' = '7')
;
2.use spark write data into paimon table
public class PaimonArrayTest {

public static void main(String[] args) {
    // 初始化 SparkSession 并启用 Hive 支持
    SparkSession spark = SparkSession.builder()
            .appName("Spark Paimon Example")
            .config("spark.sql.catalog.spark_catalog", "org.apache.paimon.spark.SparkGenericCatalog")
            .config("spark.sql.extensions", "org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions")
            .enableHiveSupport()
            .getOrCreate();

    // 定义 Schema
    StructType schema = new StructType(new StructField[]{
            DataTypes.createStructField("id", DataTypes.IntegerType, false),
            DataTypes.createStructField("name", DataTypes.StringType, false),
            DataTypes.createStructField("tags", DataTypes.createArrayType(DataTypes.StringType), false)
    });

    // 创建一些数据行
    List<Row> data = Arrays.asList(
            RowFactory.create(1, "Alice", Arrays.asList("Java", "Scala")),
            RowFactory.create(2, "Bob", Arrays.asList("Python", "R")),
            RowFactory.create(3, "Charlie", new ArrayList<>())
    );

    // 创建 DataFrame
    Dataset<Row> df = spark.createDataFrame(data, schema);

    // 显示 DataFrame 内容
    df.show();

    // 将 DataFrame 写入 paimon 表
    df.write().mode(SaveMode.Overwrite).format("paimon").save("hdfs://hadoop105:8020/paimon/warehouse/paimon_default.db/default_array_test"); // 替换为实际的数据库和表名

    spark.table("paimon_default.default_array_test").show();
    // 关闭 SparkSession
    spark.stop();
}

}
3.driver stdout

@Aiden-Dong this is a sample test

[xyk0930]

Write a piece of data using sparksql
insert into table paimon_default.default_array_test select 4,'Array',array();
The same goes for query results

[Aiden-Dong]

I will try to fix this issue.

[Aiden-Dong]

I couldn't reproduce this issue on the master branch. You can try a higher version.

[xyk0930]

Ok, I'm waiting for the 1.0 release

[xyk0930]

1.0 release Fixed the bug