Iceberg to Delta conversion. File stats not populated.

[haider24]

Hello,

I am converting an Iceberg format to Delta on S3. While checking the Iceberg metadata, I noticed that the file-level stats are populated. However, in the converted Delta metadata, the stats are empty. Is there a specific configuration I need to enable to ensure the file-level stats are converted?

[the-other-tim-brown]

@haider24 there are no special steps and will be translated by default. Can you provide the steps to reproduce the source table so we can check why the stats are not being read?

[haider24]

@the-other-tim-brown
Here is the dataset conf:

sourceFormat: ICEBERG
targetFormats:
  - DELTA
datasets:
  - tableBasePath: s3://bucket_name/table_path/
    tableDataPath: s3://bucket_name/table_path/data/
    tableName: test_table
    namespace: DS_STAGING.ICEBERG_POC

Iceberg table is registered in snowflake.
Catalog Config:

catalogImpl: org.apache.iceberg.snowflake.SnowflakeCatalog
catalogName: SNOWFLAKE
catalogOptions:
  io-impl: org.apache.iceberg.aws.s3.S3FileIO
  warehouse: s3://BUCKET_NAME/warehouse
  uri: jdbc:snowflake://tt12345.eu-west-1.snowflakecomputing.com
  jdbc.user: SA_USER
  jdbc.private_key_file: sf_pkey.pem
  jdbc.role: SA_USER_ROLE

Command to convert the metadata:

java -cp \
    jars/iceberg-spark-runtime-3.2_2.12-1.4.2.jar:jars/xtable-utilities_2.12-0.2.0-incubating-bundled:jars/snowflake-jdbc-3.13.28.jar:jars/iceberg-aws-1.4.2.jar:jars/bundle-2.23.9.jar \
    org.apache.xtable.utilities.RunSync  \
    --datasetConfig xtable/dataset_config.yml \
    --icebergCatalogConfig xtable/catalog_config.yml 

[the-other-tim-brown]

@haider24 what created the table though? We are reading the file details including stats using Iceberg libraries so I need to see how I can reproduce this case to figure out why the metadata is not being read properly for this particular case

[haider24]

@the-other-tim-brown I am using snowflake managed iceberg tables
https://docs.snowflake.com/en/user-guide/tables-iceberg

[haider24]

@the-other-tim-brown I created a test Iceberg table using AWS Wrangler, converted the metadata to Delta, and noticed that the stats were not translated in this case either.

Python code to generate the test table:

import awswrangler as wr
import pandas as pd

data = {
    "Name": ["Alice", "Bob", "Charlie", "David", "Eve"],
    "Age": [25, 30, 35, 40, 28],
    "City": ["New York", "Los Angeles", "Chicago", "Houston", "Phoenix"],
    "Score": [85.5, 90.0, 78.5, 92.0, 88.0],
}

df = pd.DataFrame(data)

result = wr.athena.to_iceberg(
    df =df,
    database='glue_database_name',
    table='test_iceberg_table',
    table_location='s3://bucket_name/test_iceberg_table',
    # mode='overwrite',
    temp_path='s3://bucket_name/test_iceberg_table_temp'
)

print(result)

Xtable catalog conf:

catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
catalogName: GLUE
catalogOptions:
  io-impl: org.apache.iceberg.aws.s3.S3FileIO

Dataset conf:

sourceFormat: ICEBERG
targetFormats:
  - DELTA
datasets:
  - tableBasePath: s3://bucket_name/test_iceberg_table/
    tableDataPath: s3://bucket_name/test_iceberg_table/data/
    namespace: glue_database_name
    tableName: test_iceberg_table

Command:

java -cp \
    jars/iceberg-spark-runtime-3.2_2.12-1.4.2.jar:jars/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:jars/snowflake-jdbc-3.13.28.jar:jars/iceberg-aws-1.4.2.jar:jars/bundle-2.23.9.jar \
    org.apache.xtable.utilities.RunSync  \
    --datasetConfig xtable/dataset_config.yml \
    --icebergCatalogConfig xtable/catalog_config.yml

[the-other-tim-brown]

@ashvina I see that there is still a TODO around adding testing for column stats in Iceberg. https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/iceberg/TestIcebergConversionTargetSource.java#L163

Do you know how we can add this so we can more easily test what may be causing this?

[the-other-tim-brown]

I have created this Bug ticket and will work on a fix: #615

[the-other-tim-brown]

#616