Iceberg Support Roadmap

[kevinzwang]

Features

• Basic reads
• Partitioned reads
• Basic writes
• Merge-on-read position delete reads
• Merge-on-read equality delete reads
  • blocked by pyiceberg
• Utilizing statistics from table format
• [PERF] [Catalogs] Push partition filter expressions into catalog file URI fetches #1953
• Field IDs
• How to filter by bucket in bucket partitioned Iceberg table #2105
• Support partition evolution (old files having different partitoning schemes vs new files) #2249
• Partitioned writes
• Upserts/MERGE
  • Copy-on-write writing
  • Merge-on-read writing
    • blocked by pyiceberg
• Add support for writing new Iceberg tables to catalogs such as AWS Glue and Iceberg REST

Bugs and Testing

• Reading only the partition column from an Iceberg/Delta table fails #2129
• Fix Iceberg round-trip support #2459
• Add end-to-end tests for reading+writing #2348

[corleyma]

@kevinzwang I am a bit confused about what copy-on-write writing represents in this set of features? Typically (at least coming from a Spark lens), copy-on-write is the default behavior for Iceberg v2 tables. Given that Daft supports basic and partitioned rights, what does it mean to say Daft does not have support for either COW or MOR writing? It has to be one or the other, right?

[kevinzwang]

@kevinzwang I am a bit confused about what copy-on-write writing represents in this set of features? Typically (at least coming from a Spark lens), copy-on-write is the default behavior for Iceberg v2 tables. Given that Daft supports basic and partitioned rights, what does it mean to say Daft does not have support for either COW or MOR writing? It has to be one or the other, right?

Hi @corleyma! COW and MOR are two different strategies we would need to support in order to do upsert (merge) operations in Iceberg. We currently only support append or overwrite for writing to Iceberg tables, which do not need either.

I've updated the roadmap to better reflect this.