From 8912509281299fcbc867020e97ec4dcf69a76586 Mon Sep 17 00:00:00 2001 From: jorritsandbrink Date: Thu, 15 Aug 2024 17:13:43 +0400 Subject: [PATCH] document delta table partitioning support --- .../dlt-ecosystem/destinations/filesystem.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/website/docs/dlt-ecosystem/destinations/filesystem.md b/docs/website/docs/dlt-ecosystem/destinations/filesystem.md index bba0ff3df3..3610b0ddf0 100644 --- a/docs/website/docs/dlt-ecosystem/destinations/filesystem.md +++ b/docs/website/docs/dlt-ecosystem/destinations/filesystem.md @@ -530,6 +530,24 @@ def my_delta_resource(): > `dlt` always uses `parquet` as `loader_file_format` when using the `delta` table format. Any setting of `loader_file_format` is disregarded. +#### Delta table partitioning +A Delta table can be partitioned ([Hive-style partitioning](https://delta.io/blog/pros-cons-hive-style-partionining/)) by specifying one or more `partition` column hints. This example partitions the Delta table by the `foo` column: + +```py +@dlt.resource( + table_format="delta", + columns={"foo": {"partition": True}} +) +def my_delta_resource(): + ... +``` + +:::caution + +It is **not** possible to change partition columns after the Delta table has been created. Trying to do so causes an error stating that the partition columns don't match. +::: + + #### Storage options You can pass storage options by configuring `destination.filesystem.deltalake_storage_options`: