The support of MS Fabric (tech stack)

[boje]

Hi
The new boy in class Microsoft Fabric is here. Fabric uses OneLake and is interesting to many because of the performance and DirectLake connection from PowerBI to OneLake.

1. Does OneTable support the OneLake storage?
2. To get high performance Microsoft add V-ordering (proprietary) to its DeltaLake files. Will OneTable support this in the future? I guess is known, as this is a big secret from MS, not open-source.

[kywe665]

Thanks for adding the great question @boje! Here is a link to documentation of how to use OneTable with Fabric: https://onetable.dev/docs/fabric

In general anywhere Delta, Iceberg, or Hudi are supported, OneTable will work. So while I don't know the intimate details of OneLake in and out, OneTable works well on ADLS as shown in the docs and this live demo: https://opensourcedatasummit.com/open-data-foundations-with-onetable/.

V-ordering, I think @ashvina or @the-other-tim-brown might want to take a peek at? @boje can you share details or docs on how v-order is implemented or represented inside the Delta Lake metadata?

[boje]

As it is proprietary, we are still determining exactly how it is implemented, but as far as I know, it is implemented directly in the Delta Lake format. If it is by adding more metadata or other compression algo for the parquet file, that is an MS secret.
This doc tells a bit of what it is: https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql

My interest goes into the 10-50% better performance. We all want that, but as far as I know, you only get this IF you optimize or write the files with a Fabric compute engine. That is the struggle for Databricks, Snowflake, and others. They can write data to OneLake, but the performance from PowerBI will not be optimal.

[ashvina]

@boje OneTable translation only generates table metadata, not its data. If I understand correctly, V-order is a different process that optimizes the parquet file format. OneTable does not access or modify a table's data.