HTAP OLAP Spark connector?

[praiaJaeyoungChung]

Hi guys,

When CosmosDB is enabled for columnar storage for HTAP, does the Spark connector leverage OLAP?

The documentation on Spark connector seems to mention OLTP only, am wondering if Azure or anybody provides OLAP enabled Spark connector.

Thank you!

References:

(mentions OLTP only)

• https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-spark?tabs=python#bk_working_with_connector
• https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3-1_2-12

[FabianMeiswinkel]

The Spark connector in this repo is only for the OLTP Spark connector. The connector for the columnar store (The OLAP Spark connector) is pre-installed in Synapse (as is the OLTP connector).

This document provides an overview on how to query the analytic store from Spark in Synapse: https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-link/how-to-query-analytical-store-spark-3?toc=%2Fazure%2Fcosmos-db%2Ftoc.json&bc=%2Fazure%2Fcosmos-db%2Fbreadcrumb%2Ftoc.json

In Synapse the OLTP connector (the one in this repo which can be used in any Spark environment with one of the supported (currently 3.1/3.2/3.3) Spark versions) is referred to as "format("cosmos.oltp")" - the OLAP connector as "format("cosmos.olap")"

[praiaJaeyoungChung]

Thanks for confirming this @FabianMeiswinkel !
I was exploring the stack solution if I should introduce Synapse for this reason or if OLAP Spark was also available for Databricks stack as well. Is there a plan for releasing OLAP Spark connector for general consumption or at least to Azure Databricks Fabian?