Add Dremio as a destination

[maxfirman]

Description

This PR adds Dremio as a destination to DLT

Additional Context

• Tested locally against OSS Dremio running in docker and against Dremio Enterprise cluster.
• dockercompose.yaml provided for creating local Dremio environment with minio as an object store for staging.

Some details for future PRs:

• implement merge write disposition using MERGE INTO
• implement table partition evolution

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	db91346
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/66142e1f18d3f100080bfe35

[rudolfix]

@maxfirman this is amazing! we'll assign a reviewer tomorrow and add tests to our CI jobs after merging

[sh-rp]

note to self: remove!

[rudolfix]

@maxfirman @sh-rp this is so good! this destination was quite different from the other but you were able to fit it perfectly into existing model and test structure (thx for making tests working)

I opened a PR that will be used that the newest code works #1194

We upgraded how we deal with configurations, parquet files and redid our CI structure so when all tests in the branch above are passing I'll push the newest version here and then we merge it.

IMO this destination is really good even now (maybe I'm missing something that dremio does and we do not support :))

• we already have ticket to start using MERGE SQL statements
• adding gcp and azure blob storages are mostly matter of config and we can do that later
• using a sentinel table for destinations that have no schemas/datasets actually makes sense. we do that for vector databases. we also support empty dataset names so tables are not prefixed (not sure it applies here after all)
• we can implement dremio adapter to expose all the partition/clustering/retention features (some are implemented but adapters let users to add the hints conveniently to the resources)

I also found following comment in tests

    # schemaless destinations allow adding of root key without the pipeline failing
    # for now this is only the case for dremio
    # doing this will result in somewhat useless behavior
    destination_allows_adding_root_key = destination_config.destination == "dremio"

tbh. to me it looks like dremio does not enforce nulls and allows to add root_id column even if it is NOT NULL to an existing table with the data. @maxfirman could that be the case?

anyway thanks again for your amazing work!

[maxfirman]

@rudolfix thanks for time and effort reviewing this PR!

Indeed Dremio does not support NOT NULL constraints on columns. I think you are right that is the cause of the test failure.

I agree with all of your points. I have a few other thoughts as well for follow up PRs:

• Dremio recently fixed a bug that enables the use of the ADBC client [Python] Querying Dremio with the ADBC Flight SQL client apache/arrow-adbc#1559. We could therefore remove the pydremio.py file and cut over to ADBC. I don't think there is any rush on this.
• I'd like to test a wider selection of destination data source types in Dremio. We are somewhat hamstrung by the lack of CREATE SCHEMA support, which is why the CI tests run against a "NAS" (really local file system). I know that Dremio does support CREATE FOLDER sql statements for Nessie catalogs, so Nessie might be next logical thing to add to the testing pipelines as it should be possible to get all the tests passing against it. It might also be worth running a subset of the tests that don't require CREATE SCHEMA against a Hive Metastore data source.
• I can foresee that we may want to expose a "path_prefix" config option to build tables at arbitrarily nested levels in a folder structure.
• MERGE INTO sql support is vital given that Dremio doesn't support transactions, so I'm really glad to hear that is on your backlog!

[rudolfix]

@maxfirman one question to:

I'd like to test a wider selection of destination data source types in Dremio. We are somewhat hamstrung by the lack of CREATE SCHEMA support, which is why the CI tests run against a "NAS" (really local file system). I know that Dremio does support CREATE FOLDER sql statements for Nessie catalogs, so Nessie might be next logical thing to add to the testing pipelines as it should be possible to get all the tests passing against it. It might also be worth running a subset of the tests that don't require CREATE SCHEMA against a Hive Metastore data source.

we will be able to test against dremio cloud right? this "NAS" is also available there - the "local" filesystem is in the cloud then, right?

also: #1199

[maxfirman]

this "NAS" is also available there - the "local" filesystem is in the cloud then, right?

@rudolfix I'm not 100% sure as I'm only familiar with their on-prem offering.

If we are running tests against Dremio Cloud it may make sense to use the built in "Arctic" catalog (which is basically managed Nessie) instead.

Nessie can be run in a docker container, so we could also extend the docker-compose.yaml to stand up a Nessie server if we wanted to test this out against a Dockerised deployment.