Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Adding databricks compatibility types #10787

Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
167fbac
adding databricks types
Shinnnyshinshin Dec 17, 2024
c18c9e5
Update databricks.py
Shinnnyshinshin Dec 17, 2024
51df6f7
Update databricks.py
Shinnnyshinshin Dec 17, 2024
b05b84c
type ignore
Shinnnyshinshin Dec 17, 2024
af769b1
a little more general
Shinnnyshinshin Dec 17, 2024
ce007c5
Update databricks.py
Shinnnyshinshin Dec 17, 2024
03e39c2
Update databricks.py
Shinnnyshinshin Dec 17, 2024
f5ccb4c
Update databricks.py
Shinnnyshinshin Dec 18, 2024
090554c
Merge branch 'm/zelda-1183/zelda-1188/adding-databricks-types' into m…
Shinnnyshinshin Dec 18, 2024
2467925
Update test_expect_column_values_to_be_in_type_list.py
Shinnnyshinshin Dec 18, 2024
806648f
Update databricks.py
Shinnnyshinshin Dec 18, 2024
cfec5e2
Merge branch 'm/zelda-1183/zelda-1188/adding-databricks-types' into m…
Shinnnyshinshin Dec 18, 2024
9d6f510
the fix
Shinnnyshinshin Dec 18, 2024
2066819
Merge branch 'develop' into m/zelda-1183/zelda-1188/adding-databricks…
Shinnnyshinshin Dec 18, 2024
0d68ca2
Update databricks.py
Shinnnyshinshin Dec 18, 2024
7efe84a
Merge branch 'm/zelda-1183/zelda-1188/adding-databricks-types' of htt…
Shinnnyshinshin Dec 18, 2024
f559b46
Merge branch 'm/zelda-1183/zelda-1188/adding-databricks-types' into m…
Shinnnyshinshin Dec 18, 2024
d6cf5dd
adding test
Shinnnyshinshin Dec 18, 2024
c0f90ea
a better import
Shinnnyshinshin Dec 18, 2024
e6f540a
Merge branch 'm/zelda-1183/zelda-1188/adding-databricks-types' into m…
Shinnnyshinshin Dec 18, 2024
9b15bb5
tests and new types
Shinnnyshinshin Dec 18, 2024
a87db03
Update databricks.py
Shinnnyshinshin Dec 18, 2024
7dba978
a bit of clean up before review.
Shinnnyshinshin Dec 18, 2024
bdfa30e
Update test_expect_column_values_to_be_in_type_list.py
Shinnnyshinshin Dec 18, 2024
c06a069
Merge branch 'develop' into m/zelda-1183/zelda-1188/adding-databricks…
Shinnnyshinshin Dec 18, 2024
4525003
only the needed changes.
Shinnnyshinshin Dec 18, 2024
58ff217
Merge branch 'm/zelda-1183/zelda-1188/adding-databricks-types' of htt…
Shinnnyshinshin Dec 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 35 additions & 3 deletions great_expectations/compatibility/databricks.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,42 @@
from __future__ import annotations

Check warning on line 1 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L1

Added line #L1 was not covered by tests

from great_expectations.compatibility.not_imported import NotImported

DATABRICKS_CONNECT_NOT_IMPORTED = NotImported(
"databricks-connect is not installed, please 'pip install databricks-connect'"
)

# The following types are modeled after the following documentation that is part
# of the databricks package.
# tldr: SQLAlchemy application should (mostly) "just work" with Databricks,
# other than the exceptions below
# https://github.com/databricks/databricks-sql-python/blob/main/src/databricks/sqlalchemy/README.sqlalchemy.md

try:
from databricks.sqlalchemy._types import TIMESTAMP_NTZ as TIMESTAMP_NTZ # noqa: PLC0414, RUF100
except (ImportError, AttributeError):
TIMESTAMP_NTZ = DATABRICKS_CONNECT_NOT_IMPORTED # type: ignore[misc, assignment]

Check warning on line 18 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L15-L18

Added lines #L15 - L18 were not covered by tests

try:
from databricks.sqlalchemy._types import DatabricksStringType as STRING # noqa: PLC0414, RUF100
except (ImportError, AttributeError):
STRING = DATABRICKS_CONNECT_NOT_IMPORTED # type: ignore[misc, assignment]

Check warning on line 23 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L20-L23

Added lines #L20 - L23 were not covered by tests

try:
from databricks import connect
except ImportError:
connect = DATABRICKS_CONNECT_NOT_IMPORTED
from databricks.sqlalchemy._types import TIMESTAMP as TIMESTAMP # noqa: PLC0414, RUF100
except (ImportError, AttributeError):
TIMESTAMP = DATABRICKS_CONNECT_NOT_IMPORTED # type: ignore[misc, assignment]

Check warning on line 28 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L26-L28

Added lines #L26 - L28 were not covered by tests

try:
from databricks.sqlalchemy._types import TINYINT as TINYINT # noqa: PLC0414, RUF100
except (ImportError, AttributeError):
TINYINT = DATABRICKS_CONNECT_NOT_IMPORTED # type: ignore[misc, assignment]

Check warning on line 33 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L30-L33

Added lines #L30 - L33 were not covered by tests


class DATABRICKS_TYPES:

Check warning on line 36 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L36

Added line #L36 was not covered by tests
"""Namespace for Databricks dialect types"""

TIMESTAMP_NTZ = TIMESTAMP_NTZ
STRING = STRING
TINYINT = TINYINT
TIMESTAMP = TIMESTAMP

Check warning on line 42 in great_expectations/compatibility/databricks.py

View check run for this annotation

Codecov / codecov/patch

great_expectations/compatibility/databricks.py#L39-L42

Added lines #L39 - L42 were not covered by tests
Loading