Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for Redshift connections #2215

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

owenjonesuob
Copy link
Contributor

@owenjonesuob owenjonesuob commented May 3, 2024

An attempt to add support for Amazon Redshift connections.

Redshift is based on Postgres - which is great, because {dm} already has excellent Postgres support 😁

So for the most part, we can just borrow existing functionality.


There are some differences between Redshift and Postgres - more detains in the AWS docs. One note in particular caught my eye:

Unique, primary key, and foreign key constraints are permitted, but they are informational only. They are not enforced by the system, but they are used by the query planner. - https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-features.html

I'm not sure if this has any impact on whether we can build a dm by learning the database keys.

I also tried, but failed, to write a Redshift-friendly version of postgres_column_constraints since Redshift doesn't support some of the syntax used there (e.g. array_position()). This affects dm_meta_raw() - for now I have added an implementation based on the final else clause.


Testing any Redshift functionality directly might be tricky, since it would require a Redshift warehouse in an AWS organisation - i.e. a lot of additional infrastructure (and cost?!). Would we be happy to rely on the fact that it's "nearly Postgres", and therefore mostly covered by Postgres testing? Or can we think of some way to test directly?

Edit: that is, testing via CI/CD - otherwise we are perhaps relying on ad hoc testing by users who happen to have access to a Redshift warehouse...


Requested indirectly in #1678.

Copy link
Contributor

aviator-app bot commented May 3, 2024

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This pull request is currently open (not queued).

How to merge

To merge this PR, comment /aviator merge or add the mergequeue label.


See the real-time status of this PR on the Aviator webapp.
Use the Aviator Chrome Extension to see the status of your PR within GitHub.

@krlmlr
Copy link
Collaborator

krlmlr commented Nov 21, 2024

Thanks! I have no easy way to test this, I trust this works for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants