feat: add support for Redshift connections #2215
Open
+51
−27
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An attempt to add support for Amazon Redshift connections.
Redshift is based on Postgres - which is great, because {dm} already has excellent Postgres support 😁
So for the most part, we can just borrow existing functionality.
There are some differences between Redshift and Postgres - more detains in the AWS docs. One note in particular caught my eye:
I'm not sure if this has any impact on whether we can build a dm by learning the database keys.
I also tried, but failed, to write a Redshift-friendly version of
postgres_column_constraints
since Redshift doesn't support some of the syntax used there (e.g.array_position()
). This affectsdm_meta_raw()
- for now I have added an implementation based on the finalelse
clause.Testing any Redshift functionality directly might be tricky, since it would require a Redshift warehouse in an AWS organisation - i.e. a lot of additional infrastructure (and cost?!). Would we be happy to rely on the fact that it's "nearly Postgres", and therefore mostly covered by Postgres testing? Or can we think of some way to test directly?
Edit: that is, testing via CI/CD - otherwise we are perhaps relying on ad hoc testing by users who happen to have access to a Redshift warehouse...
Requested indirectly in #1678.