-
Notifications
You must be signed in to change notification settings - Fork 186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sqlalchemy destination #1734
Sqlalchemy destination #1734
Conversation
✅ Deploy Preview for dlt-hub-docs canceled.
|
aee8ada
to
1e7fa6a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
I was expecting that we need some kind of settings per dialect (ie DialectCapabilities
) so we have optimized inserts. but it seems it is not necessary.
There are a lot of standard tests where this thing should work:
- restore state tests
- drop command / refresh mode tests
- all pipeline tests that do not requires merge write disposition
if we enable them we are good.
MERGE write disposition looks complicated, right? We surely can generate all required statements (we have only SELECT / INSERT / DELETE) but creating temporary tables looks like a stretch. Still we can do the Athena trick and create real tables for temporary data
A REPLACE mode that uses staging dataset would be also cool.
maybe those two could go to a separate ticket
def _create_date_time_type(self, sc_t: str, precision: Optional[int]) -> sa.types.TypeEngine: | ||
"""Use the dialect specific datetime/time type if possible since the generic type doesn't accept precision argument""" | ||
precision = precision if precision is not None else self.capabilities.timestamp_precision | ||
if sc_t == "timestamp": |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we support datetimes without timezones now see #1492
792f7a1
to
b9b317a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please merge current devel. all big changes for 1.0 are in. this looks really good!
what about merges? IMO our standard set of sql jobs will work just fine with mysql and sqllite. maybe you could try that out?
from typing import List | ||
|
||
import sqlalchemy as sa | ||
from alembic.runtime.migration import MigrationContext |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hope this is not an overkill... alembic deps look quite minimal so probably OK
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah would have liked to avoid this, but at least it was less complicated than I expected since alembic is not really used like this normally.
Begin implementing sqlalchemy loader SQLA load job, factory, schema storage, POC sqlalchemy tests attempt Implement SqlJobClient interface Parquet load, some tests running on mysql update lockfile Limit bulk insert chunk size, sqlite create/drop schema, fixes Generate schema update Get more tests running with mysql More tests passing Fix state, schema restore
remove secrets toml remove secrets toml Revert "remove secrets toml" This reverts commit 7dd189c. Fix default pipeline name test
b9b317a
to
45e386b
Compare
45e386b
to
a69d749
Compare
ed60447
to
c25932b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pls see comments on engine.dispose
self.engine = credentials.engine | ||
self.external_engine = True | ||
else: | ||
self.engine = sa.create_engine( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we create engine when first connection is created? sometimes sql client is created to just use a few internal functions and never opens the connection
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@steinitzu all good
but we do not need
self.engine.dispose()
we have null pool. so it will dispose of external engines which should not happen. WDYT?
Hmm I have the the flag for external engine so it's not closed: But yeah not sure if there's any need for it since the connection is closed anyway. Maybe in the special case where you override poolclass? |
heh you are right. I overlooked the external flag. then all good |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Draft Sqlalchemy loader. Basic things work so far.
Description
main
.SqlJobClient
interface so most/all sql loader tests should run (many already run against mysql)What's left:
Related Issues
Additional Context