-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(filesystem): implement Google Drive source #932
Conversation
✅ Deploy Preview for dlt-hub-docs canceled.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the biggest missing piece are tests:
the minimum option: we need to test glob functionality
the tests are in tests/load/filesystem
- add gdrive bucket to
test_filesystem_dict
- buckets are defined in tests/load/utils
- bucket url is defined in tests/.dlt/config.toml, you can use https://drive.google.com/drive/u/0/folders/15eC3e5MNew2XAIefWNlG8VlEa0ISnnaG as the bucket url (contains
standard_source
inside)
NOTE: if you modifyDEFAULT_BUCKETS
to include gdrive, all other tests (that test write files) will pick it up.
in theory they should all pass so you may try it out, to pass them may be our maximum plan (and maybe that's the easiest). otherwise you need to create another fixture...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, at the first gaze, there is one suspicious item about split()
.
Also, there is a bunch of changes to Athena, BigQuery and Snowflake.
Trying to run tests now to make sure it works.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Towards: dlt-hub/verified-sources#308
I took the original implementation, moved it into the
dlt
repo and fixed several small problems, which were preventing the driver from working with Google Drive. Most of the original driver code is left untouched - even though I think some parts of it are not needed (file uploading, deleting, etc.)