Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support csv file format in snowflake #1470

Closed
rudolfix opened this issue Jun 16, 2024 · 2 comments
Closed

support csv file format in snowflake #1470

rudolfix opened this issue Jun 16, 2024 · 2 comments
Labels
community This issue came from slack community workspace good first issue Good for newcomers

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Jun 16, 2024

Background
It looks like csv is still the fastest (like 3x faster: https://community.snowflake.com/s/article/How-to-Load-Terabytes-Into-Snowflake-Speeds-Feeds-and-Techniques) so it makes sense to enable it (since we have support to write csvs already)

Tasks

    • enable csv as supported file format
    • add csv option to COPY job of snowflake, make sure you configure it for how our writers work
    • find tests we use for postgres and run them on snowflake. make sure that arrow and python writers work

Previously, the biggest problem was to recognize quoted empty strings as NULL. fingers crossed Snowflake supports that (FORCE NULLs on postgres) - see Postgres copy command

@rudolfix rudolfix added good first issue Good for newcomers community This issue came from slack community workspace labels Jun 16, 2024
@rudolfix rudolfix moved this from Todo to Planned in dlt core library Jun 17, 2024
@rudolfix rudolfix moved this from Planned to In Progress in dlt core library Jun 17, 2024
@krishna06nani
Copy link

Hi @rudolfix, I see this issue is not assigned to any developer. I have 4 years of experience on ETL tools(Mulesoft and Azure synapse) + Snowflake. Can i take this up?

@sh-rp
Copy link
Collaborator

sh-rp commented Jun 25, 2024

@krishna06nani thanks for the offer, It already has been worked on in #998. If you'd like to help out, find us in our community and we can talk about what you might implement :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community This issue came from slack community workspace good first issue Good for newcomers
Projects
Status: Done
Development

No branches or pull requests

3 participants