Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make COPY INTO statements idempotent when backfilling dates #373

Open
jkarpen opened this issue Aug 28, 2024 · 0 comments
Open

Make COPY INTO statements idempotent when backfilling dates #373

jkarpen opened this issue Aug 28, 2024 · 0 comments

Comments

@jkarpen
Copy link

jkarpen commented Aug 28, 2024

From Ian:
We want the loads from the S3 bucket to Snowflake to be idempotent when backfilling dates. Right now we use Snowpipe to automatically load any file that is uploaded to the S3 stage. In the above example, every time we run the backfill process, those duplicated files will also be loaded to Snowflake.

In order to make this idempotent, it should be enough to also clear the target table of data from the relevant date before executing the COPY INTO statements (either manually or using Snowpipe).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants