You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
From Ian:
We want the loads from the S3 bucket to Snowflake to be idempotent when backfilling dates. Right now we use Snowpipe to automatically load any file that is uploaded to the S3 stage. In the above example, every time we run the backfill process, those duplicated files will also be loaded to Snowflake.
In order to make this idempotent, it should be enough to also clear the target table of data from the relevant date before executing the COPY INTO statements (either manually or using Snowpipe).
The text was updated successfully, but these errors were encountered:
From Ian:
We want the loads from the S3 bucket to Snowflake to be idempotent when backfilling dates. Right now we use Snowpipe to automatically load any file that is uploaded to the S3 stage. In the above example, every time we run the backfill process, those duplicated files will also be loaded to Snowflake.
In order to make this idempotent, it should be enough to also clear the target table of data from the relevant date before executing the COPY INTO statements (either manually or using Snowpipe).
The text was updated successfully, but these errors were encountered: