Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove files after partial unload #94

Open
tvogels01 opened this issue Feb 16, 2018 · 0 comments
Open

Remove files after partial unload #94

tvogels01 opened this issue Feb 16, 2018 · 0 comments

Comments

@tvogels01
Copy link
Contributor

Summary

Problem: Unload in Redshift happens distributed over the cluster. If unload fails on one node, the others will continue to write and leave files behind. These extraneous files will then block re-running the unload.
Solution: If there is an error during unload and the manifest file is missing, then remove all csv files.

Details

Maybe our --force option to unload is enough? This is likely to add around incident recovery.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant