Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

writeSingleFile doesn't create folders in s3 #150

Open
jeremy-giaco opened this issue Feb 22, 2022 · 1 comment
Open

writeSingleFile doesn't create folders in s3 #150

jeremy-giaco opened this issue Feb 22, 2022 · 1 comment

Comments

@jeremy-giaco
Copy link

jeremy-giaco commented Feb 22, 2022

I had some code that was working correctly with DariaWriters, where i wrote data to a Staging directory under a bucket in s3. The data was effectively moved from there to the bucket (root) and merged into a single file with a given name. When i changed my configuration to write the file with a prefix (folder) to the object name {bucket}/project-name/yyyymmdd/filename.csv} it looks successful, but file is not actually ever written. After going into s3 console and creating the folder structure, the same code wrote the file successfully.

@MrPowers
Copy link
Collaborator

Can you please send the exact code you ran? The code has this comments:

  // tmpFolder should look like s3a://bucket/data/src
  // filename should look like s3a://bucket/data/dest/my_cool_file.csv

Did you add s3a to your S3 paths?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants