Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: no need to create directory before creating a blob #113

Merged
merged 1 commit into from
Mar 19, 2024

Conversation

loicmathieu
Copy link
Member

@loicmathieu loicmathieu commented Mar 18, 2024

Directory creation is very slow in GCS and no necessary before creating a blob in a directory as directories are part of the name of a blob not a real filesystem structure.

Fixes #112

For the linked issue, times to split a file in 10 goes from 22s to less than 3s

@loicmathieu loicmathieu force-pushed the fix/directory-creation-time branch 2 times, most recently from 6347813 to 7f671a7 Compare March 18, 2024 16:39
@loicmathieu loicmathieu changed the title fix: check if a directory exist before creating it fix: no need to create directory before creating a blog Mar 18, 2024
@loicmathieu loicmathieu changed the title fix: no need to create directory before creating a blog fix: no need to create directory before creating a blob Mar 18, 2024
@loicmathieu loicmathieu force-pushed the fix/directory-creation-time branch 2 times, most recently from 79ef99b to 386fe21 Compare March 19, 2024 09:15
@loicmathieu loicmathieu merged commit c885936 into master Mar 19, 2024
1 check passed
@loicmathieu loicmathieu deleted the fix/directory-creation-time branch March 19, 2024 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GCS Storage is very slow to split files in the context of a ForEachItem
1 participant