You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm running azcopy sync command to backup data from azure data lake to azure storage account.
I'm using command like this: azcopy sync https://mydatalake.blob.core.windows.net/my-container/ https://mystorageaccount.blob.core.windows.net/my-container/ --recursive --log-level=NONE --delete-destination=true
My azure data lake contains tens of millions small and medium files. Sync operation and copying of newly created files takes ~10 minutes, while deleting extra objects on destination takes many hours.
Moreover, even I specified --log-level=NONE I see messages like follow for each removed file: 6142703 Files Scanned at Source, 6844507 Files Scanned at Destination, 2-sec Throughput (Mb/s): 0 INFO: Deleting extra object: DELTA/path/to/my/file.parquet
Questions:
Is it possible to delete files on destination by batches?
The text was updated successfully, but these errors were encountered:
ppolushkin
changed the title
azcopy sync with --delete-destination=true option takes too much time on INFO: Deleting extra object
azcopy sync --delete-destination=true takes too much time on Deleting extra object
Oct 27, 2024
Dear all,
I'm running azcopy sync command to backup data from azure data lake to azure storage account.
I'm using command like this:
azcopy sync https://mydatalake.blob.core.windows.net/my-container/ https://mystorageaccount.blob.core.windows.net/my-container/ --recursive --log-level=NONE --delete-destination=true
My azure data lake contains tens of millions small and medium files. Sync operation and copying of newly created files takes ~10 minutes, while deleting extra objects on destination takes many hours.
Moreover, even I specified --log-level=NONE I see messages like follow for each removed file:
6142703 Files Scanned at Source, 6844507 Files Scanned at Destination, 2-sec Throughput (Mb/s): 0 INFO: Deleting extra object: DELTA/path/to/my/file.parquet
Questions:
Details:
ubuntu:22.04
azcopy version 10.26.0
Environment variables:
Kind regards,
The text was updated successfully, but these errors were encountered: