Data Transfer between Clouds using rclone

[beepdot]

Hello community, in this post I put some some information on rclone and data transfer strategy between clouds

Data Copy

We are moving our data store from Azure blob storage to AWS S3 buckets. We have lot of data and files to be moved over (around 2000TB of data with millions of files). I have tried rlcone to copy a sample set of data from Azure storage to AWS S3 and it worked well. Here is some results -

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 64 --multi-thread-cutoff 128Mi --transfers 64 --buffer-size 64Mi --azureblob-chunk-size 64Mi --s3-list-chunk 20000 --s3-chunk-size 64Mi sync azure:content-store aws:content-store

1.258 TiB / 1.258 TiB, 100%, 10.898 MiB/s, ETA 0s

real    297m56.035s

It took around 5 hours to copy approximately 1.26 TB of data from Azure storage to S3. Also we had 55,26,697 objects that required to be copied over.

I did another run with some updated numbers which resulted in slightly better copy duration

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 128 --multi-thread-cutoff 64Mi --transfers 128 --buffer-size 128Mi --azureblob-chunk-size 32Mi --s3-list-chunk 40000 --s3-chunk-size 32Mi sync azure:content-store aws:content-store

1.264 TiB / 1.264 TiB, 100%, 40.424 MiB/s, ETA 0s

real    200m42.776s

Around same 1.26 TB of data with a little more than 55,26,697 objects took 3 hours 20 minutes with the updated parameters.

I ran another run with some more updated numbers

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 256 --multi-thread-cutoff 32Mi --transfers 256 --buffer-size 256Mi --azureblob-chunk-size 16Mi --s3-list-chunk 80000 --s3-chunk-size 16Mi copy azure:content-store aws:content-store
1.264 TiB / 1.264 TiB, 100%, 40.424 MiB/s, ETA 0s

real    178m10.623s

Time to copy down to 2 hours 58m

I did one more run with some more updated numbers on a 8 core and 32GB machine (these numbers take more memory, around 26GB, so we need to increase the VM size)

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 512 --multi-thread-cutoff 16Mi --transfers 512 --buffer-size 512Mi copy azure:content-store aws:content-store

real    100m50.592s

The time to transfer reduced to just 1 hour and 40m

I did one last run with some more updated numbers on is the 8 core and 32GB machine

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 1024 --multi-thread-cutoff 8Mi --transfers 1024 --buffer-size 1024Mi copy azure:content-store aws:content-store

real    104m52.112s

In above run load avg crossed 8 and reached around 12 (on a 8 core VM), due to which the time might have increased.

Rerun of above with an upgraded VM (16 Core and 64GB Ram)

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 1024 --multi-thread-cutoff 4Mi --transfers 1024 --buffer-size 1024Mi copy azure:content-store aws:content-store

From this point onwards, I started to see S3 errors while copying with response code as 503 and error message as Failed to copy: SlowDown: Please reduce your request rate It looks like with the upgraded VM and 1024 as threads, we are hitting the max copy rate of S3 which is 3,500 PUT/COPY/POST/DELETE requests/second

So we end the test on S3 here for now. In future, we might pick it up again from this point to see if we can get the copy operation completed in under 60m.

Some more details about the VM if it's useful for someone:

Ulimit values set to 65K on the VM so that we don't have issues in max open files.

cat /etc/security/limits.conf  | tail -n 10

* soft     nproc          65000   
* hard     nproc          65000   
* soft     nofile         65000   
* hard     nofile         65000
root soft     nproc          65000    
root hard     nproc          65000   
root soft     nofile         65000   
root hard     nofile         65000
# End of file

Data Sync

In the first section, I was able to copy the data in 1 hour 40m. Now the use case is, I need to keep both the cloud stores (Azure blob and AWS S3) in sync. I ran the same rclone sync command which I used in the first section, and it took more than 10 hours to complete. I was hoping the sync operation would be much faster than the initial copy (Even if we use 1024 as the number of threads for sync operation, it would still take almost same time as the copy operation, I haven't tried this bigger number though)

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 64 --multi-thread-cutoff 128Mi --transfers 64 --buffer-size 64Mi --azureblob-chunk-size 64Mi --s3-list-chunk 20000 --s3-chunk-size 64Mi  sync azure:content-store aws:content-store

real    631m53.338s

I tried using --fast-list and --max-age options independently as well as combined and could not see any improvements in sync. I am suspecting the sync is taking time as it needs to scan and compare every object in source and destination and then decide if it needs to copy or not.

So to achieve faster sync, we have come up with a new strategy

1. In Azure storage, setup a object replication policy to copy only new objects to a new storage account
2. Initiate copy using rclone from Azure storage to AWS S3
3. Once the entire data is copied, use the rclone move command to move the newly created objects from the new storage account to AWS S3
4. The rclone move is fast since the number of objects are low (replication happens only on the newly created objects)
5. The objects in the new storage account will get deleted on using rclone move, so on a daily basis, we will have only limited number of objects to be moved over
6. This strategy will not be able to keep the source and destination consistent if we have deletion scenarios in the source storage account. If you don't care about object deletion, then this works well.

If anyone has any other solutions or recommendations for syncing the objects quickly with or without any additional steps (object replication etc), do let me know. If we can keep the source and destination consistent (with object deletion), that would be a cherry on the pie.

[beepdot]

Data copy runs on OCI

• On a vm with size - Standard D4s v3 (4 vcpus, 16 GiB memory)

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 256 --multi-thread-cutoff 32Mi --transfers 256 --buffer-size 256
Mi --azureblob-chunk-size 16Mi --s3-list-chunk 80000 --s3-chunk-size 16Mi copy azure:content-store oci:content-store

1.258 TiB / 1.258 TiB, 100%, 4.943 MiB/s, ETA 0s

real    163m37.774s

• On a upgraded vm - Standard D16s v3 (16 vcpus, 64 GiB memory)

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 1024 --multi-thread-cutoff 4Mi --transfers 1024 --buffer-size 1024Mi copy azure:content-store oci:content-store

1.276 TiB / 1.276 TiB, 100%, 115.668 MiB/s, ETA 0s

real    68m25.124s

• Reran with some higher numbers, but it did not increase the performance

time rclone --progress --human-readable --stats-one-line --multi-thread-streams 1536 --multi-thread-cutoff 2Mi --transfers 1536 --buffer-size 15
36Mi copy azure:content-store oci:content-store

1.276 TiB / 1.276 TiB, 100%, 172.824 MiB/s, ETA 0s

real    71m10.541s

[beepdot]

Rclone Configurations

• Config file for rclone is usually located at ~/.config/rclone/rclone.conf
• The below config file sample includes configurations for Azure, AWS and OCI cloud.

[azure]
type = azureblob
sas_url = SAS_URL

[aws]
type = s3
provider = AWS
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
region = REGION
acl = public-read
storage_class = STANDARD

[oci]
type = s3
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
region = REGION
endpoint = https://NAMESPACE.compat.objectstorage.REGION.oraclecloud.com
provider = Other

[damithkothalawala]

You should use native OCI rclone plugin for higher throughput. Which is being maintained by Oracle.

[damithkothalawala]

AZ to OCI Migration documented here -> https://github.com/damithkothalawala/rclone-scripts-azue-oci/blob/main/README.md