AzCopy V10.5.1 Release

New features

• Allow more accurate values for job status in jobs commands, e.g. completed with failed or skipped transfers.

Bug fixes

• Fixed issue with removing blobs with hdi_isfolder=true metadata when the list-of-files flag is used.
• Manually unfurl symbolic links to fix long file path issue on UNC locations.

AzCopy V10.5.0 Release

New features

1. Improved scanning performance for most cases by adding support for parallel local and Blob enumeration.
2. Added download support for the benchmark command.
3. A new way to quickly copy only files changed after a certain date/time. The copy command now accepts
   the parameter --include-after. It takes an ISO 8601-formatted date, and will copy only those files that were
   changed on or after the given date/time. When processing large numbers of files, this is faster than sync or
   --overwrite=IfSourceNewer. But it does require the user to specify the date to be used. E.g. 2020-08-19T15:04:00Z
   for a UTC time, 2020-08-19T15:04 for a time in the local timezone of the machine running Azcopy,
   or 2020-08-19 for midnight (00:00), also in the local timezone.
4. When detecting content type for common static website files, use the commonly correct values instead of looking them up in the registry.
5. Allow the remove command to delete blob directory stubs which have metadata hdi_isfolder=true.
6. The S3 to Blob feature now has GA support.
7. Added support for load command on Linux based on Microsoft Avere's CLFSLoad extension.
8. Each job now logs its start time precisely in the log file, using ISO 8601 format. This is useful if you want to
   use that start date as the --include-after parameter to a later job on the same directory. Look for "ISO 8601 START TIME"
   in the log.
9. Stop treating zero-item job as failure, to improve the user experience.
10. Improved the naming of files being generated in benchmark command, by reversing the digits.
    Doing so allows the names to not be an alphabetic series, which used to negatively impact the performance on the service side.
11. Azcopy can now detect when setting a blob tier would be impossible. If azcopy cannot check the destination account type, a new transfer failure status will be set: TierAvailabilityCheckFailure

Bug fixes

1. Fixed the persistence of last-write-time (as part of SMB info when uploading) for Azure Files. It was using the creation time erroneously.
2. Fixed the SAS timestamp parsing issue.
3. Transfers to the File Service with a read-only SAS were failing because we try listing properties for the parent directories.
   The user experience is improved by ignoring this benign error and try creating parent directories directly.
4. Fixed issue with mixed SAS and AD authentication in the sync command.
5. Fixed file creation error on Linux when decompression is turned on.
6. Fixed issue on Windows for files with extended charset such as [%00 - %19, %0A-%0F, %1A-%1F].
7. Enabled recovering from unexpectedEOF error.
8. Fixed issue in which attribute filters does not work if source path contains an asterisk in it.
9. Fixed issue of unexpected upload destination when uploading a whole drive in Windows (e.g. "D:").

AzCopy V10.4.3 Release

Bug fixes

1. Fixed bug where AzCopy errored if a filename ended with slash character. (E.g. backslash at end of a Linux filename.)

AzCopy V10.3.4 Release

Version 10.3.4

New features

1. Fixed feature parity issue by adding support for "ifSourceNewer" option on the overwrite flag. It serves as a replacement of the '\XO' flag in V8.

Bug fixes

1. Fixed jobs clean command on Windows which was previously crashing when the with-status flag was used.

AzCopy V10.4.2 Release

Bug fixes

1. Fixed bug in overwrite prompt for folders.

AzCopy V10.4.1 Release

New features

1. Added overwrite prompt support for folder property transfers.
2. Perform proxy lookup when the source is S3.

Bug fixes

1. When downloading from Azure Files to Windows with the --preserve-smb-permissions flag, sometimes
   the resulting permissions were not correct. This was fixed by limiting the concurrent SetNamedSecurityInfo operations.
2. Added check to avoid overwriting the file itself when performing copy operations.

AzCopy V10.4.0 Release

Version 10.4

New features

1. azcopy copy now supports the persistence of ACLs between supported resources (Windows and Azure Files) using the --persist-smb-permissions flag.
2. azcopy copy now supports the persistence of SMB property info between supported resources (Windows and Azure Files) using the --persist-smb-info flag. The information that can be preserved is Created Time, Last Write Time and Attributes (e.g. Read Only).
3. AzCopy can now transfer empty folders, and also transfer the properties of folders. This applies when both the source and destination support real folders (Blob Storage does not, because it only supports virtual folders).
4. On Windows, AzCopy can now activate the special privileges SeBackupPrivilege and SeRestorePrivilege. Most admin-level accounts have these privileges in a deactivated state, as do all members of the "Backup Operators" security group. If you run AzCopy as one of those users and supply the new flag --backup, AzCopy will activate the privileges. (Use an elevated command prompt, if running as Admin). At upload time, this allows AzCopy to read files which you wouldn't otherwise have permission to see. At download time, it works with the --preserve-smb-permissions flag to allow preservation of permissions where the Owner is not the user running AzCopy. The --backup flag will report a failure if the privileges cannot be activated.
5. Status output from AzCopy copy, sync, jobs list, and jobs status now contains information about folders.
   This includes new properties in the JSON output of copy, sync, list and jobs status commands, when --output-type json is used.
6. Empty folders are deleted when using azcopy rm on Azure Files.
7. Snapshots of Azure File Shares are supported, for read-only access, in copy,sync and list. To use, add a sharesnapshot parameter at end of URL for your Azure Files source. Remember to separate it from the existing query string parameters (i.e. the SAS token) with a &. E.g.
   https://<youraccount>.file.core.windows.net/sharename?st=2020-03-03T20%3A53%3A48Z&se=2020-03-04T20%3A53%3A48Z&sp=rl&sv=2018-03-28&sr=s&sig=REDACTED&sharesnapshot=2020-03-03T20%3A24%3A13.0000000Z
8. Benchmark mode is now supported for Azure Files and ADLS Gen 2 (in addition to the existing benchmark support for Blob Storage).
9. A special performance optimization is introduced, but only for NON-recursive cases in this release. An --include-pattern that contains only * wildcards will be performance optimized when querying blob storage without the recursive flag. The section before the first * will be used as a server-side prefix, to filter the search results more efficiently. E.g. --include-pattern abc* will be implemented as a prefix search for "abc". In a more complex example, --include-pattern abc*123, will be implemented as a prefix search for abc, followed by normal filtering for all matches of abc*123. To non-recursively process blobs contained directly in a container or virtual directory include /* at the end of the URL (before the query string). E.g. http://account.blob.core.windows.net/container/*?<SAS>.
10. The --cap-mbps parameter now parses floating-point numbers. This will allow you to limit your maximum throughput to a fraction of a megabit per second.

Special notes

1. A more user-friendly error message is returned when an unknown source/destination combination is supplied
2. AzCopy has upgraded to service revision 2019-02-02. Users targeting local emulators, Azure Stack, or other private/special instances of Azure Storage may need to intentionally downgrade their service revision using the environment variable AZCOPY_DEFAULT_SERVICE_API_VERSION. Prior to this release, the default service revision was 2018-03-28.
3. For Azure Files to Azure Files transfers, --persist-smb-permissions and --persist-smb-info are available on all OS's. (But for for uploads and downloads, those flags are only available on Windows.)
4. AzCopy now includes a list of trusted domain suffixes for Azure Active Directory (AAD) authentication.
   After azcopy login, the resulting token will only be sent to locations that appear in the list. The list is:
   *.core.windows.net;*.core.chinacloudapi.cn;*.core.cloudapi.de;*.core.usgovcloudapi.net.
   If necessary, you can add to the the list with the command-line flag: --trusted-microsoft-suffixes. For security, you should only add Microsoft Azure domains.
5. When transferring over a million files, AzCopy will reduces its progress reporting frequency from every 2 seconds to every 2 minutes.

Breaking changes

1. To accommodate interfacing with JavaScript programs (and other languages that have similar issue with number precision), all the numbers in the JSON output have been converted to strings (i.e. with quotes around them).
2. The TransferStatus value SkippedFileAlreadyExists has been renamed SkippedEntityExists and may now be used both for when files are skipped and for when the setting of folder properties is skipped. This affects the input and output of azcopy jobs show and the status values shown in the JSON output format from copy and sync.
3. The format and content of authentication information messages, in the JSON output format, e.g.
   "Using OAuth token for authentication" has been changed.

Bug fixes

1. AzCopy can now overwrite even Read-Only and Hidden files when downloading to Windows. (The read-only case requires the use of the new --force-if-read-only flag.)
2. Fixed a nil dereference when a prefetching error occurs in a upload
3. Fixed a nil dereference when attempting to close a log file while log-level is none
4. AzCopy's scanning of Azure Files sources, for download or Service to Service transfers, is now much faster.
5. Sources and destinations that are identified by their IPv4 address can now be used. This enables usage with storage emulators. Note that the from-to flag is typically needed when using such sources or destinations. E.g. --from-to BlobLocal if downloading from a blob storage emulator to local disk.
6. Filenames containing the character : can now safely be downloaded on Windows and uploaded to Azure Files
7. Objects with names containing + can now safely be used in imported S3 object names
8. The check-length flag is now exposed in benchmark mode, so that length checking can be turned off for more speed, when benchmarking with small file sizes. (When using large file sizes, the overhead of the length check is
   insignificant.)
9. The in-app documentation for Service Principal Authentication has been corrected, to now include the application-id parameter.
10. ALL filter types are now disallowed when running azcopy rm against ADLS Gen2 endpoints. Previously include/exclude patterns were disallowed, but exclude-path was not. That was incorrect. All should have been disallowed because none (other than include-path) are respected.
11. Fixed empty page range optimization when uploading Managed Disks. In an edge case, there was previously a risk of data corruption if the user uploaded two different images into the same Managed Disk resource one after the other.

AzCopy V10.3.3 Release

Version 10.3.3

New features

1. azcopy list is now supported on Azure Files and ADLS Gen 2, in addition to Blob Storage.
2. The --exclude-path flag is now supported in the sync command.
3. Added new environment variable AZCOPY_USER_AGENT_PREFIX to allow a prefix to be appended to the user agent strings.

Bug fixes

1. Content properties (such as Content-Encoding and Cache-Control) are now included when syncing Blob -> Blob and Azure
   Files -> Azure Files
2. Custom metadata is now included when syncing Blob -> Blob and Azure Files -> Azure Files
3. The azcopy list command no longer repeats parts of its output. (Previously it would sometimes repeat itself and show the same blob multiple times in the output.)
4. The --aad-endpoint parameter is now visible, instead of hidden. It allows use of Azure Active Directory
   authentication in national clouds (e.g. Azure China).
5. On Windows, AzCopy now caches information about which proxy server should be used, instead of looking it up every
   time. This significantly reduces CPU
   usage when transferring many small files. It also solves a rare bug when transfers got permanently "stuck" with
   one uncompleted file.
6. When uploading to a write-only destination, there is now a clearer error message when the built-in file length check
   fails. The message says how to fix the problem using --check-length=false.
7. Size checks on managed disk imports are now clearer, and all run at the start of the import process instead of the end.

AzCopy V10.3.2 Release

Version 10.3.2

Bug fixes

1. Jobs could not be cancelled while scanning was still in progress.
2. Downloading large managed disks (8 TB and above) failed with errors.
3. Downloading large page blobs might make no progress for the first 15 or 20 minutes.
4. There was a rare error where the final output could under-report the total number of files in the job. That error has been fixed.
5. When using JSON output mode, the output from the rm command on ADLS Gen2 was inconsistent with the output from other commands
6. After authentication errors, files in progress were not cleaned up (deleted) at the destination. If there was an
   authentication failure during a job (e.g. a SAS token expired while in use) this could result in files being left
   behind that had incomplete contents (even though their size looked correct).
7. The AUTO concurrency option, for automatically tuning concurrency as AzCopy runs, started working too late if scanning (aka enumeration) took a long time. This resulted in reduced throughput when using this setting.
8. It was not possible to access the root of Windows drives with lowercase drive letters. E.g. d:\
9. Service to Service transfers would fail when using environment variable to specify OAuth authentication.
10. Certain errors parsing URLs were not reported clearly.
11. When downloading to NUL (/dev/null on Linux), files of zero length no longer trigger errors. (Downloads to NUL can be used in performance testing and bulk MD5 checking.

AzCopy V10.3.1 Release

Version 10.3.1

New features

1. Added helpful deprecation notice for legacy include/exclude flags.
2. Added back request ID at log level INFO.
3. Added back cancel-from-stdin option for partner integration.
4. Added flag to define delete snapshot options for the remove command.

Bug fix

1. Fixed race condition in shutdown of decompressingWriter.
2. Made progress reporting more accurate.