CI builds frequently fail with MSB3021: Access to the path xxx.dll is denied

[RobSwDev]

A couple of weeks ago, our continuous integration builds started failing frequently with the above error.
It always seems to be due to copying dlls from various nuget packages we reference: e.g. Azure.Core.dll, or Microsoft.Data.SqlClient.resources.dll, etc.

We assume it's some change in our build infrastructure (possibly more aggressive AntiVirus?), but we're not making much progress with corporate IT in working out what may have changed and what we can do about it.

Is there anything I can do to make the build more tolerant, and retry any failed copy?

I tried setting the following in Directory.Build.props, but as far as I can see, it had zero effect.

  <PropertyGroup>
    <!-- Attempt to make the build more reliable when copying dependencies conflict -->
    <CopyRetryCount>5</CopyRetryCount>
    <CopyRetryDelayMilliseconds>400</CopyRetryDelayMilliseconds>
    <SkipUnchangedFiles>true</SkipUnchangedFiles>
  </PropertyGroup>

Bizarrely, my last failure logged an error and 2 warnings. The warnings appear to when it is attempting be doing the retry. But it looks like Azure.Core.dll failed without any retry.

Microsoft.Common.CurrentVersion.targets(4879,5): error MSB3021: Unable to copy file "C:\Users\***\.nuget\packages\azure.core\1.25.0\lib\net461\Azure.Core.dll" to "F:\***\Debug\net472\Azure.Core.dll". Access to the path 'F:\***\Debug\net472\Azure.Core.dll' is denied.

Microsoft.Common.CurrentVersion.targets(4879,5): warning MSB3026: Could not copy "C:\Users\***\.nuget\packages\microsoft.identity.client\4.47.2\lib\net461\Microsoft.Identity.Client.dll" to "F:\net472\Microsoft.Identity.Client.dll". Beginning retry 1 in 400ms. The process cannot access the file 'F:\net472\Microsoft.Identity.Client.dll' because it is being used by another process.

Microsoft.Common.CurrentVersion.targets(4879,5): warning MSB3026: Could not copy "C:\Users\***\.nuget\packages\microsoft.data.sqlclient\5.1.0\lib\net462\Microsoft.Data.SqlClient.dll" to "F:\net472\Microsoft.Data.SqlClient.dll". Beginning retry 1 in 400ms. The process cannot access the file 'F:\net472\Microsoft.Data.SqlClient.dll' because it is being used by another process.

Notes:

• We are building to a common directory, and also building in parallel, which I understand is a bit hairy (and not strictly supported), but has worked well enough for us for years. We ensure all projects agree on 3rd party package versions, which makes the build deterministic.
• We're using VS2022 / equivalent build tools. Multi-targeting NetFramework & Net6, though I doubt that's relevant
• Never seem to see the conflicts when building locally in Visual Studio.

[KalleOlaviNiemitalo]

Setting the MSBUILDALWAYSRETRY environment variable (with any value) would make MSBuild retry the copy after the access-denied error, but I don't know whether those retries would succeed. See also #9169.

msbuild/src/Tasks/Copy.cs

Line 26 in 971bf70

internal const string AlwaysRetryEnvVar = "MSBUILDALWAYSRETRY";

msbuild/src/Tasks/Copy.cs

Lines 83 to 88 in 971bf70

	/// <summary>
	/// Force the copy to retry even when it hits ERROR_ACCESS_DENIED -- normally we wouldn't retry in this case since
	/// normally there's no point, but occasionally things get into a bad state temporarily, and retrying does actually
	/// succeed. So keeping around a secret environment variable to allow forcing that behavior if necessary.
	/// </summary>
	private static bool s_alwaysRetryCopy = Environment.GetEnvironmentVariable(AlwaysRetryEnvVar) != null;

[RobSwDev]

There were no retries for that dll.
But it's exactly the same Copy task:
Both attempts were at the same location in the targets file: Microsoft.Common.CurrentVersion.targets(4879,5)

I don't see any reason why 1 of the invocations would not receive the correct $(CopyRetryCount) value, though I can't tell for certain from the logs I have. I'll see if I can get further diagnostics next week.

[RobSwDev]

Can confirm that the failing Copy Task had $(CopyRetryCount) set to 5:
But the copy failed with no retries. Looks like the CopyTask code does not handle the STIERR_SHARING_VIOLATION error.

C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(4879,5): Got System.IO.IOException: The process cannot access the file 'F:*\net6.0-windows\ja\Microsoft.SqlServer.Types.resources.dll' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite)
at Microsoft.Build.Tasks.Copy.CopyFileWithLogging(FileState sourceFileState, FileState destinationFileState)
at Microsoft.Build.Tasks.Copy.DoCopyWithRetries(FileState sourceFileState, FileState destinationFileState, CopyFileWithState copyFile) copying d:\TeamCity\Nuget\packages\microsoft.sqlserver.types\160.1000.6\lib\netstandard2.1\ja\Microsoft.SqlServer.Types.resources.dll to F:*\net6.0-windows\ja\Microsoft.SqlServer.Types.resources.dll and HR is -2147024864 [F:\XXX.csproj]

Could not copy "d:\TeamCity\Nuget\packages\microsoft.sqlserver.types\160.1000.6\lib\netstandard2.1\ja\Microsoft.SqlServer.Types.resources.dll" to "F:*\net6.0-windows\ja\Microsoft.SqlServer.Types.resources.dll". Beginning retry 1 in 400ms. The process cannot access the file 'F:*\net6.0-windows\ja\Microsoft.SqlServer.Types.resources.dll' because it is being used by another process.

[RobSwDev]

Actually, I'm wrong: it does retry.
Unfortunately, it still seems to fail the build. I need to look into why.
Possibly because we have WarnAsError set (with various MSB exclusions for Copy failures). Perhaps I just need to widen the exclusions to include this error/exception.

[RobSwDev]

I'm working around this by disabling parallel building on our "WarnAsError" configuration.
None of the exclusions in /nowarn:"MSB3270;MSB3026;MSB3021" prevent the exception from being promoted to an error: there's no MSB code logged above to exclude.

Would still like to understand what changed on or around 10th August to cause all this trouble. Oh well :-)

[RobSwDev]

The reason I can't avoid promoting the warning to error with /nowarn is this line:

msbuild/src/Tasks/Copy.cs

Line 826 in 3c910ba

LogDiagnostic("Got {0} copying {1} to {2} and HR is {3}", e.ToString(), sourceFileState.Name, destinationFileState.Name, code);

LogDiagnostic logs a warning without an "excludable" error code :-(

[gowindu123]

Hi @RobSwDev @KalleOlaviNiemitalo ,

Could you please help me in understanding the conclusion on this issue. We are also facing this random issue on our build machine.

I have set the set MSBUILDALWAYSRETRY=1 before invoking the msbuild command. But still no luck. We have checked the process monitor and could see no files being used by any process like Anti Virus.

TIA

[KalleOlaviNiemitalo]

The "Got {0} copying {1} to {2} and HR is {3}" and "Retrying on ERROR_ACCESS_DENIED because MSBUILDALWAYSRETRY = 1" warnings were made suppressible in #9217 + #9284. The diagnostic codes are MSB3894 and MSB3895.

[gowindu123]

@KalleOlaviNiemitalo ,

Thanks for the reply, Does it mean, updating the msbuild version will fix the issue?

[gowindu123]

This is the version as of now present on out=r build server:
MSBuild version 17.7.2+d6990bcfa for .NET Framework

[KalleOlaviNiemitalo]

It means, if your version of MSBuild includes those pull requests, then you can use WarnAsError but suppress the warnings that are emitted because of MSBUILDALWAYSRETRY = 1.

If you are not using WarnAsError at all, then those pull requests don't help.