Occasional MSBUILD error when running dotnet store on Windows

[JunTaoLuo]

We occasionally see the error when generating our runtime package store:

[Step 1/2] C:\b\w\33bf2c3795c96c7d\.dotnet\sdk\2.0.0-preview2-006479\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.CrossGen.targets(180,5):
error MSB3027: Could not copy "C:\b\w\33bf2c3795c96c7d\.rw\Build.RuntimeStore.References_2.0.0-preview2-25769\runtimesymbols\
microsoft.identitymodel.clients.activedirectory\3.13.9\lib\netstandard1.4\Microsoft.IdentityModel.Clients.ActiveDirectory.ni.dll"
to "C:\b\w\33bf2c3795c96c7d\.rw\SymbolsStagingDir\microsoft.identitymodel.clients.activedirectory\
3.13.9\lib\netstandard1.4\Microsoft.IdentityModel.Clients.ActiveDirectory.ni.dll".
Exceeded retry count of 10. Failed.
[C:\b\w\33bf2c3795c96c7d\src\Microsoft.AspNetCore.RuntimeStore\Microsoft.AspNetCore.RuntimeStore.csproj] 
[C:\b\w\33bf2c3795c96c7d\.build\KoreBuild.proj]

These failures are not persistent and occurs in about 1 in 10 runs on Windows only. Linux and macOS do not run into this behaviour. This started to occur about 2 weeks ago after we ingested the latest changes in the SDK for 2.0.0.

Any idea what's causing this?

cc @gkhanna79 @mikeharder @livarcocc

[JunTaoLuo]

This flakiness is regularly failing our build chains. Can we assign someone to investigate this with a higher priority?

cc @mikeharder @Eilon @livarcocc @gkhanna79 @muratg

[muratg]

@nguerrera Any thoughts on this one?

[Eilon]

Looks like some folks are OOF right now so we might have to wait a bit.

[nguerrera]

This started to occur about 2 weeks ago after we ingested the latest changes in the SDK for 2.0.0.

Do you have the build numbers before and after it started happening. I'm not aware of anything that changed late around this scenario.

[JunTaoLuo]

I think the issue started to manifest in our builds after we turned symbols generation on by default in #1231. Though I suspect the issue was already present when the symbols generation feature was added in https://github.com/dotnet/core-setup/issues/2144

[JunTaoLuo]

We are still seeing these failures, any updates? From the logs I'm also seeing this error:

09:37:09]	[Step 2/2] C:\b\w\33bf2c3795c96c7d\.dotnet\x86\sdk\2.1.0-preview1-007042\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.CrossGen.targets(180,5): error MSB3030: Could not copy the file "C:\b\w\33bf2c3795c96c7d\.rw\Build.RuntimeStore.References_2.1.0-preview1-27146\runtimesymbols\microsoft.identitymodel.clients.activedirectory\3.14.1\lib\netstandard1.3\Microsoft.IdentityModel.Clients.ActiveDirectory.ni.dll" because it was not found. 

[ryanbrandenburg]

This issue continues to happen for us:

[Step 2/2] C:\b\w\33bf2c3795c96c7d\.dotnet\x86\sdk\2.1.0-preview1-007042\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.CrossGen.targets(180,5): warning MSB3026: Could not copy "C:\b\w\33bf2c3795c96c7d\.rw\Build.RuntimeStore.References_2.1.0-preview1-27392\runtimesymbols\microsoft.identitymodel.clients.activedirectory\3.14.1\lib\netstandard1.3\Microsoft.IdentityModel.Clients.ActiveDirectory.ni.pdb" to "C:\b\w\33bf2c3795c96c7d\.rw\SymbolsStagingDir\microsoft.identitymodel.clients.activedirectory\3.14.1\lib\netstandard1.3\Microsoft.IdentityModel.Clients.ActiveDirectory.ni.pdb". Beginning retry 1 in 1000ms. The process cannot access the file 'C:\b\w\33bf2c3795c96c7d\.rw\Build.RuntimeStore.References_2.1.0-preview1-27392\runtimesymbols\microsoft.identitymodel.clients.activedirectory\3.14.1\lib\netstandard1.3\Microsoft.IdentityModel.Clients.ActiveDirectory.ni.pdb' because it is being used by another process. [C:\b\w\33bf2c3795c96c7d\src\Microsoft.AspNetCore.RuntimeStore\Microsoft.AspNetCore.RuntimeStore.csproj] [C:\b\w\33bf2c3795c96c7d\.dotnet\buildtools\korebuild\2.1.0-preview1-15538\KoreBuild.proj]

[Step 2/2] C:\b\w\33bf2c3795c96c7d\.dotnet\x86\sdk\2.1.0-preview1-007042\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.CrossGen.targets(180,5): error MSB3030: Could not copy the file "C:\b\w\33bf2c3795c96c7d\.rw\Build.RuntimeStore.References_2.1.0-preview1-27392\runtimesymbols\sqlitepclraw.bundle_green\1.1.8\lib\netcoreapp\SQLitePCLRaw.batteries_v2.ni.dll" because it was not found. [C:\b\w\33bf2c3795c96c7d\src\Microsoft.AspNetCore.RuntimeStore\Microsoft.AspNetCore.RuntimeStore.csproj] [C:\b\w\33bf2c3795c96c7d\.dotnet\buildtools\korebuild\2.1.0-preview1-15538\KoreBuild.proj]

[nguerrera]

@JunTaoLuo Same here. Have you tried building with Preview 7 and without the workarounds? I'm closing this as fixed by #3345, but if you are still having issues, I can reopen.

[JunTaoLuo]

I'll give it a try soon. We haven't updated our builds to use preview 7 just yet but will do so this week.