Hardcode LKG version of ucrtbased.dll

[wtgodbe]

Should unblock CI errors that look like:

F:\workspace_work\1\s.dotnet\sdk\3.0.100-preview6-012264\Microsoft.Common.CurrentVersion.targets(4583,5): error MSB3030: Could not copy the file "C:\Program Files (x86)\Microsoft SDKs\Windows Kits\10\ExtensionSDKs\Microsoft.UniversalCRT.Debug\10.0.18362.0\Redist\Debug\x64\ucrtbased.dll" because it was not found.

@MattGal and I investigated this, it appears to be a bug with the new version of VS that incorrectly sets UCRTVersion to a version that isn't installed on the machine. This workaround hard-codes the LKG version until the bug gets resolved (or permanently, if we decide that we don't want to risk getting broken like this again).

@MattGal @hoyosjs @BruceForstall @jkoritzinsky @AaronRobinsonMSFT PTAL

[wtgodbe]

Crap, formatting is still failing. Looks like they get UCRT a different way. looking.

[wtgodbe]

CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.12/Modules/CMakeTestCCompiler.cmake:52 (message):
The C compiler
"C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x64/cl.exe"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: F:/workspace/_work/1/s/bin/nmakeobj/Windows_NT.x64.Checked/CMakeFiles/CMakeTmp

Run Build Command:"nmake" "/nologo" "cmTC_dd24d\fast"
	"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86\nmake.exe" -f CMakeFiles\cmTC_dd24d.dir\build.make /nologo -L                  CMakeFiles\cmTC_dd24d.dir\build
Building C object CMakeFiles/cmTC_dd24d.dir/testCCompiler.c.obj
	C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx86\x64\cl.exe @C:\Users\vsagent\AppData\Local\Temp\nm325F.tmp
testCCompiler.c
Linking C executable cmTC_dd24d.exe
	"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_exe --intdir=CMakeFiles\cmTC_dd24d.dir --manifests  -- C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx86\x64\link.exe /nologo @CMakeFiles\cmTC_dd24d.dir\objects1.rsp @C:\Users\vsagent\AppData\Local\Temp\nm35FA.tmp
LINK Pass 1: command "C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx86\x64\link.exe /nologo @CMakeFiles\cmTC_dd24d.dir\objects1.rsp /out:cmTC_dd24d.exe /implib:cmTC_dd24d.lib /pdb:F:\workspace\_work\1\s\bin\nmakeobj\Windows_NT.x64.Checked\CMakeFiles\CMakeTmp\cmTC_dd24d.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:CMakeFiles\cmTC_dd24d.dir/intermediate.manifest CMakeFiles\cmTC_dd24d.dir/manifest.res" failed (exit code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'ucrtd.lib'
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe"' : return code '0xffffffff'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX86\x86\nmake.exe"' : return code '0x2'
Stop.

[wtgodbe]

Formatting doesn't happen in official builds, so if this passes everything else, it may be best to merge it (to unblock preview7 builds), then investigate the formatting failure in master

[RussKeldorph]

@hoyosjs Can you help with this? You mentioned something about ucrt earlier.

[wtgodbe]

OSX failure may be unrelated, there's nothing about ucrt in there. It was the Windows jobs that were failing previously.

MSBUILD : error MSB4166: Child node "4" exited prematurely. Shutting down. Diagnostic information may be found in files in "/Users/vsts/agent/2.153.2/work/1/s/bin/Logs/MsbuildDebugLogs" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.

[wtgodbe]

I missed this before (it's in the logs of other failing PRs):

CSC : warning CS1668: Invalid search path 'C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64' specified in 'LIB environment variable' -- 'directory does not exist' [F:\workspace_work\1\s\tests\src\GC\API\GCHandle\AddrOfPinnedObject_neg.csproj]

This shows up for every test project.

[RussKeldorph]

@wtgodbe It looks like the windows jobs have gotten past the Build Tests phase this time, so it probably is something about certain build machines.

[wtgodbe]

If the OSX failure is unrelated, then this is good enough as a workaround for now.

[hoyosjs]

My main question here is will this require every dev on the team to install the given version of the package. (For example, if Dev16 is detected on the box, it gets used as the default. And how will this translate for people who already nuked Dev15)

That being said, I'm fine with this as a stop gap solution only if you can ensure this one is always installed in the machines we build on.

[wtgodbe]

Yes, @MattGal and I confirmed that the hardcoded version is installed on the build machines we use. We can remove this workaround once either the VS bug is fixed, or we move up to the next dot version.

[hoyosjs]

Cool, can you please open an issue to track it?

[wtgodbe]

https://github.com/dotnet/coreclr/issues/25447

[jkotas]

So does this mean that close to everybody building CoreCLR locally will get a build break, will need to trace down this issue, and apply a workaround (what is the workaround exactly?) somehow?

This is major time sink and productivity loss across both our team and our community. Can we do better? E.g. apply this workaround conditionally?

[MattGal]

So does this mean that close to everybody building CoreCLR locally will get a build break, will need to trace down this issue, and apply a workaround (what is the workaround exactly?) somehow?

This is major time sink and productivity loss across both our team and our community. Can we do better? E.g. apply this workaround conditionally?

Best I can tell, this would be "only people who keep their VS 2017 fully patched," and possibly actually "only people who recently have done a clean install of VS 2017"

[hoyosjs]

This is major time sink and productivity loss across both our team and our community. Can we do better? E.g. apply this workaround conditionally?

Good idea.

@wtgodbe can we try:

<None Include="$(ExtensionSdkDir)/Microsoft.UniversalCRT.Debug/$(UCRTVersion)/Redist/Debug/$(Platform)/ucrtbased.dll" CopyToOutputDirectory="Always"
        Condition="Exists($(ExtensionSdkDir)/Microsoft.UniversalCRT.Debug/$(UCRTVersion)/Redist/Debug/$(Platform)/ucrtbased.dll)" />
    
<!-- There's a bug in VS that causes UCRTVersion env var to get set to a version that isn't actually installed on the machine. -->
<!-- Until that gets resolved, we need this workaround to grab the last "known" good version of ucrtbased.dll -->
<None Include="$(ExtensionSdkDir)/Microsoft.UniversalCRT.Debug/10.0.17763.0/Redist/Debug/$(Platform)/ucrtbased.dll" CopyToOutputDirectory="Always"
        Condition="!Exists($(ExtensionSdkDir)/Microsoft.UniversalCRT.Debug/$(UCRTVersion)/Redist/Debug/$(Platform)/ucrtbased.dll)" />

[wtgodbe]

Good call, I've pushed your suggestion @hoyosjs

[hoyosjs]

Looks like I forgot to quote the string in the exists condition for the snippet I sent.

[wtgodbe]

Windows failure is a test timeout

Timed out after 60000ms waiting for remote process.

\tProcess ID: 4088
\tHandle: 1008
\tName: dotnet
\tMainModule: C:\dotnetbuild\work\8e0dd96f-bcbd-4152-8db4-1b7abae7ec94\Payload\dotnet.exe
\tStartTime: 6/27/2019 2:33:33 AM
\tTotalProcessorTime: 00:00:00.9375000
\tThreads:
\t\tThread #1 (OS 0x904)
\t\t\t[PrestubMethodFrame] (System.Text.RegularExpressions.RegexInterpreter.Go)
\t\t\tSystem.Text.RegularExpressions.RegexRunner.Scan(System.Text.RegularExpressions.Regex, System.String, Int32, Int32, Int32, Int32, Boolean, System.TimeSpan)
\t\t\tSystem.Text.RegularExpressions.Regex.Run(Boolean, Int32, System.String, Int32, Int32, Int32)
\t\t\tSystem.Text.RegularExpressions.Regex.IsMatch(System.String, Int32)
\t\t\tSystem.Text.RegularExpressions.Tests.RegexCacheTests.<Ctor_Cache_Shrink_cache>b__3_0()
\t\t\t[DebuggerU2MCatchHandlerFrame]
\t\t\t[HelperMethodFrame_PROTECTOBJ]
\t\t\tSystem.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
\t\t\tMicrosoft.DotNet.RemoteExecutorHost.Program.Main(System.String[])
\t\t\t[GCFrame]
\t\t\t[GCFrame]
\t\tThread #2 (OS 0x504) [Finalizer]
\t\t\t[DebuggerU2MCatchHandlerFrame]

[jkotas]

@wtgodbe Are you working on fixing the Formatting job failures too?

[wtgodbe]

I haven't taken a look at the formatting job, as it isn't blocking the official builds. It's likely related to the LIB environment variable though, as per #25444 (comment). It may be possible to apply a similar workaround there by appending 10.0.17763.0 to the search paths in LIB. CC @MattGal, who had a copy of the string on his box yesterday.

[MattGal]

Sure, here's the default Lib paths from the 15.9.12 machine:

LIB=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x86;C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\lib\x86;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x86;C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x86;C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x86;
LIBPATH=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x86;C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\lib\x86;C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.16.27023\lib\x86\store\references;C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.17763.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.17763.0;C:\Windows\Microsoft.NET\Framework\v4.0.30319;

... appending the relevant 17763 ones to the existing lib should work, or overriding the value entirely as a workaround if files that should be on those paths cannot be located.

[jkotas]

I have opened https://github.com/dotnet/coreclr/issues/25499 to have this tracked.

I tried to find the right place where to add this workaround, but I was not able to figure it out. The python formatting script fails on my local machine much sooner due number of other problem that makes diagnosing this hard.

It would be great if somebody more familiar with this can take a look.

cc @RussKeldorph