Windows CoreFX CI (#18365)

* Add test list CL switch

* End-To-End Test Run on Windows

* Cleanup

* MAX_PATH Workaround

* Set Execution directory for CoreFX tests

* Add All CoreFX PR Tests

* Add test dependencies

* Add extra dependencies

* Add parallel test execution

* Disable OuterLoop tests and System.Data.SqlClient.*  tests

* Initialize maximum degree of parallelization to Environment.ProcessCount

* Remove unnecessary cli option

* Update Dependencies

* Add "enabled" property to tests

* Remove exclusions due to TestUtilities mismatch

* Add capability to run all tests for running Helix test lists directly

* Refactor build script to build testhost when skipping managed tests

* Disable failing System.Threading.Tests.EventWaitHandleTests.Ctor_InvalidMode

* Add switch to skip native test build

* Add testing documentation

* Don't run tests marked as "disabled" when running all available tests

* Add switch to build only testhost and remove Core_Root_Stage

* Clean up TopN.CoreFX.Windows.issues.json

* Refactor build-test.cmd

* PR feedback - build pipeline and documentation

* PR Feedback - Test Helper headers and comments

* Fix buildtesthost option for only building CoreFX test dependencies

* Disable intermittently failing test DrawBezier_PointFs

Author: A-And

Documentation/building/testing-with-corefx.md

@@ -3,6 +3,7 @@ Testing with CoreFX
 
 It may be valuable to use CoreFX tests to validate your changes to CoreCLR or mscorlib.
 
+## Building CoreFX against CoreCLR
 **NOTE:** The `BUILDTOOLS_OVERRIDE_RUNTIME` property no longer works.
 
 To run CoreFX tests with an updated System.Private.Corelib.dll, [use these instructions](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits).
@@ -27,3 +28,74 @@ Use the following instructions to test a change to the dotnet/coreclr repo using
 
 [run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) will clone dotnet/corefx and run steps 2-4 above automatically.  It is primarily intended to be run by the dotnet/coreclr CI system, but it might provide a useful reference or shortcut for individuals running the tests locally.
 
+## Using the built CoreCLR testhost 
+**These instructions are currently Windows only.**
+
+Instead of copying CoreCLR binaries you can also test your changes with an existing CoreFX build or CoreCLR's CI assemblies
+
+### Locally-built CoreFX 
+Once you have finished steps 1, 2. and 4. above execute the following instructions to test your local CLR changes with the built-CoreFX changes.
+
+1. From `<coreclr_root>` run `build-test.cmd <arch> <build_type> skipmanaged` to generate the test host.
+2. Navigate to `<corefx_root>\bin\tests\` and then the test you would like to run
+3. Run
+
+```cmd
+<coreclr_root>\bin\<os>.<arch>.<build_type>\testhost\dotnet.exe <corefx_root>\bin\tests\<testname>\xunit.console.netcore.exe <testname>.dll
+```
+followed by any extra command-line arguments.
+
+For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR.
+```
+pushd C:\corefx\bin\tests\System.Collections.Tests
+C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonwindowstests
+```
+
+### CI Script
+CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs. To run tests against pre-built binaries you can execute the following from the CoreCLR repo root:
+
+1. `.\build.cmd <arch> <build_type>`
+2. `.\build-test.cmd <arch> <build_type> skipmanaged` - generates the test host
+3. `.\tests\runtest.cmd <arch> <build_type> corefxtests|corefxtestsall` - runs CoreFX tests
+
+CoreFXTests - runs all tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList`
+CoreFXTestsAll - runs all tests available in the test list found at the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt`.
+
+### Helix Testing
+To use Helix-built binaries, substitute the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt` with one acquired from a Helix test run and run the commands above.
+
+### Test List Format
+The tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList` should conform to the following format -
+```json
+    {
+        "name": "<Fully Qualified Assembly Name>", //e.g. System.Collections.Concurrent.Tests
+        "enabled": true|false, // Defines whether a test assembly should be run. If set to false any tests with the same name will not be run even if corefxtestsall is specified
+        "exclusions": {
+            "namespaces": // Can be null
+              [
+                {
+                    "name": "System.Collections.Concurrent.Tests", // All test methods under this namespace will be skipped
+                    "reason": "<Reason for exclusion>" // This should be a link to the GitHub issue describing the problem
+                }
+              ]
+            "classes": // Can be null
+            [
+                {
+                    "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests", // All test methods in this class will be skipped
+                    "reason": "<Reason for exclusion>"
+                }
+            ]
+            "methods": // Can be null
+            [
+                {
+                    "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ReferenceType_null",
+                    "reason": "<Reason for exclusion>"
+                },
+                {
+                    "name": "System.Collections.Concurrent.Tests.ConcurrentDictionaryTests.TestAddNullValue_IDictionary_ValueType_null_add",
+                    "reason": "<Reason for exclusion>"
+                }
+            ]
+        }
+    }
+```

build-test.cmd

@@ -46,6 +46,9 @@ set processedArgs=
 set __unprocessedBuildArgs=
 set __RunArgs=
 set __BuildAgainstPackagesArg=
+set __SkipRestorePackages=
+set __SkipManaged=
+set __SkipNative=
 set __RuntimeId=
 set __ZipTests=
 set __TargetsWindows=1
@@ -74,8 +77,11 @@ if /i "%1" == "release"               (set __BuildType=Release&set processedArgs
 if /i "%1" == "checked"               (set __BuildType=Checked&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 
 if /i "%1" == "skipmanaged"           (set __SkipManaged=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "skipnative"            (set __SkipNative=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "buildtesthostonly"     (set __SkipNative=1&set __SkipManaged=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "toolset_dir"           (set __ToolsetDir=%2&set __PassThroughArgs=%__PassThroughArgs% %2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
 if /i "%1" == "buildagainstpackages"  (set __ZipTests=1&set __BuildAgainstPackagesArg=-BuildTestsAgainstPackages&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "skiprestorepackages"   (set __SkipRestorePackages=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "ziptests"              (set __ZipTests=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "crossgen"              (set __DoCrossgen=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "runtimeid"             (set __RuntimeId=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
@@ -160,6 +166,8 @@ REM ============================================================================
 call "%__TestDir%\setup-stress-dependencies.cmd" /arch %__BuildArch% /outputdir %__BinDir%
 @if defined _echo @echo on
 
+if defined __SkipNative goto skipnative
+
 REM =========================================================================================
 REM ===
 REM === Native test build section
@@ -224,8 +232,7 @@ if errorlevel 1 (
 :skipnative
 
 set "__TestWorkingDir=%__RootBinDir%\tests\%__BuildOS%.%__BuildArch%.%__BuildType%"
-
-if not defined __BuildAgainstPackagesArg goto SkipRestoreProduct
+if "%__SkipRestorePackages%" == 1 goto SkipRestoreProduct
 REM =========================================================================================
 REM ===
 REM === Restore product binaries from packages
@@ -245,6 +252,7 @@ set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
 
 call "%__ProjectDir%\run.cmd" build -Project=%__ProjectDir%\tests\build.proj -BatchRestorePackages -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %__unprocessedBuildArgs%
 
+if not defined __BuildAgainstPackagesArg goto SkipRestoreProduct
 set __BuildLogRootName=Tests_GenerateRuntimeLayout
 set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log
 set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn
@@ -263,7 +271,7 @@ echo %__MsgPrefix% Restored CoreCLR product from packages
 
 :SkipRestoreProduct
 
-if defined __SkipManaged exit /b 0
+if defined __SkipManaged goto SkipManagedBuild
 
 REM =========================================================================================
 REM ===
@@ -316,6 +324,7 @@ for /l %%G in (1, 1, %__BuildLoopCount%) do (
     set __AppendToLog=true
 )
 
+:SkipManagedBuild
 REM Prepare the Test Drop
 REM Cleans any NI from the last run
 powershell -NoProfile "Get-ChildItem -path %__TestWorkingDir% -Include '*.ni.*' -Recurse -Force | Remove-Item -force"
@@ -344,46 +353,44 @@ if defined __BuildAgainstPackagesArg (
   )
 )
 
-echo %__MsgPrefix%Creating test wrappers...
-
-set RuntimeIdArg=
-set TargetsWindowsArg=
-
-if defined __RuntimeId (
-    set RuntimeIdArg=-RuntimeID="%__RuntimeId%"
-)
-
-if "%__TargetsWindows%"=="1" (
-    set TargetsWindowsArg=-TargetsWindows=true
-) else if "%__TargetsWindows%"=="0" (
-    set TargetsWindowsArg=-TargetsWindows=false
-)
+REM =========================================================================================
+REM ===
+REM === Create the test overlay
+REM ===
+REM =========================================================================================
+echo %__MsgPrefix%Creating test overlay...
 
-set __BuildLogRootName=Tests_XunitWrapper
+set __BuildLogRootName=Tests_Overlay_Managed
 set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log
 set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn
 set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err
 set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%"
 set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%"
 set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
 
-call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -BuildWrappers -MsBuildEventLogging=" " -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %TargetsWindowsArg% %__unprocessedBuildArgs%
+call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -testOverlay -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %RuntimeIdArg% %__unprocessedBuildArgs%
 if errorlevel 1 (
-    echo Xunit Wrapper build failed
+    echo %__MsgPrefix%Error: build failed. Refer to the build log files for details:
+    echo     %__BuildLog%
+    echo     %__BuildWrn%
+    echo     %__BuildErr%
     exit /b 1
 )
 
-echo %__MsgPrefix%Creating test overlay...
+xcopy /s /y "%CORE_ROOT_STAGE%" "%CORE_ROOT%"
 
-set __BuildLogRootName=Tests_Overlay_Managed
+REM Create the test host necessary for running CoreFX tests 
+REM The test host includes a dotnet executable, system libraries and CoreCLR assemblies found in CoreRoot
+
+set __BuildLogRootName=Tests_CoreFX_Testhost
 set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log
 set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn
 set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err
 set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%"
 set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%"
 set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
 
-call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -testOverlay -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %RuntimeIdArg% %__unprocessedBuildArgs%
+call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -testHost -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %RuntimeIdArg% %__unprocessedBuildArgs%
 if errorlevel 1 (
     echo %__MsgPrefix%Error: build failed. Refer to the build log files for details:
     echo     %__BuildLog%
@@ -392,7 +399,37 @@ if errorlevel 1 (
     exit /b 1
 )
 
-xcopy /s /y "%CORE_ROOT_STAGE%" "%CORE_ROOT%"
+if defined __SkipManaged goto SkipBuildingWrappers
+
+echo %__MsgPrefix%Creating test wrappers...
+
+set RuntimeIdArg=
+set TargetsWindowsArg=
+
+if defined __RuntimeId (
+    set RuntimeIdArg=-RuntimeID="%__RuntimeId%"
+)
+
+if "%__TargetsWindows%"=="1" (
+    set TargetsWindowsArg=-TargetsWindows=true
+) else if "%__TargetsWindows%"=="0" (
+    set TargetsWindowsArg=-TargetsWindows=false
+)
+
+set __BuildLogRootName=Tests_XunitWrapper
+set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log
+set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn
+set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err
+set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%"
+set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%"
+set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
+
+call %__ProjectDir%\run.cmd build -Project=%__ProjectDir%\tests\runtest.proj -BuildWrappers -MsBuildEventLogging=" " -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %TargetsWindowsArg% %__unprocessedBuildArgs%
+if errorlevel 1 (
+    echo Xunit Wrapper build failed
+    exit /b 1
+)
+:SkipBuildingWrappers
 
 set __CrossgenArg = ""
 if defined __DoCrossgen (

config.json

@@ -336,6 +336,12 @@
       "values": [],
       "defaultValue": ""
     },
+    "CreateTestHost": {
+      "description": "Place test dependencies in the /bin/testhost folder to be used to run CoreFX tests",
+      "valueType": "target",
+      "values": [],
+      "defaultValue": ""
+    },
     "Verbosity": {
       "description": "Sets build verbosity.",
       "valueType": "passThrough",
@@ -504,6 +510,12 @@
             "CreateTestOverlay": "default"
           }
         },
+        "testHost": {
+          "description": "Runs test host target, which creates a dotnet executable from the built CoreCLR repo to run pre-built CoreFX test assemblies",
+          "settings": {
+            "CreateTestHost": "default"
+          }
+        },
         "priority": {
           "description": "Sets CLRTestPriorityToBuild property.",
           "settings": {

dependencies.props

@@ -36,7 +36,7 @@
     <PgoDataPackageVersion>99.99.99-master-20180621-0050</PgoDataPackageVersion>
     <MicrosoftNETCoreRuntimeCoreCLRPackageVersion>2.2.0-preview1-26621-01</MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
     <XunitPackageVersion>2.2.0-beta2-build3300</XunitPackageVersion>
-    <XunitConsoleNetcorePackageVersion>1.0.2-prerelease-00177</XunitConsoleNetcorePackageVersion>
+    <XunitConsoleNetcorePackageVersion>2.2.0-preview1-02830-02</XunitConsoleNetcorePackageVersion>
     <XunitPerformanceApiPackageVersion>1.0.0-beta-build0015</XunitPerformanceApiPackageVersion>
     <MicrosoftDiagnosticsTracingTraceEventPackageVersion>2.0.4</MicrosoftDiagnosticsTracingTraceEventPackageVersion>
     <CommandLineParserVersion>2.2.0</CommandLineParserVersion>
@@ -137,6 +137,7 @@
     <XUnitDependency Include="xunit"/>
     <XUnitDependency Include="xunit.assert"/>
     <XUnitDependency Include="xunit.core"/>
+    <XUnitDependency Include="xunit.extensibility.core" />
     <XUnitDependency Include="xunit.runner.console"/>
     <XUnitDependency Include="xunit.runner.msbuild"/>
     <XUnitDependency Include="xunit.runner.utility"/>

tests/CoreFX/CoreFXTestListURL.txt

@@ -0,0 +1 @@
+https://cloudcijobs.blob.core.windows.net/coreclrci/CoreFXArchives/TestList.json

tests/CoreFX/CoreFXTestListURL_Linux.txt

@@ -0,0 +1 @@
+https://cloudcijobs.blob.core.windows.net/corertci/TestList_Linux.json

tests/CoreFX/CoreFXTestListURL_OSX.txt

@@ -0,0 +1 @@
+https://cloudcijobs.blob.core.windows.net/corertci/TestList_OSX.json

tests/CoreFX/TopN.CoreFX.x64.Unix.issues.json

@@ -0,0 +1,48 @@
+// The expected format of this file can be found at https://github.com/dotnet/coreclr/blob/master/Documentation/building/testing-with-corefx.md
+[
+    {
+        "name": "System.Collections.Tests",
+        "exclusions": {
+            "namespaces": null,
+            "classes": null,
+            "methods": [
+                {
+                    "name": "System.Collections.Tests.SortedList_Generic_Tests_string_string.ICollection_Generic_Remove_DefaultValueContainedInCollection",
+                    "reason": "Exclusion Sample"
+                }
+            ]
+        }
+    },
+    {
+        "name": "System.Runtime.Extensions.Tests",
+        "exclusions": {
+            "namespaces": null,
+            "classes": null,
+            "methods": []
+        }
+    },
+    {
+        "name": "System.Runtime.Tests",
+        "exclusions": {
+            "namespaces": null,
+            "classes": null,
+            "methods": []
+        }
+    },
+    {
+        "name": "System.Threading.Tasks.Tests",
+        "exclusions": {
+            "namespaces": null,
+            "classes": null,
+            "methods": []
+        }
+    },
+    {
+        "name": "System.Threading.Tests",
+        "exclusions": {
+            "namespaces": null,
+            "classes": null,
+            "methods": []
+        }
+    }
+]