Enable dotnet test in libraries

[ViktorHofer]

No Merge. Part of batched rollout in May: #35202

Fixes #945
Requires #35538

• Stop inheriting IsTestSupportProject from IsTestProject as the VSTest target runs based on the latter one. This enables dotnet test on sln projects.
• Upgrading the SDK for vstest default filter support in .runsettings files.
• Replacing the coverlet.console with the inbuilt VSTest coverlet datacollector.
• Add both TRX and HTML logger for test runs
• Remove dead code and fix chmod condition
• Remove obsolete vstest.props/targets files as VSTest can now be invoked directly via dotnet test

Feature additions

• All of dotnet test can be used (either on a project or directly on the compiled assembly) means advanced filtering, test-listing (without running), blame data-collector, html logger, etc.
• HTML logger
• Ability to run dotnet test directly in the repo root (e.g. dotnet test -f netcoreapp5.0 /p:Subset=libs.tests or on a solution (e.g. dotnet test -f netcoreapp5.0 src\libraries\System.Text.RegularExpressions)
• Enabling F5 support for both .NETCoreApp and .NETFramework (+ without the need for launchSettings.json files).

Behavior Change

For measuring code coverage, the inbuilt dotnet test data collector is now used (which improves perf and stability and is actually the only supported way to measure coverage with dotnet test). The command changes from dotnet build /t:Test /p:Coverage=true to dotnet test -f netcoreapp5.0 /p:Coverage=true.

Plan for migrating from xunit.console -> dotnet test

1. This PR adds the option to use dotnet test.
2. In the (hopefully near) future when we believe that we covered all existing use cases with dotnet test without regressing the existing workflow and efficiency, we will add a warning to the test target indicating that it will be removed with the next batched rollout and will recommend to switch to dotnet test.
3. In the then next batched rollout, the Test target will be removed.

cc @stephentoub @danmosemsft

[ghost]

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[ViktorHofer]

cc @davidfowl

[stephentoub]

Part of batched rollout in May

What is the impact this has on the commands we use in the repo?

[ViktorHofer]

What is the impact this has on the commands we use in the repo?

Sorry, mentioned the behavior change only in the batched rollout issues and forgot to add it here as well. Please see the updated top post.

[EgorBo]

@ViktorHofer
um.. so it broke running tests for Android and iOS 😢
Any idea how can I run them now?
I used to run them via

./build.sh -os Android -arch arm64 -test

Now it's no-op.

Running individual tests still works:

cd src/libraries/System.Collections/tests
dotnet build /t:Test /p:TargetOS=Android /p:TargetArchitecture=arm64

But I don't know which folder and how should I run this command from to run all of them.

[ViktorHofer]

Now it's no-op.

It's not doing anything?

[ViktorHofer]

Just tried this out locally and the test execution of all projects still works (it invokes the Test target). Can I try your command out somehow?

[EgorBo]

It's not doing anything?

@ViktorHofer it only builds libs but doesn't launch RunTests.sh

Can I try your command out somehow?

Oh, you will need to install Android NDK/SDK so I better try to figure out myself. So are you saying -test should still work?

[ViktorHofer]

Isn't the right command build.sh libs.tests - test? I don't see the subset specified in your command.

[EgorBo]

@ViktorHofer
I've rebuilt the repo and tried again and now it works so sorry, everything is ok 👍

[ViktorHofer]

No problem at all :)