NUnit VSIX test adapters deprecation

[Shyam-Gupta]

At present, test execution from Visual Studio causes Test Platform to load test adapters from multiple locations. For details refer RFC: https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0022-User-Specified-TestAdapter-Lookup.md

For managed test projects, test adapters are generally available in the test dll output folder itself. Hence, if the adapter lookup is limited to the test dll folder only, then it can provide significant perf improvement. Our analysis shows 20% perf improvement when user executes <= 10 tests in a 10k NUnit test project. Similar improvement is seen for MSTest and xUnit test projects as well.

However, in case when NUnit VSIX adapter is installed, then the above assumption of adapter present in test dll output folder, does not hold true. In this scenario Test Explorer has to fallback to the old behavior of loading adapters from Extensions folder as well, thus incurring a perf hit.

Since in Dev16 we plan to deprecate VSIX test adapters, it will be good to put a message on NUnit VSIX adapters installation webpage recommending users to move to nuget based test adapter so that less users get impacted with perf degradation.

[rprouse]

Thanks for the update. Definitely something we should consider and think about. Can you match Dev16 to a VS release like 15.8 so that we have a better idea of timing?

[OsirisTerje]

Dev 16 should be VS 2019. No timeline yet, but should not be too far away, see https://blogs.msdn.microsoft.com/visualstudio/2018/06/06/whats-next-for-visual-studio/

[OsirisTerje]

@Shyam-Gupta Is the fact that the adapter is present in the output folder of a project dependent upon the adapter being referenced by the same project? Or are you doing "magic" stuff in the background to ensure this?
I ask because with the old project system the adapter is normally only referenced from one of the projects. With the new project system it is more likely to be present in all csprojs, since one often use a template (or copy/paste the basics) for those.

[Shyam-Gupta]

@OsirisTerje Yes adapter present in output folder of a project is dependent upon the adapter being referenced by the same project.

Though the scenario - 'adapter being referenced by a single project in a solution supports discovery\execution for other test projects having only framework reference' was working earlier, its not the supported one. With this change, discovery\execution will not work by default for the projects which do not have adapter reference and those projects will need to add adapter reference to take advantage of perf improvements.

Following is a snapshot from an internal build of VS, which shows the new options we are adding in Tools -> Options -> Test dialog:

As mentioned in the options, for improved performance, Test Platform will load test adapters from test assembly folder or as specified in runsettings file. Only for NUnit VSIX and MSTest v1 adapters, we will fallback to current behavior of loading adapters from all locations.

Also we found that NUnit nuget package: NunitTestAdapter.WithFramework adds both framework and adapter to the test project but only framework dll gets copied to the test dll output folder. With this change, Discovery\Execution will not work for projects referencing this nuget package. Since in the documentation of this package, its mentioned that its supported only for VS versions <= 2015, do you think we need to update this package so that the test adapter dll starts getting copied to project output folder ?

[OsirisTerje]

The adapter with framework package only exist for NUnit2, not for NUnit3, and I don't think we should move that forward.

[OsirisTerje]

Updated information in marketplace for :
NUnit3 adapter
NUnit2 adapter