Issues with NUnit

[boldt]

I cannot get coverlet running. I added via NuGet:

• NUnit: 3.12.0
• NUnit.ConsoleRunner: 3.10.0
• NUnit3TestAdapter: 3.13.0
• coverlet.msbuild: 2.6.1

I ran my build and test as follows:

dotnet build
dotnet test --no-build /p:CollectCoverage=true

I'm getting the "strong name" issue, see #338 (partly German log):

[xUnit.net 00:00:00.78] Test: Catastrophic failure: System.IO.FileLoadException: Die Datei oder Assembly "xunit.runner.utility.net452, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c" oder eine Abhängigkeit davon wurde nicht gefunden. Die Signatur mit starkem Namen konnte nicht überprüft werden. Die Assembly wurde möglicherweise geändert oder verzögert aber nicht vollständig mit dem richtigen privaten Schlüssel signiert. (Ausnahme von HRESULT: 0x80131045)
Dateiname: "xunit.runner.utility.net452, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c"
   bei System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   bei System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   bei System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   bei System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   bei System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
   bei System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   bei System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   bei System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
   bei Xunit.AppDomainManager_AppDomain.CreateObject[TObject](AssemblyName assemblyName, String typeName, Object[] args) in C:\Dev\xunit\xunit\src\xunit.runner.utility\AppDomain\AppDomainManager_AppDomain.cs:Zeile 78.
   bei Xunit.DiaSessionWrapper..ctor(String assemblyFilename) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Utility\DiaSessionWrapper.cs:Zeile 22.
   bei Xunit.VisualStudioSourceInformationProvider..ctor(String assemblyFileName) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\VisualStudioSourceInformationProvider.cs:Zeile 24.
   bei Xunit.XunitFrontController..ctor(AppDomainSupport appDomainSupport, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:Zeile 70.
   bei Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo) in C:\Dev\xunit\xunit\src\xunit.runner.visualstudio\VsTestRunner.cs:Zeile 495.

WRN: Protokollierung der Assemblybindung ist AUS.
Sie können die Protokollierung der Assemblybindungsfehler aktivieren, indem Sie den Registrierungswert [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) auf 1 festlegen.
Hinweis: Die Protokollierung der Assemblybindungsfehler führt zu einer gewissen Leistungseinbuße.
Sie können dieses Feature deaktivieren, indem Sie den Registrierungswert [HKLM\Software\Microsoft\Fusion!EnableLog] entfernen.

In #338. two workarounds are desribed:

a) Downgrade to 2.4.0
b) Add /p:Exclude="[xunit.*]*"

Workaround a) Downgrade to 2.4.0

When I downgrade, i get the following error:

The NUnit 3 driver encountered an error while executing reflected code.
   bei NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   bei NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
   bei NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   bei NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   bei NUnit.Engine.Runners.TestDomainRunner.LoadPackage()
   bei NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   bei NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
   bei NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
   bei NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter)
   bei NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
Innerexception: System.TypeInitializationException: Der Typeninitialisierer für "Coverlet.Core.Instrumentation.Tracker.nunit.framework_9fb539db-510a-43a3-8fc0-8f135ed99489" hat eine Ausnahme verursacht. ---> System.Security.VerificationException: Dieser Vorgang kann die Laufzeit destabilisieren.
   bei Coverlet.Core.Instrumentation.Tracker.nunit.framework_9fb539db-510a-43a3-8fc0-8f135ed99489..cctor()
   --- Ende der internen Ausnahmestapelüberwachung ---
   bei Coverlet.Core.Instrumentation.Tracker.nunit.framework_9fb539db-510a-43a3-8fc0-8f135ed99489.RecordHit(Int32 )
   bei NUnit.Framework.Api.FrameworkController..ctor(String assemblyNameOrPath, String idPrefix, IDictionary settings)
[xUnit.net 00:00:01.03] Skipping: Test (could not find dependent assembly 'nunit.framework, Version=3.12.0')
In C:\Users\dennis.boldt\git\smart-factory\zenon-addins\Build\Test.dll ist kein Test verfügbar. Stellen Sie sicher, dass Testdiscoverer und -executor registriert und die Versionseinstellungen für Plattform und Framework richtig sind, und versuchen Sie es anschließend noch mal.

Workaround b) Add /p:Exclude="[xunit.*]*"

When I add /p:Exclude="[xunit.*]*", I see some of the following warnings:

warning : [coverlet] Hits file: '[...]' not found for module: 'Utils' 

Additionally, I see a coverage report with 0%.

Install coverlet globally

dotnet tool install --global coverlet.console
coverlet Build\Test.dll --target "dotnet" --targetargs "test --no-build" --exclude "[xunit.*]*"

It shows again warning : [coverlet] Hits file: '[...]' not found for module: 'Utils'

Conclusion

I cannot get coverlet work. Any idea is appreciated.

[MarcoRossignoli]

Can you try this #347?
exclude /p:Exclude="[xunit.*]*" with last version(no downgrade) <- sorry you've already done if I understood well.

[boldt]

I just switched to 2.5.1 and it works with dotnet test --no-build /p:CollectCoverage=true now:

[Edit] Also coverlet standalone works with 2.5.1: coverlet Build\Test.dll --target "dotnet" --targetargs "test --no-build"

[MarcoRossignoli]

Ok, please can you try last version with verbose logging to understand what's happen? https://github.com/tonerdo/coverlet/blob/master/Documentation/Troubleshooting.md#msbuild-integration

[boldt]

To debug with msbuildlog, I swicthed back to 2.6.1 and it works now even without /p:Exclude="[xunit.*]*". I also do not have the "strong name"-error anymore and it prints out percentages (two digits instead of one):

No clue what's different. I still have the following warning:

warning : [coverlet] Hits file: '[...]' not found for module: 'Utils'

[MarcoRossignoli]

Yes we rounded percentages #397
And for warning issue we fixed in master version #406 because is a false "negative" if module is not "tested"

You could try our nightly build to understand if it's fixed https://www.myget.org/feed/coverlet-dev/package/nuget/coverlet.msbuild/2.6.1-g0dd56242f2 (Wed, 29 May 2019 00:03:09 GMT)

[MarcoRossignoli]

You can fix verbosity with -v:q for msbuild(https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test?tabs=netcore21)
and with -v quiet for dotnet tool(console) https://github.com/tonerdo/coverlet#global-tool

[MarcoRossignoli]

Feel free to close this issue if your issue is resolved.

[MarcoRossignoli]

Ah for 0% coverage is possible you hit a know transient bug #110 we'll resolve ASAP with new collector features https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md (near to release).

[boldt]

I will check the Nightly on Monday.

[MarcoRossignoli]

Sure let us know!

[MarcoRossignoli]

@boldt we released new version https://www.nuget.org/packages/coverlet.msbuild/ let me know.

[boldt]

The warning is gone with 2.6.2. Thanks.