API Approval tests are skipped

[michaelklishin]

In master, when I run

dotnet test projects/Unit --filter "FullyQualifiedName~RabbitMQ.Client.Unit.APIApproval"

then the test reports that it has unsufficient source symbol information or something of that kind and stops with a fairly specific set of recommendations:

dotnet test projects/Unit --filter "FullyQualifiedName~RabbitMQ.Client.Unit.APIApproval"
/Users/antares/.nuget/packages/microsoft.build.tasks.git/1.0.0/build/Microsoft.Build.Tasks.Git.targets(24,5): warning : Could not find file '/Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbitmq_dotnet_client/_site/.git'. The source code won't be available via Source Link. [/Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbitmq_dotnet_client/projects/RabbitMQ.Client/RabbitMQ.Client.csproj]
/Users/antares/.nuget/packages/microsoft.build.tasks.git/1.0.0/build/Microsoft.Build.Tasks.Git.targets(24,5): warning : Could not find file '/Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbitmq_dotnet_client/_site/.git'. The source code won't be available via Source Link. [/Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbitmq_dotnet_client/projects/RabbitMQ.Client/RabbitMQ.Client.csproj]
Test run for /Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbitmq_dotnet_client/projects/Unit/bin/Debug/net461/Unit.dll(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
  X Approve [430ms]
  Error Message:
   System.Exception :
ApprovalTests is not detecting the proper source path

This is probably because you're missing the following
line in your .csproj file:
	  <DebugType>full</DebugType>
in the
<Project>
  <PropertyGroup>
element.

Solution:
a) Add <DebugType>full</DebugType> to your .csproj file.
b) OR Build->Advanced->DebugInfo to Full
  Stack Trace:
    at ApprovalTests.Namers.StackTraceParsers.StackTraceParser.get_SourcePath () [0x00033] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Namers.UnitTestFrameworkNamer.get_SourcePath () [0x00000] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Approvers.FileApprover.Approve () [0x00001] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Core.Approver.Verify (ApprovalTests.Core.IApprovalApprover approver, ApprovalTests.Core.IApprovalFailureReporter reporter) [0x00001] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Approvals.Verify (ApprovalTests.Core.IApprovalApprover approver, ApprovalTests.Core.IApprovalFailureReporter reporter) [0x00001] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Approvals.Verify (ApprovalTests.Core.IApprovalWriter writer, ApprovalTests.Core.IApprovalNamer namer, ApprovalTests.Core.IApprovalFailureReporter reporter) [0x00044] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Approvals.Verify (ApprovalTests.Core.IApprovalWriter writer) [0x0000f] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Approvals.VerifyWithExtension (System.String text, System.String fileExtensionWithDot, System.Func`2[T,TResult] scrubber) [0x00059] in <09be150c07864de7bafbca0757a4fd61>:0
  at ApprovalTests.Approvals.Verify (System.String text, System.Func`2[T,TResult] scrubber) [0x00001] in <09be150c07864de7bafbca0757a4fd61>:0
  at RabbitMQ.Client.Unit.APIApproval.Approve () [0x00032] in <729deeb8484847dcabbc2e050a761992>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <b6f643ae6ccb410db69142d5eaf9b9c1>:0

Test Run Failed.
Total tests: 1
     Failed: 1
 Total time: 1.8044 Seconds
Test run for /Users/antares/Development/RabbitMQ/umbrella.git/deps/rabbitmq_dotnet_client/projects/Unit/bin/Debug/netcoreapp3.1/Unit.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1.3850 Seconds

My .NET build system knowledge is nearly non-existent so I'm not sure what would be the best way to address this. My guess is that this started with #797 and related changes for 6.0.

[michaelklishin]

Also note that we seem to have this test executed twice, which I observed in other cases. This is on MacOS with .NET Core 3.1.

@lukebakken @bording does this make any sense to you? Can/should having two target frameworks result in tests being run N times?

[bording]

@michaelklishin The error you're seeing is due to the specific approval test library being used currently and how it tries to find the approval file location. Sometimes it does need the settings it mentions to work correctly.

Instead of making those changes, I think it makes sense to change approval libraries to one that doesn't have those requirements. There are a couple other options, so I'll open a PR to switch to one of those, probably can do that this weekend.

@lukebakken @bording does this make any sense to you? Can/should having two target frameworks result in tests being run N times?

Yes, since the test project itself is now multi-targeted, it will run the tests suite once for each target. If you only want to run it for one, you can pass in an additional parameter to specify the framework.

[michaelklishin]

OK, thank you for the explanation. If there are alternative libraries we would definitely consider a PR :)