ASP.NET functional test projects not detected by convention

[bricelam]

To match the ASP.NET guidelines we'd need to update this line to include assemblies that end with .FunctionalTests.

I believe these should be categorized as unit tests and not integration tests because they're not typically not as heavy-weight as the integration test category implies. They can probably run during live testing. But that's just my opinion.

Another, somewhat related thing that KoreBuild did was run unit tests before these functional tests. The idea being that if a unit test failed, functional tests would probably fail too. We wanted to fail fast and the unit tests are typically a lot faster to run than the functionals.

cc @natemcmaster @ryanbrandenburg @rynowak

[rynowak]

Is there any documented guidance about how these categories of tests differ within arcade? I couldn't find anything when I was porting razor.

[bricelam]

No docs. Looking at the implementation, I think integration tests don’t run during CI and they’re excluded from live unit testing in VS.

[markwilkie]

@garath - could you follow up with arcade working group folks so someone can plug this doc hole please?

[markwilkie]

Did this doc hole get plugged yet? cc/ @jcagme

[jcagme]

The last thing I added to docs was related to subscriptions no tests. Maybe @garath has a better idea?

[garath]

@markwilkie @jcagme No movement yet; will pick up now.

[garath]

The naming standards are defined in Arcade SDK but no description of the behavior differences. To be clear, @bricelam and @markwilkie, is ASP asking for .FunctionalTests to be automatically recognized by Arcade, or is this simply clarifying Arcade's treatment of the different test types?

[tmat]

Can ASP rename .FunctionalTests to .Tests or .UnitTests?

UnitTests/IntegrationTests/PerformanceTests are categories that can be used on CI to choose tests to run on specific CI leg.

Typically IntegrationTests are heavy weight, need some extra environment to be prepared for them (e.g. Visual Studio with certain extensions installed) or need to run on a specific machine pool with that environment, etc. Used usually when repo produces packages/installers/vsixes that are installed prior running IntergrationTests.

PerformanceTests are meant to be run on machine that's specifically set up to run performance tests, track results over time, etc.

UnitTests/Tests are any other tests.

[bricelam]

is ASP asking for .FunctionalTests to be automatically recognized by Arcade, or is this simply clarifying Arcade's treatment of the different test types?

I'm mostly concerned about documenting Arcade's treatment of the different test types and how to include other patterns.