Track1 testfx re-design and improvement

documentation/development-docs/azure-powershell-developer-guide.md

@@ -248,17 +248,58 @@ _Note_: As mentioned in the prerequisites section, set the PowerShell [execution
 
 ## Using Azure TestFramework
 
-Please see our guide on [Using Azure TestFramework](../testing-docs/using-azure-test-framework.md) for information on how to setup the appropriate connection string and record tests using the `Microsoft.Rest.ClientRuntime.Azure.TestFramework` package.
+Please see our guide on [Using Azure TestFramework](../testing-docs/using-azure-test-framework.md) for information on how to setup the appropriate connection string and record tests.
 
 ## Scenario Tests
 
+### Adding Test Project
+
+- Create a new folder called `ScenarioTests`
+- Create a new folder called `SessionRecords`
+- Inside the `ScenarioTests` folder, create a new class called `<SERVICE>TestRunner`
+- In the `<SERVICE>TestRunner` class, it should have the similar field and constructor like shown below. The parameter values passed in are based on the real situation.
+```csharp
+    protected readonly ITestRunner TestRunner;
+
+    protected <SERVICE>TestRunner(ITestOutputHelper output)
+    {
+        TestRunner = TestManager.CreateInstance(output)
+            .WithProjectSubfolderForTests("ScenarioTests")
+            .WithNewPsScriptFilename($"{GetType().Name}.ps1")
+            .WithCommonPsScripts(new[]
+            {
+                @"Common.ps1",
+                @"../AzureRM.Resources.ps1",
+                @"../AzureRM.Storage.ps1"
+            })
+            .WithNewRmModules(helper => new[]
+            {
+                helper.RMProfileModule,
+                ...
+            })
+            .WithNewRecordMatcherArguments(
+                userAgentsToIgnore: new Dictionary<string, string>
+                {
+                    ...
+                },
+                resourceProviders: new Dictionary<string, string>
+                {
+                    ...
+                }
+            )
+            .Build();
+    }
+```
+
 ### Adding Scenario Tests
 
 - Create a new class in `<SERVICE>.Test`
+    - The new class must inherit from the `<SERVICE>TestRunner` class in this project.
     - Add `[Fact]` as an attribute to every test
     - Add `[Trait(Category.AcceptanceType, Category.CheckIn)]` as an attribute to any test that should be run during CI in Playback mode.
     - Add `[Trait(Category.AcceptanceType, Category.LiveOnly)]` as an attribute to any test that cannot be run in Playback mode (for example, if a test depends on a Dataplane SDK).
-- Create a ps1 file in the same folder that contains the actual tests ([see sample](../../src/Media/Media.Test/ScenarioTests))
+- Create a ps1 file in the same folder that contains the actual tests.
+- The name of the ps1 file should exactly match with name of the class. ([see sample](../../src/Media/Media.Test/ScenarioTests))
     - Use `Assert-AreEqual x y` to verify that values are the same
     - Use `Assert-AreNotEqual x y` to verify that values are not the same
     - Use `Assert-Throws scriptblock message` to verify an exception is being thrown
@@ -302,10 +343,7 @@ Create this environment variables for the AD scenario tests:
 ### Recording/Running Tests
 
 - Set up environment variables using New-TestCredential as described [here](../testing-docs/using-azure-test-framework.md#new-testcredential)
-- Run the test in Visual Studio in the Test Explorer window and make sure you got a generated JSON file that matches the test name in the bin folder under the `SessionRecords` folder
-- Copy this `SessionRecords` folder and place it inside the test project
-  - Inside Visual Studio, add all of the generated JSON files, making sure to change the "Copy to Output Directory" property for each one to "Copy if newer"
-  -  Make sure that all of these JSON files appear in your `<SERVICE>.Test.csproj` file
+- Run the test in Visual Studio in the Test Explorer window and make sure you got a generated JSON file that matches the test name under the `SessionRecords` folder
 
 # After Development
 

documentation/testing-docs/using-azure-test-framework.md

@@ -28,11 +28,9 @@
 - Import the `Repo-Tasks` module that helps to perform basic repository tasks
 	- Run the command `Import-Module .\Repo-Tasks.psd1`
 
-## Acquiring TestFramework
+## Azure PowerShell TestFramework
 
-The `TestFramework` library is available on NuGet at https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime.Azure.TestFramework/ .
-
-Instructions on manually downloading the library are available on NuGet, however, `TestFramework` will automatically be downloaded as part of the build process, so the manual download is usually not necessary.
+Azure PowerShell repo now has its own test framework located under `tools\TestFx`, which supports recording all the HTTP requests from behind Azure PowerShell cmdlets and then playing them back.
 
 The target framework of test is .Net Core 3.1, please ensure .Net runtime Microsoft.NETCore.App 3.1 is installed. You can list all installed version via `dotnet --info`.
 
@@ -129,8 +127,7 @@ AZURE_TEST_MODE=Record
 
 ## Record or Playback Tests
 
-- [Run the tests](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/azure-powershell-developer-guide.md#recordingrunning-tests) and make sure that you got a generated `.json` file that matches the test name in the bin folder under the `SessionRecords` folder
-- Copy the `SessionRecords` folder inside the test project and add all `*.json` files in Visual Studio setting "Copy to Output Directory" property to "Copy if newer"
+- [Run the tests](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/azure-powershell-developer-guide.md#recordingrunning-tests) and make sure that you got a generated `.json` file that matches the test name under the `SessionRecords` folder in the test project.
 - To assure that the records work fine, delete the connection string (default mode is Playback mode) OR change HttpRecorderMode within the connection string to "Playback" and run the tests
 
 ## Change Test Environment settings at run-time

src/Accounts/Accounts.Test/AutosaveTests.cs

@@ -25,6 +25,7 @@
 using Microsoft.Azure.Commands.Profile.Context;
 using Microsoft.Azure.Commands.ScenarioTest;
 using Microsoft.Azure.Commands.ResourceManager.Common;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 
 namespace Microsoft.Azure.Commands.Profile.Test
 {

src/Accounts/Accounts.Test/AzureRMProfileTests.cs

@@ -21,6 +21,7 @@
 using Microsoft.Azure.Commands.Profile.Models;
 using Microsoft.Azure.Commands.Profile.Test;
 using Microsoft.Azure.Commands.ScenarioTest;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 using Microsoft.Azure.Management.ResourceManager.Version2021_01_01.Models;
 using Microsoft.Azure.ServiceManagement.Common.Models;
 using Microsoft.Rest;
@@ -61,11 +62,11 @@ private static RMProfileClient SetupTestEnvironment(List<string> tenants, params
                 Guid.NewGuid().ToString(), DefaultTenant.ToString());
             var subscriptionList = new Queue<List<string>>(subscriptionLists);
             var clientFactory = new MockSubscriptionClientFactory(tenants, subscriptionList);
-            var mock = new MockClientFactory(new List<object>
+            var mock = new MockClientFactory(null, new List<object>
             {
                 clientFactory.GetSubscriptionClientVerLatest(),
                 clientFactory.GetSubscriptionClientVer2016()
-            }, true);
+            });
             mock.MoqClients = true;
             AzureSession.Instance.ClientFactory = mock;
             var sub = new AzureSubscription()
@@ -89,11 +90,11 @@ private static AzureRmProfile SetupLogin(List<string> tenants, params List<strin
             AzureSession.Instance.AuthenticationFactory = new AuthenticationFactory();
             var subscriptionList = new Queue<List<string>>(subscriptionLists);
             var clientFactory = new MockSubscriptionClientFactory(tenants, subscriptionList);
-            var mock = new MockClientFactory(new List<object>
+            var mock = new MockClientFactory(null, new List<object>
             {
                 clientFactory.GetSubscriptionClientVerLatest(),
                 clientFactory.GetSubscriptionClientVer2016()
-            }, true);
+            });
             mock.MoqClients = true;
             AzureSession.Instance.ClientFactory = mock;
             var sub = new AzureSubscription()

src/Accounts/Accounts.Test/AzureRMProfileTestsForMultitenant.cs

@@ -18,6 +18,7 @@
 using Microsoft.Azure.Commands.Profile.Models;
 using Microsoft.Azure.Commands.Profile.Test.Mocks;
 using Microsoft.Azure.Commands.ScenarioTest;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 using Microsoft.Azure.ServiceManagement.Common.Models;
 using Microsoft.WindowsAzure.Commands.Common.Test.Mocks;
 using Microsoft.WindowsAzure.Commands.ScenarioTest;

src/Accounts/Accounts.Test/ContextCmdletTests.cs

@@ -14,8 +14,6 @@
 
 using Microsoft.Azure.Commands.Common.Authentication;
 using Microsoft.Azure.Commands.Common.Authentication.Models;
-using Microsoft.Azure.Commands.Profile;
-using Microsoft.Azure.Commands.Profile.Models;
 // TODO: Remove IfDef
 #if NETSTANDARD
 using Microsoft.Azure.Commands.Profile.Models.Core;
@@ -29,12 +27,13 @@
 using Xunit.Abstractions;
 using Microsoft.Azure.Commands.Common.Authentication.Abstractions;
 using System;
-using Microsoft.Azure.Commands.ScenarioTest.Extensions;
 using Microsoft.Azure.Commands.Profile.Context;
 using System.Linq;
 using Microsoft.Azure.Commands.Common.Authentication.ResourceManager;
 using Microsoft.Azure.Commands.Profile.Common;
 using Microsoft.Azure.Commands.ScenarioTest.Mocks;
+using Microsoft.Azure.Commands.TestFx.Mocks;
+using Microsoft.Azure.Commands.TestFx;
 
 namespace Microsoft.Azure.Commands.Profile.Test
 {

src/Accounts/Accounts.Test/Mocks/AccountMockClientFactory.cs

@@ -13,6 +13,7 @@
 // ----------------------------------------------------------------------------------
 
 using Microsoft.Azure.Commands.Common.Authentication;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 using Microsoft.Rest;
 using Microsoft.WindowsAzure.Commands.Common.Test.Mocks;
 using System.Collections.Generic;
@@ -24,7 +25,7 @@ class AccountMockClientFactory : MockClientFactory, IClientFactory
         public delegate object NextClient();
 
         private NextClient nextClient;
-        public AccountMockClientFactory(NextClient next,  bool throwIfClientNotSpecified = true):base(new List<object>(), throwIfClientNotSpecified)
+        public AccountMockClientFactory(NextClient next, bool throwIfClientNotSpecified = true) : base(null, null)
         {
             nextClient = next;
         }

src/Accounts/Accounts.Test/ProfileCmdletTests.cs

@@ -16,6 +16,7 @@
 using Microsoft.Azure.Commands.Common.Authentication.Abstractions;
 using Microsoft.Azure.Commands.Common.Authentication.Models;
 using Microsoft.Azure.Commands.ScenarioTest;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 using Microsoft.Azure.ServiceManagement.Common.Models;
 using Microsoft.WindowsAzure.Commands.Common.Test.Mocks;
 using Microsoft.WindowsAzure.Commands.ScenarioTest;

src/Accounts/Accounts.Test/SubscriptionClientSwitchTest.cs

@@ -18,6 +18,7 @@
 using Microsoft.Azure.Commands.Profile.Models;
 using Microsoft.Azure.Commands.Profile.Test.Mocks;
 using Microsoft.Azure.Commands.ScenarioTest;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 using Microsoft.Azure.Management.ResourceManager.Version2021_01_01.Models;
 using Microsoft.Azure.ServiceManagement.Common.Models;
 using Microsoft.Rest.Azure;

src/Accounts/Accounts.Test/TenantCmdletMockTests.cs

@@ -18,6 +18,7 @@
 using Microsoft.Azure.Commands.Profile;
 using Microsoft.Azure.Commands.Profile.Models;
 using Microsoft.Azure.Commands.ScenarioTest;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 using Microsoft.Azure.ServiceManagement.Common.Models;
 using Microsoft.WindowsAzure.Commands.Common.Test.Mocks;
 using Microsoft.WindowsAzure.Commands.ScenarioTest;

src/Accounts/Authentication.Test/AuthenticationFactoryTests.cs

@@ -37,6 +37,7 @@
 using Moq;
 using System.ServiceModel.Channels;
 using Azure.Core;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 
 namespace Common.Authentication.Test
 {

src/Accounts/Authentication.Test/AuthenticatorsTest/ServicePrincipalAuthenticatorTests.cs

@@ -18,19 +18,17 @@
 using Microsoft.Azure.Commands.Common.Authentication.Abstractions;
 using Microsoft.Azure.PowerShell.Authenticators;
 using Microsoft.Azure.PowerShell.Authenticators.Factories;
-using Microsoft.Azure.Test.HttpRecorder;
 using Microsoft.WindowsAzure.Commands.Common.Test.Mocks;
 using Microsoft.WindowsAzure.Commands.ScenarioTest;
+using Microsoft.WindowsAzure.Commands.Utilities.Common;
 using Moq;
-using Xunit;
-using Xunit.Abstractions;
 using System;
 using System.Security;
 using System.Security.Cryptography.X509Certificates;
 using System.Threading;
 using System.Threading.Tasks;
-
-
+using Xunit;
+using Xunit.Abstractions;
 
 namespace Common.Authenticators.Test
 {

src/Accounts/Authentication.Test/ClientFactoryTests.cs

@@ -21,6 +21,7 @@
 using Microsoft.Azure.Commands.Common.Authentication.Abstractions;
 using Microsoft.WindowsAzure.Commands.Test.Utilities.Common;
 using System.Net.Http.Headers;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 
 namespace Common.Authentication.Test
 {

src/Accounts/Authentication.Test/PSSerializationTests.cs

@@ -11,19 +11,19 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 // ----------------------------------------------------------------------------------
+using Microsoft.Azure.Commands.Common.Authentication;
 using Microsoft.Azure.Commands.Common.Authentication.Abstractions;
 using Microsoft.Azure.Commands.Common.Authentication.Abstractions.Core;
 using Microsoft.Azure.Commands.Common.Authentication.Models;
 using Microsoft.Azure.Commands.Profile.Models;
 using Microsoft.Azure.Commands.Profile.Models.Core;
-using Microsoft.Azure.Commands.ScenarioTest.Extensions;
+using Microsoft.Azure.Commands.TestFx;
 using Microsoft.WindowsAzure.Commands.ScenarioTest;
 using System;
 using System.Globalization;
+using System.Linq;
 using System.Management.Automation;
 using Xunit;
-using System.Linq;
-using Microsoft.Azure.Commands.Common.Authentication;
 
 namespace Common.Authentication.Test
 {

src/Accounts/Authentication.Test/SessionInitializationTests.cs

@@ -26,6 +26,7 @@
 using Moq;
 using System.IO;
 using System.Text;
+using Microsoft.Azure.Commands.TestFx.Mocks;
 
 namespace Common.Authentication.Test
 {

src/Aks/Aks.Test/ScenarioTests/AksTestRunner.cs

@@ -60,7 +60,7 @@ protected AksTestRunner(ITestOutputHelper output)
                         {"Microsoft.Features", null},
                         {"Microsoft.Authorization", null}
                     }
-                ).WithMockContextAction(() =>
+                ).WithMockContextAction(mockContext =>
                 {
                     if (HttpMockServer.GetCurrentMode() == HttpRecorderMode.Playback)
                     {

src/AnalysisServices/AnalysisServices.Test/ScenarioTests/UrlDecodingRecordMatcher.cs

@@ -18,6 +18,7 @@
 using System.Linq;
 using System.Net.Http;
 using System.Text;
+using Microsoft.Azure.Commands.TestFx.Recorder;
 using Microsoft.WindowsAzure.Commands.ScenarioTest;
 
 namespace Microsoft.Azure.Commands.DataLake.Test.ScenarioTests

src/ApiManagement/ApiManagement.ServiceManagement.Test/ScenarioTests/TestsFixture.cs → src/ApiManagement/ApiManagement.ServiceManagement.Test/ScenarioTests/ApiManagementHelper.cs

@@ -10,41 +10,18 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-using Microsoft.Rest.ClientRuntime.Azure.TestFramework;
+using Microsoft.Azure.Commands.TestFx;
 using Microsoft.Azure.Management.ApiManagement;
 using Microsoft.Azure.Management.ApiManagement.Models;
-using Microsoft.Azure.Test;
-using System;
-using System.Diagnostics;
-using System.Linq;
-using System.Net;
 using Microsoft.Azure.Management.ResourceManager;
 using Microsoft.Azure.Management.ResourceManager.Models;
-using Microsoft.Azure.Test.HttpRecorder;
-using Microsoft.WindowsAzure.Commands.Test.Utilities.Common;
-using TestEnvironmentFactory = Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestEnvironmentFactory;
+using Microsoft.Rest.ClientRuntime.Azure.TestFramework;
+using System;
+using System.Linq;
 using ApiManagementClient = Microsoft.Azure.Management.ApiManagement.ApiManagementClient;
 
 namespace Microsoft.Azure.Commands.ApiManagement.ServiceManagement.Test.ScenarioTests
 {
-    public class TestsFixture : RMTestBase
-    {
-        public TestsFixture()
-        {
-            // Initialize has bug which causes null reference exception
-            HttpMockServer.FileSystemUtilsObject = new FileSystemUtils();
-            var sf = new StackTrace().GetFrame(1);
-            var callingClassType = sf.GetMethod().ReflectedType?.ToString();
-            var mockName = sf.GetMethod().Name;
-
-            using (var context = MockContext.Start(callingClassType, mockName))
-            {
-                var resourceManagementClient = ApiManagementHelper.GetResourceManagementClient(context);
-                resourceManagementClient.TryRegisterSubscriptionForResource();
-            }
-        }
-    }
-
     public static class ApiManagementHelper
     {
         public static ApiManagementClient GetApiManagementClient(MockContext context)

src/Attestation/Attestation.Test/ScenarioTests/AttestationTestRunner.cs

@@ -86,7 +86,7 @@ private static AttestationClient GetAttestationClient(MockContext context)
                 var connectionInfo = new ConnectionString(Environment.GetEnvironmentVariable("TEST_CSM_ORGID_AUTHENTICATION"));
                 string servicePrincipal = connectionInfo.GetValue<string>(ConnectionStringKeys.ServicePrincipalKey);
                 string servicePrincipalSecret = connectionInfo.GetValue<string>(ConnectionStringKeys.ServicePrincipalSecretKey);
-                string aadTenant = connectionInfo.GetValue<string>(ConnectionStringKeys.AADTenantKey);
+                string aadTenant = connectionInfo.GetValue<string>(ConnectionStringKeys.TenantIdKey);
 
                 // Create credentials
                 var clientCredentials = new ClientCredential(servicePrincipal, servicePrincipalSecret);

src/Az.Shared.props

@@ -8,9 +8,15 @@
 
   <!-- Build -->
   <PropertyGroup>
-    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
+    <GenerateAssemblyCompanyAttribute>false</GenerateAssemblyCompanyAttribute>
+    <GenerateAssemblyConfigurationAttribute>false</GenerateAssemblyConfigurationAttribute>
+    <GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
+    <GenerateAssemblyInformationalVersionAttribute>false</GenerateAssemblyInformationalVersionAttribute>
+    <GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
+    <GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
+    <GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
     <WarningsAsErrors />
   </PropertyGroup>
 
-</Project>
+</Project>