Improve automated browser testing with real server

[danroth27]

Feedback from @shanselman:

We should do better with testing. There's issues with moving from the inside out:

LEVELS OF TESTING

• GOOD - Unit Testing - Make a PageModel and call On Get
• GOOD - Functional Testing - Make a WebApplicationFactory and make in-memory HTTP calls
• BAD - Automated Browser Testing with Real Server - can't easily use Selenium or call a real server. I shouldn't have to do this. We should decouple the WebApplicationFactory from the concrete TestServer implementation. @davidfowl

public class RealServerFactory<TStartup> : WebApplicationFactory<Startup> where TStartup : class
{
    IWebHost _host;
    public string RootUri { get; set; }
    public RealServerFactory()
    {
        ClientOptions.BaseAddress = new Uri("https://localhost");
    }

    protected override void ConfigureWebHost(IWebHostBuilder builder)
    {
        builder.UseEnvironment("Development"); //will be default in RC1
    }

    protected override TestServer CreateServer(IWebHostBuilder builder)
    {
        //Real TCP port
        _host = builder.Build();
        _host.Start();
        RootUri = _host.ServerFeatures.Get<IServerAddressesFeature>().Addresses.FirstOrDefault();

        //Fake Server we won't use...sad!
        return new TestServer(new WebHostBuilder().UseStartup<TStartup>());
    }

    protected override void Dispose(bool disposing) 
    {
         base.Dispose(disposing);
         if (disposing) 
        {
                _host.Dispose();
            }
        }
}

/cc @javiercn

[mkArtakMSFT]

As this is not a priority for 2.2 for now, moving to the backlog.

[davidfowl]

@mkArtakMSFT just to clarify the concrete change we should make in 2.2:

We should make it so that it's possible to boot up the WebApplicationFactory (or another derived type) without a test server. It makes functional testing of your application absolutely trivial and the changes required to do this should be small.

/cc @javiercn

[steveoh]

I'm not sure I follow. How would these changes make it easier to use something like puppeteer and chrome headless/selenium for automated browser testing. What is a real server?

[shanselman]

@steveoh the current set up allows for very convenient Unit Testing by spinning up the App/WebHost and talking to it 'in memory." So no HTTP, no security issue, you're basically talking HTTP without actually putting bytes on the wire (or localhost). Super useful.

But if you want to use Automated Browser Testing and have it driven by a Unit Test Framework, you are hampered by the concrete TestServer class and have to do a little dance (above) to fake it out and start up the WebHost yourself. I'd propose no regular person could/would figure that out.

David is saying that if WebApplicationFactory could be started without the fake TestServer we could easily and cleanly do Unit Testing AND Browser Automation Testing with the piece we have here.

[rgamage]

This is why we love Scott! Always the advocate for the mainstream developers out there wanting to use MS tools, but stymied by various obscure limitations. Thank you sir!

[giggio]

The provided workaround has problems.

The problems start with the fact that now we have 2 hosts. One from the TestServer, and another built to work with http. And WebApplicationFactory.Server references the first one, which we are not testing against. And to make things worse, calls to methods that configure the builder, such as WebApplicationFactory.WithWebHostBuilder will not work with the dummy TestServer.

Because of these problems we cannot easily interact with the real host, the one being tested. It is very common that I change some backend service and configure it before a test is run. Suppose I need to access some service that is not callable during development, only production. I replace that service when I configure the services collection with a fake, and then configure it to respond the way I want it to respond. I can't do that through WebApplicationFactory.Server.Host.Services.

The resulting code I have works, but it is ugly as hell, it is an ugly ugly hack.

I hope we can move this forward and do not require TestServer, maybe an IServer. I thought about forking the whole TestServer and WebApplicationFactory infrastructure, but as this is somewhat planning I'll wait. I hope it gets fixed soon. I am just commenting to complement that the provided workaround is not enough and to really work around you have to avoid WebApplicationFactory.Server and create a terrible work around it.

[bchavez]

One way I solved this is to stop using WebApplicationFactory<T> all together.

Refactored Program.Main() to:

public static Task<int> Main(string[] args)
{
   return RunServer(args);
}
public static async Task<int> RunServer(string[] args,
                                        CancellationToken cancellationToken = default)
{
    ...
    CreateWebHostBuilder()
          .Build()
          .RunAsync(cancellationToken)
}

So, my unit test fixtures new up a var cts = new CancellationTokenSource(), then pass the cancellation token by calling Program.RunServer(new string[0], cts.Token). The server starts up as normal without having to create a separate process.

Make real HTTP calls like normal. When your done and your unit test completes, call cts.Cancel() to clean up and shutdown the HTTP server.

One down side is you need to copy appsettings.json to an output directory from your test project; potentially managing two appsettings.json files (one in your server project, and one in your test project). Maybe a linked project file could help eliminate the issue.

YMMV.

🤔 ✨ "Do you ponder the manner of things... yeah yeah... like glitter and gold..."

[giggio]

Hello everyone, this issue is still unresolved and seems to keep being postponed. Just so we know the planning, are you considering resolving this for ASP.NET Core 3.0? If not, do you have a workaround that does not incur on the problems I mentioned earlier (#4892 (comment))?

[BennyM]

This would be very welcome. The workaround mentioned here is great, except when you have html files and what not that you would also need to copy over.

[Tratcher]

WebApplicationFactory is designed for a very specific in-memory scenario. Having it start Kestrel instead and wire up HttpClient to match is a fairly different feature set. We actually do have components that do this in Microsoft.AspNetCore.Server.IntegrationTesting but we've never cleaned them up for broader usage. It might make more sense to leave WebApplicationFactory for in-memory and improve the IntegrationTesting components for this other scenario.

[giggio]

I'm fine with that, as long as we have a good end to end testing story.

[M-Yankov]

Thanks guys for this discussion and specially to @bchavez
I succeeded to run Selenium tests in Azure pipelines with the idea to start a local server on the agent, run tests and stop the server.

My test class:

public class SelenuimSampleTests : IDisposable
{
    private const string TestLocalHostUrl = "http://localhost:8080";

    private readonly CancellationTokenSource tokenSource;

    public SelenuimSampleTests()
    {
        this.tokenSource = new CancellationTokenSource();

        string projectName = typeof(Web.Startup).Assembly.GetName().Name;

        string currentDirectory = Directory.GetCurrentDirectory();
        string webProjectDirectory = Path.GetFullPath(Path.Combine(currentDirectory, $@"..\..\..\..\{projectName}"));

        IWebHost webHost = WebHost.CreateDefaultBuilder(new string[0])
            .UseSetting(WebHostDefaults.ApplicationKey, projectName)
            .UseContentRoot(webProjectDirectory) // This will make appsettings.json to work.
            .ConfigureServices(services =>
            {
                services.AddSingleton(typeof(IStartup), serviceProvider =>
                {
                    IHostingEnvironment hostingEnvironment = serviceProvider.GetRequiredService<IHostingEnvironment>();
                    StartupMethods startupMethods = StartupLoader.LoadMethods(
                        serviceProvider, 
                        typeof(TestStartup),
                        hostingEnvironment.EnvironmentName);

                    return new ConventionBasedStartup(startupMethods);
                });
            })
            .UseEnvironment(EnvironmentName.Development)
            .UseUrls(TestLocalHostUrl)
            //// .UseStartup<TestStartUp>() // It's not working
            .Build();

        webHost.RunAsync(this.tokenSource.Token);
    }

    public void Dispose()
    {
        this.tokenSource.Cancel();
    }

    [Fact]
    public void TestWithSelenium()
    {
        string assemblyLocation = System.Reflection.Assembly.GetExecutingAssembly().Location;
        string currentDirectory = Path.GetDirectoryName(assemblyLocation);

        using (ChromeDriver driver = new ChromeDriver(currentDirectory))
        {
            driver.Navigate().GoToUrl(TestLocalHostUrl);
            IWebElement webElement = driver.FindElementByCssSelector("a.navbar-brand");

            string expected = typeof(Web.Startup).Assembly.GetName().Name;

            Assert.Equal(expected, webElement.Text);

            string appSettingValue = driver.FindElementById("myvalue").Text;
            const string ExpectedAppSettingsValue = "44";
            Assert.Equal(ExpectedAppSettingsValue, appSettingValue);
        }
    }
}

Very similar to the @bchavez 's example, I'm starting the CreateDefaultBuilder. That approach gives me a little bit more flexibility to set custom settings. For example using a TestStartup. the .UseStartup<TestStartUp>() wasn't working, so I used logic from WebHostBuilderExtensions.UseStartUp to set WebHostDefaults.ApplicationKey and configure services.

I'm sure there is better approach somewhere, but after а few days of research, nothing helped me. So I share this solution in case someone need it.

The test method is just for an example, there are different approaches to initialize browser driver.

[arkiaconsulting]

@M-Yankov I had your code work by using IWebHostBuilder.UseStartup<>, and IWebHost.StartAsync as well as IWebHost.StopAsync, without CancellationTokenSource.
=> .NetCore 2.2

[M-Yankov]

The idea of the IWebHostBuilder.UseStartup<> is to simplify the code. But for me it was not enough: after applying the UseStartup<TestStartup>(), ChromeWEbDriver cannot open the home page.
I didn't mention that the TestStartup class inherits the real startup class with overriding two custom methods (if it matters).
That's why I've used the .ConfigureServices(services => ... approach.
About the IWebHost.StartAsync & IWebHost.StopAsync - it seems that they are working. Thanks.

[Sebazzz]

This is a solution that worked for me in the mean time. It ensures that everything on WebApplicationFactory, like the Services property keep working as expected:

    protected override void ConfigureWebHost(IWebHostBuilder builder) {
        if (builder == null) throw new ArgumentNullException(nameof(builder));

        IPEndPoint endPoint;
        // Assign available TCP port
        using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)){
            socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
            socket.Listen(1);
            endPoint = (IPEndPoint)socket.LocalEndPoint;
        }

        return builder
            .ConfigureKestrel(k => k.Listen(new IPEndPoint(IPAddress.Loopback, 0)));
    }

    protected override TestServer CreateServer(IWebHostBuilder builder)
    {
        // See: https://github.com/aspnet/AspNetCore/issues/4892
        this._webHost = builder.Build();

        var testServer = new TestServer(new PassthroughWebHostBuilder(this._webHost));
        var address = testServer.Host.ServerFeatures.Get<IServerAddressesFeature>();
        testServer.BaseAddress = new Uri(address.Addresses.First());

        return testServer;
    }

    private sealed class PassthroughWebHostBuilder : IWebHostBuilder
    {
        private readonly IWebHost _webHost;

        public PassthroughWebHostBuilder(IWebHost webHost)
        {
            this._webHost = webHost;
        }

        public IWebHost Build() => this._webHost;

        public IWebHostBuilder ConfigureAppConfiguration(Action<WebHostBuilderContext, IConfigurationBuilder> configureDelegate){
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(this.ConfigureAppConfiguration)}");
            return this;
        }

        public IWebHostBuilder ConfigureServices(Action<WebHostBuilderContext, IServiceCollection> configureServices) {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(ConfigureServices)}");
            return this;
        }

        public IWebHostBuilder ConfigureServices(Action<IServiceCollection> configureServices)
        {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(ConfigureServices)}");
            return this;
        }

        public string GetSetting(string key) => throw new NotImplementedException();

        public IWebHostBuilder UseSetting(string key, string value)
        {
            TestContext.WriteLine($"Ignoring call: {typeof(PassthroughWebHostBuilder)}.{nameof(this.UseSetting)}({key}, {value})");
            return this;
        }
    }

[JeroMiya]

Just a note: with .net core 3.0 apps using IHostBuilder, @danroth27's workaround no longer works.

Now I just have a test script that runs the server, then runs the tests, waits for them to finish, then kills the server. Have to start up the server manually when running in the Visual Studio test runner. Not a great experience.

[Sebazzz]

Have you looked at what I posted above? Met vriendelijke groet, Sebastiaan Dammann

…

________________________________ Van: Jeremy <[email protected]> Verzonden: Friday, October 25, 2019 6:23:35 PM Aan: aspnet/AspNetCore <[email protected]> CC: Sebastiaan Dammann <[email protected]>; Comment <[email protected]> Onderwerp: Re: [aspnet/AspNetCore] Improve automated browser testing with real server (#4892) Just a note: with .net core 3.0 apps using IHostBuilder, @danroth27<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdanroth27&data=02%7C01%7C%7C431d232f4a1e48a4fdae08d75967af9e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637076174165457064&sdata=nnjWPdeQbclhT8tqRJ0dTvT0%2B9%2BIVVy59xlR6YroZ9A%3D&reserved=0>'s workaround no longer works. Now I just have a test script that runs the server, then runs the tests, waits for them to finish, then kills the server. Have to start up the server manually when running in the Visual Studio test runner. Not a great experience. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faspnet%2FAspNetCore%2Fissues%2F4892%3Femail_source%3Dnotifications%26email_token%3DAAK4FMKG45JYM75DPWDPXCDQQMMQPA5CNFSM4GKKY3G2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECI3HWQ%23issuecomment-546419674&data=02%7C01%7C%7C431d232f4a1e48a4fdae08d75967af9e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637076174165457064&sdata=RBes58Dw7x44ts%2F0lcG4UoyiWSCrOtAruhhqNIdqmX0%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAK4FMIWUOFTNNP5HZ54KNDQQMMQPANCNFSM4GKKY3GQ&data=02%7C01%7C%7C431d232f4a1e48a4fdae08d75967af9e%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637076174165467069&sdata=7%2BkYk6vmLkKcfCmQF23laO3E1b2%2FsE2bhc3t3DNufSI%3D&reserved=0>.

[JeroMiya]

I'm not using IWebHostBuilder so CreateServer is never called.

[Sebazzz]

Yes but you might be able to use a similar approach 👍 Met vriendelijke groet, Sebastiaan Dammann

…

________________________________ Van: Jeremy <[email protected]> Verzonden: Friday, October 25, 2019 6:30:51 PM Aan: aspnet/AspNetCore <[email protected]> CC: Sebastiaan Dammann <[email protected]>; Comment <[email protected]> Onderwerp: Re: [aspnet/AspNetCore] Improve automated browser testing with real server (#4892) I'm not using IWebHostBuilder so CreateServer is never called. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faspnet%2FAspNetCore%2Fissues%2F4892%3Femail_source%3Dnotifications%26email_token%3DAAK4FMJDHYG5EVV7DEXXI2TQQMNLXA5CNFSM4GKKY3G2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECI35HQ%23issuecomment-546422430&data=02%7C01%7C%7C5c08ad3d4c9142a4bf7208d75968b3c0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637076178530399604&sdata=ssIMCSG9VHxrAhaDKP49o3cXXy79hOkd1k4rH8ZneJM%3D&reserved=0>, or unsubscribe<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAK4FMJJAN4VBVYAYJVKAGLQQMNLXANCNFSM4GKKY3GQ&data=02%7C01%7C%7C5c08ad3d4c9142a4bf7208d75968b3c0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637076178530409614&sdata=r7Id092T3ADSKoHnnTKJSzvBsmgis1Vu698UMYEWOn8%3D&reserved=0>.

[JeroMiya]

There's an override you can define create a custom IHostBuilder and another to create a custom IHost with a builder passed in, and that's likely where the solution would be. However, it didn't work when I tried it. I don't know what part of the internals of WebApplicationFactory adds the in-memory restrictions but it might to be outside of those two overloads. I've already spent too much time on it, and running the app directly without WebApplicationFactory seems to work fine for now.

[ffMathy]

What is the status on this? @davidfowl did this ever make it to 2.2? In that case, how do we use it?

[Tratcher]

@ffMathy not much progress has been made on this scenario, it's still in the backlog.

[ffMathy]

Alright. Is there an ETA? Rough estimate?

[Tratcher]

@ffMathy this is an uncommitted feature, there's no ETA until we decide to move it from the backlog to a milestone.

[lukos]

@Sebazzz Your code above does not compile.

• You are returning ConfigureKestrel, even though the method is void.
• You are assigning a variable _webHost in CreateServer that is not in the listing. Is this just a class variable or is it supposed to be coming from somewhere else?
• You setup an endpoint in ConfigureWebHost and then you don't use it, you just create another one in the call to ConfigureKestrel.

[martincostello]

Yes it is possible, a potential approach is shown here #4892 (comment)

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[IEvangelist]

It's really sad that this is going on 5 years old now, and it's still not a priority. Automated testing of ASP.NET Core apps all up should be easy. The "testing" story for .NET app in general feels a bit like an after thought. With every new version of .NET, we have an opportunity to improve this, and I would certainly vote for it to become a priority.

I hope that we will start shipping first-class NuGet testing packages, that we ourselves use - so that our customers can use the same packages.

[JeroMiya]

Just an update on my side, I've long since moved to container based E2E testing and this works perfectly with .net core. I just run the API, frontend, and database containers (i.e. sql server for linux) using the .net wrapper over the docker api. Works great and the app runs in an environment that more closely simulates the production environment.

[ffMathy]

@JeroMiya that's not the problem though.

The problem is that that way of testing is not really exposed via actual network connections. So if you want (for instance) puppeteer, playwright, selenium etc, it can't reach the server.

Or if you're opening a reverse tunnel with ngrok to test that your webhooks are working properly in a test. That won't work either.

[BennyM]

I've been following this topic for quite some time. For the past year or two I've been running selenium tests against an angular app, you need to override some functionality of the WebApplicationFactory as illustrated in the sample code but it is possible.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[giggio]

This is again removed from the backlog. That's sad.
We need a good story for e2e testing, this is something that is easily doable in every other platform. ASPNET Core is almost 10 years old, now.
Please reconsider putting this back on the .NET 8 milestone. It is not so much work and it would help a lot of people.

[giggio]

Ok, I've managed to get the code to work with .NET 6. I'll post most of it here, but I'm thinking about making it a NuGet pkg, to make it available to everyone. I'm running my tests and they are passing, as soon as I get it more refined I'll do that.

I am having to use reflection on only a small location (Microsoft.Extensions.Hosting.HostFactoryResolver.ResolveHostFactory()), and with that I'm using the same infrastructure as the TestServer from Microsoft. It opens ports, and services work. But remember that this is taking a dependency into an internal method, so new .NET versions could break it, so be careful when you upgrade, even minor versions.

I have not tested it with Startup.cs, only with top level statements. This is rough code, it probably does not work in all cases, but as some asked here for help, I'm posting what I have, feel free to use, adapt, etc.

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;

public class TestServer<TTypeAssemblyEntrypoint> : IAsyncDisposable where TTypeAssemblyEntrypoint : class
{
    private readonly string connectionString;

    public IHost Host { get; set; }

    public TestServer(string connectionString) => this.connectionString = connectionString;

    public async Task StartAsync()
    {
        Host = CreateWebHost(connectionString);
        var lifetime = Host.Services.GetRequiredService<IHostLifetime>();
        await lifetime.WaitForStartAsync(CancellationToken.None);
    }

    private Func<string[], object> ResolveHostFactory(Assembly assembly,
        TimeSpan? waitTimeout = null,
        bool stopApplication = true,
        Action<object> configureHostBuilder = null,
        Action<Exception> entrypointCompleted = null)
    {
        var hostFactoryResolver = typeof(Microsoft.AspNetCore.TestHost.TestServer).Assembly.GetType("Microsoft.Extensions.Hosting.HostFactoryResolver", true);
        var method = hostFactoryResolver.GetMethod("ResolveHostFactory", BindingFlags.Public | BindingFlags.Static)
            ?? throw new InvalidOperationException("Could not find method 'ResolveHostFactory' on type 'HostFactoryResolver'.");
        var hostFactory = (Func<string[], object>)method.Invoke(null, new object[] { assembly, waitTimeout, stopApplication, configureHostBuilder, entrypointCompleted });
        return hostFactory;
    }

    private IHost ResolveHost(Assembly assembly,
        IConfiguration configuration = null,
        Action<IHostBuilder> configureHostBuilder = null)
    {
        configuration ??= new ConfigurationBuilder().Build();
        var factory = ResolveHostFactory(assembly, null, false, (object o) => { configureHostBuilder((IHostBuilder)o); }, null);
        var configurationManager = new ConfigurationManager();
        configurationManager.AddConfiguration(configuration);
        var args = new List<string>();
        foreach (var (key, value) in configurationManager.AsEnumerable())
            args.Add($"--{key}={value}");
        var host = (IHost)factory(args.ToArray());
        return host;
    }

    private IHost CreateWebHost(string connectionString)
    {
        var applicationPath = typeof(TTypeAssemblyEntrypoint).Assembly.Location;
        var staticWebAssetsFile = Path.ChangeExtension(applicationPath, ".staticwebassets.runtime.json");
        if (!File.Exists(staticWebAssetsFile))
            throw new FileNotFoundException($"Static web assets file not found: {staticWebAssetsFile}");
        var webrootPath = GetWebRootPath();
        var wwwrootPath = Path.Combine(webrootPath, "wwwroot");
        if (!Directory.Exists(wwwrootPath))
            throw new DirectoryNotFoundException($"Directory does not exist at '{wwwrootPath}'.");
        var hostConfiguration = new ConfigurationBuilder()
            .AddInMemoryCollection(new Dictionary<string, string>
            {
                //[WebHostDefaults.StaticWebAssetsKey] = staticWebAssetsFile,
                [HostDefaults.ApplicationKey] = typeof(TTypeAssemblyEntrypoint).Assembly.GetName()?.Name ?? string.Empty,
                [HostDefaults.EnvironmentKey] = "Test",
                [WebHostDefaults.ContentRootKey] = webrootPath,
                [WebHostDefaults.WebRootKey] = wwwrootPath,
                [WebHostDefaults.ServerUrlsKey] = AmbienteTestesFuncionais.PublicUrl,
            })
            .Build();

        var host = ResolveHost(typeof(TTypeAssemblyEntrypoint).Assembly,
            hostConfiguration,
            configureHostBuilder: (IHostBuilder hostBuilder) =>
            {
                hostBuilder
                .ConfigureAppConfiguration(configurationBuilder =>
                {
                    var configuration = new ConfigurationBuilder()
                        .AddInMemoryCollection(new Dictionary<string, string>
                        {
                            ["ConnectionStrings:X"] = connectionString,
                        })
                        .Build();
                    configurationBuilder.AddConfiguration(configuration);
                })
                .ConfigureServices(services =>
                {
                    // add mocks
                })
                .ConfigureWebHost(webHostBuilder =>
                {
                    webHostBuilder.UseKestrel();
                });
            });
        return host;
    }

    private static string GetWebRootPath()
    {
        var currentExecutingAssemblyFile = new Uri(Assembly.GetExecutingAssembly().Location).LocalPath;
        var dir = Path.GetDirectoryName(currentExecutingAssemblyFile);
        var testCsproj = $"{Path.GetFileNameWithoutExtension(currentExecutingAssemblyFile)}.csproj";
        var testCsprojFullPath = "";
        while (!File.Exists(testCsprojFullPath))
        {
            testCsprojFullPath = Path.Combine(dir, testCsproj);
            var root = Path.GetPathRoot(testCsprojFullPath);
            if (root == dir)
                throw new Exception($"File {testCsproj} does not exist.");
            dir = Path.GetFullPath(Path.Combine(dir, ".."));
        }
        var testProjectDir = Path.GetDirectoryName(testCsprojFullPath);
        var assetsDoc = JsonDocument.Parse(File.ReadAllText(Path.Combine(testProjectDir, "obj", "project.assets.json")));
        var path = assetsDoc.RootElement.GetProperty("libraries").GetProperty("Web/1.0.0").GetProperty("path").GetString();
        var siteWebCsrojFile = Path.GetFullPath(Path.Combine(testProjectDir, path));
        var diretorioSiteWeb = Path.GetFullPath(Path.GetDirectoryName(siteWebCsrojFile));
        if (!Directory.Exists(diretorioSiteWeb))
            throw new Exception($"Directory does not exist at '{diretorioSiteWeb}'.");
        return diretorioSiteWeb;
    }

    public async ValueTask DisposeAsync()
    {
        if (Host == null)
            return;
        await Host.StopAsync();
        Host.Dispose();
    }
}

[mkArtakMSFT]

I've been looking into this and have found that there are actually 3 issues at play here:

1. Breaking up the dependency between WebApplicationFactory and TestServer
2. Utilizing WebApplicationFactory for testing against a real server
3. MvcTestingAppManifest.json manifest file using absolute paths which makes the tests not runnable on other machines (because paths' mismatch)

The first and the last issues here are things I'm going to try to address.
As for the second issue, after some discussions with team members, it looks like WebApplicationFactory isn't the mechanism designed for E2E testing. For that very reason we haven't used it in Blazor for E2E testing. I'm learning more about this now and will try to come up with a sample to share here, for how to do that.

[egil]

For anybody looking and this and unable to wait till net 10, here is the best hacky version I know for doing this today, that doesn’t cause the target app to start twice during a test run: https://github.com/egil/BlazorTestingAZ/blob/main/BlazorTestingAZ.Tests/BlazeWright/BlazorApplicationFactory.cs

[danroth27]

Here's the API proposal for this: #60758. Feedback is welcome!