Add UseKestrel() API to the WebApplicationFactory

[mkArtakMSFT]

Background and Motivation

We have a long-standing popular issue tracking improvement of automated browser testing with real server.

To simplify this process, customers were asking for decoupling the WebApplicationFactory from TestServer, as they're currently tightly coupled. That would allow one to implement alternative implementation and use that in WebApplicaitonFactory.

While that was an option to pursue an alternative (perhaps a more straight forward) approach has been proposed during a design discussion with @captainsafia , @javiercn and @halter73. That is, instead of focusing on achieving the goal utilizing the current constructs, come up with a more straight forward way to enable utilizing Kestrel server in WebApplicationFactory. This proposal was great, because that's what many customers wanted to accomplish anyway and that's why they wanted to have some way to not utilize the TestServer implementation.

Another issue is that today exposing the initialization logic publicly. Without this change, customers had to call the CreateClient() API which would internally initialize the server but wouldn't use the returned HttpClient instance. This will make it more intuitive and avoid creating unnecessary objects. Here is a screenshot from a blogpost pointing out this odd usage pattern:

Another example of not-so-great solution that the community came up with can be found here:

Here, the developer calls the CreateDefaultClient() in a derived class to force initialization.

With the proposed changes, customers won't need to worry about this, as the real server initialization is still going to be handled by WebApplicationFactory, so there will be no need to interfere in this process. However, there may still be a need to initialize the server without explicitly getting a client. This is useful in browser-based testing scenarios with Selenium, where the framework will be interacting with the browser and the browser will handle the communication with the server internally.

Proposed API

PR: #60635

namespace Microsoft.AspNetCore.Mvc.Testing;

public class WebApplicationFactory<TEntryPoint>
{
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.StartServer() -> void
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.UseKestrel() -> void
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.UseKestrel(int port) -> void
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.UseKestrel(System.Action<Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions!>! configureKestrelOptions) -> void

Usage Examples

UseKestrel API(s)

Most of the time, developers use some helper class that encapsulates all the configuration of the WebApplicationFactory. To configure that type to utilize Kestrel server in tests, developers can call the UseKestrel() API from the constructor, as shown below.

public class KestrelBasedWapFactory : WebApplicationFactory<SimpleWebSite.Startup>
{
    public KestrelBasedWapFactory()
    {
        this.UseKestrel();
    }
}

Alternatively, consumers could call the UseKestrel() on the WebApplicationFactory instance directly, if they're not utilizing and intermediate helper. That way, the UseKestrel will need to be called during the Test initialization / setup phase.

The UseKestrel overloads can be called one after another. For example, the developer may want to specify both a custom port for the Kestrel to use as well as customize the server when it starts. To achieve that, two different overloads should be called:

int dynamicPortNumber = 0;
this.UseKestrel(dynamicPortNumber);
this.UseKestrel(options=>{ /* Customize the options for kestrel here */ });

Note: the UseKestrel() call is not reversible and must be called before the WebApplicationFactory is initialized, which is done through the StartServer() call.

StartServer API

This API allows starting the underlying server of the WebApplicationFactory without the need of creating a client. This comes handy is scenarios related to browser-based testing, where developers will be interacting with the server only through the browser instance provided via test-framework. Below is an excerpt from a test which utilizes Selenium:

public BrowserBasedTests(WebApplicationFactory<MyApp.Program> factory){
    _factory = factory;
    _factory.StartServer();
}

[Fact]
public void BrowserBasedTest(){
 Browser.Navigate().GoToUrl("[url that the server listens to]");

var selector = By.TagName("h1");
Assert.Equal("Page title is here", Browser.FindElement(selector).Text);
}

Alternative Designs

Here is the alternative that was considered first: #60247

Risks

Right now, there are no mechanisms in place to enable much customization of the kestrel creation process, other than interacting with the DI before the server creation. This may become something that customers need, but I hold off from exposing the CreateKestrelServer to the customers to gauge the need for it first. We have time so if needed, that can be done later. Mitigated by the UseKestrel (Action) API

[halter73]

API Review Notes:

• Should Initialize() be made async?
  • No
• Should Initialize() be renamed to InitializeServer()?
  • Or Start()? Or Run()? StartServer()?
  • StartServer() it is.
• Should UseKestrel() take a port argument? e.g. UseKestrel(port: 5005)
  • This could be done with the IServerAddressesFeature.
  • We could add an overload later.
• Can we keep the Server property nullable and then throw if it's accessed after calling UseKestrel()?
  • Yes.

API Approved!

namespace Microsoft.AspNetCore.Mvc.Testing;

public class WebApplicationFactory<TEntryPoint>
{
+      public void StartServer();
+      public void UseKestrel();
}

[Tratcher]

Can you show an example of configuring Kestrel?

How about WebListener?

[halter73]

@Tratcher You can see the PR here: #60635

It's currently Kestrel-only.

[mkArtakMSFT]

I'm going to revoke the approval for this one, as I plan to follow @halter73's and @Tratcher's suggestion to make the Kestrel server more configurable when using the UseKestrel API. Will update the proposal with the details soon.

[mkArtakMSFT]

Updated Proposal

(Also updated the description)

Below are the APIs to be used:

+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.StartServer() -> void
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.UseKestrel() -> void
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.UseKestrel(int port) -> void
+ Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory<TEntryPoint>.UseKestrel(System.Action<Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions!>! configureKestrelOptions) -> void

[dotnet-policy-service]

Thank you for submitting this for API review. This will be reviewed by @dotnet/aspnet-api-review at the next meeting of the ASP.NET Core API Review group. Please ensure you take a look at the API review process documentation and ensure that:

• The PR contains changes to the reference-assembly that describe the API change. Or, you have included a snippet of reference-assembly-style code that illustrates the API change.
• The PR describes the impact to users, both positive (useful new APIs) and negative (breaking changes).
• Someone is assigned to "champion" this change in the meeting, and they understand the impact and design of the change.

[halter73]

API Review Notes:

• We had already approved the UseKestrel() and StartServer() methods, and this just adds two UseKestrel() overloads:
  • UseKestrel(int port)
  • UseKestrel(Action<KestrelServerOptions> configureKestrelOptions)
• The overload signature and parameter names seem reasonable
• We could see a scenario where someone wanted to configure both the port and provide an options callback but think that’s niche.
  • Things should still work if I just called both methods

  • waf.UseKestrel(0);
    waf.UseKestrel(options =>
    {
       options.AllowSynchronousIO = true;
    });

• Should UseKestrel return this (i.e. WebApplicationFactory<TEntryPoint>) to be more consistent with other Use... methods?
  • No. WAF is not really a builder type, and it's unclear what you'd chain off it.

Updated API Approved!

namespace Microsoft.AspNetCore.Mvc.Testing;

public class WebApplicationFactory<TEntryPoint>
{
+    public void StartServer();
+    public void UseKestrel();
+    public void UseKestrel(int port);
+    public void UseKestrel(Action<KestrelServerOptions> configureKestrelOptions);
}