Add Counters & Logs for 404s and MapFallback

[Tratcher]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

If a request reaches the end of the pipeline (e.g. isn't handled anywhere else) there is a default middleware that sends an empty 404 response.

aspnetcore/src/Http/Http/src/Builder/ApplicationBuilder.cs

Lines 138 to 141 in 0d52a44

	if (!context.Response.HasStarted)
	{
	context.Response.StatusCode = StatusCodes.Status404NotFound;
	}

Similarly MapFallback catches any request that didn't map to another route.

aspnetcore/src/Http/Routing/src/Builder/EndpointRouteBuilderExtensions.cs

Lines 388 to 391 in 0d52a44

	public static RouteHandlerBuilder MapFallback(this IEndpointRouteBuilder endpoints, Delegate handler)
	{
	return endpoints.MapFallback("{*path:nonfile}", handler);
	}

Problem: There's no visibility in the system for tracking how many requests are missing routes and why. This information would be needed to identify bad links, misconfigured routes, and monitor the customer experience.

Describe the solution you'd like

• Add a meter/counter that tracks how many requests reach the 404 middleware.
• Add a meter/counter that tracks how many requests reach the fallback route. This may already be covered by Add Meter/Counters to track hits for each route/endpoint #46361
• Add an Info or Debug level log in each of these places that includes the following:
  • Scheme, host, port, path base, path
  • Method
  • Protocol
• Use structured logging to allow low level filtering and avoid unnecessary allocations
• These should be under granular log prefixes for targeted enablement.
  • Microsoft.AspNetCore.Routing.EndOfPipeline.UnexpectedRequest(scheme...
  • Microsoft.AspNetCore.Routing.Fallback.HandledRequest(scheme...

Additional context

The counters would be used to monitor the volume of unexpected requests, and then the logs could be enabled to determine the specific request types.

Having sampling support in the loggers would also help deal with high volume output.

[davidfowl]

Use structured logging to allow low level filtering and avoid unnecessary allocations

What is this? How are these logs going to be consumed?

Thanks for contacting us.

We're moving this issue to the .NET 8 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[Tratcher]

Triage feedback (@halter73): The counter can go directly into hosting, not the app builder, so we know what status code was actually sent to the client. Similarly, we can already use hosting's ResponseFinished log for this.

info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:5000/favicon.ico - 404 0 - 1.7527ms

Hmm, using ResponseFinished doesn't account for the requirement about not logging paths if they're considered sensitive.

Since it's hosting, would you do a counter for all responses with a status code dimension? There's only a few common ones (200, 301, 400, 401, 404, 500, etc.).

Having sampling support in the loggers would also help deal with high volume output.

Submit this to the runtime/logging owners.

[Tratcher]

I think we'd still want an info level log in the places indicated above, that covers our pattern of logging when we've taken decisive action on a request (e.g. produced a response, refused the request, etc.). The hosting level logs have a few drawbacks:

• They cover every request, not just these unmatched ones. That's a lot more noise to sort through.
• They don't address the question of avoiding logging paths with sensitive data. If we're only logging unmatched requests they're less likely to contain sensitive data.

[Tratcher]

The customer reviewed this and said that the host metric is not sufficient for fallback routes. They also requested tweaks for missed routes.

• Fallback routes are not called out as fallbacks, you have to know which route pattern is which. These should still have their own metric for easy discovery and tracking.
• Do not add another dimension identifying these as fallback or missed, too many dimensions.
• Do not add a data within the dimension value ('fallback: {rout pattern}'), the tools don't want to parse the field to understand if it's a fallback route.
• Missed routes should set the route description metadata to something like '-missed-' to be easily identified.

Note, since the logs are debug and info respectively, and in specific categories, dotnet/runtime#82465 is not needed at this time.