Graceful Shutdown Empty Response #1501

cphillips83 · 2017-03-17T22:09:25Z

I have attached a demo gist of the problem. I have tried shutting down Kestrel in several ways and they all produce the same result.

Test:

First run this command which will start a request that delays until the 2nd request is sent
curl http://localhost:5000/api/values/1

This request attempts to shutdown Kestrel
curl -X DELETE http://localhost:5000/api/values/1

Problem:

Kestrel keeps running until the ShutdownTimeout duration elapses, even after all requests have finished. If you set this to something like 5 minutes it will just sit there the entrie time.
Kestrel does not return a response even after return "value"; executes
Curl finally exits with curl: (52) Empty reply from server after Kestrel ShutdownTimeout elapses

References:
#1026 - Graceful shutdown and draining
#247 - Hangs for 2-3 seconds when a request is made even after its served

test code

The text was updated successfully, but these errors were encountered:

cesarblum · 2017-03-17T22:29:24Z

Investigating.

cesarblum · 2017-03-17T23:57:05Z

Your code causes a deadlock higher up in the stack.

Cancelling the token calls ApplicationLifetime.StopApplication(), on which WebHostExtensions.Run() waits while running. After that is signaled, WebHost is disposed. WebHost.Dispose() disposes DI ServiceProvider. ServiceProvider.Dispose() locks on its ResolvedServices property. ServiceProvider.Dispose() in turn calls KestrelService.Dispose(), which will wait for some time (ShutdownTimeout) until all connections are drained.

On the thread were your first request is being serviced, Get has returned a value and MVC is trying to execute the result of that action. ObjectResult.ExecuteResultAsync() calls ServiceProvider.GetRequiredService<>(), and that leads to a call to CallSiteRuntimeResolver.VisitScoped(), which will try to lock on ServiceProvider.ResolvedServices, which is already taken as a lock in ServiceProvider.Dispose() as outlined above.

Here are the call stacks:

Thread 1

 	System.Private.CoreLib.ni.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.WaitAllBlockingCore(System.Collections.Generic.List<System.Threading.Tasks.Task> tasks, int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[] tasks, int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
 	Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.Dispose()	Unknown
 	Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Dispose()	Unknown
 	Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceProvider.Dispose()	Unknown
 	Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose() Line 256	C#
 	Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(Microsoft.AspNetCore.Hosting.IWebHost host, System.Threading.CancellationToken token, string shutdownMessage) Line 96	C#
 	Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(Microsoft.AspNetCore.Hosting.IWebHost host, System.Threading.CancellationToken token) Line 60	C#
 	1501.dll!_1501.Program.Main(string[] args) Line 25	C#

Thread 2

 	Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite scopedCallSite, Microsoft.Extensions.DependencyInjection.ServiceProvider provider)	Unknown
 	Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceProvider.RealizeService.AnonymousMethod__0(Microsoft.Extensions.DependencyInjection.ServiceProvider provider)	Unknown
 	Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(System.IServiceProvider provider, System.Type serviceType)	Unknown
 	Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService<Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor>(System.IServiceProvider provider)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.ObjectResult.ExecuteResultAsync(Microsoft.AspNetCore.Mvc.ActionContext context)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeResultAsync(Microsoft.AspNetCore.Mvc.IActionResult result)	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeResultAsync>d__15>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeResultAsync>d__15 stateMachine)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeResultAsync(Microsoft.AspNetCore.Mvc.IActionResult result)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.State next, ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Scope scope, ref object state, ref bool isCompleted)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextResultFilterAsync()	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResultFilterAsync>d__18>(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResultFilterAsync>d__18 stateMachine)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextResultFilterAsync()	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.State next, ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Scope scope, ref object state, ref bool isCompleted)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__20>(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__20 stateMachine)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Next(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.State next, ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Scope scope, ref object state, ref bool isCompleted)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeNextResourceFilter()	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__18>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__18 stateMachine)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeNextResourceFilter()	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Next(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.State next, ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Scope scope, ref object state, ref bool isCompleted)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeFilterPipelineAsync()	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__13>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__13 stateMachine)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeFilterPipelineAsync()	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeAsync()	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeAsync>d__11>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeAsync>d__11 stateMachine)	Unknown
 	Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeAsync()	Unknown
 	Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext)	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4>(ref Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4 stateMachine)	Unknown
 	Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext)	Unknown
 	Microsoft.AspNetCore.StaticFiles.dll!Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)	Unknown
 	Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6>(ref Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6 stateMachine)	Unknown
 	Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)	Unknown
 	Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext) Line 53	C#
 	Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context) Line 86	C#
 	Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.RequestProcessingAsync()	Unknown
 	System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.<RequestProcessingAsync>d__2>(ref Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.<RequestProcessingAsync>d__2 stateMachine)	Unknown
 	Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.RequestProcessingAsync()	Unknown
 	Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame.Start()	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.QueueUserWorkItemCallbackDefaultContext.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()	Unknown
 	System.Private.CoreLib.ni.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()	Unknown

I'm not sure what you're trying to do is a supported scenario. Paging @pakrym @davidfowl @NTaylorMullen @rynowak

cphillips83 · 2017-03-18T00:47:02Z

This was an over simplification of a problem that I was trying to resolve but was able to reproduce with this example. I wanted to make sure it wasn't something deeper in our code base so I created an empty dotnet webapi project to run my tests in.

Our scenario is that we spin up new micro services to replace old ones as new versions become available (continuous delivery). We use Consul for service discovery and when the new ones come online they replace the old serviceID of the previous micro service. We want the old version of the microservice to drain and finish all its requests and exit.

The problem we kept running in to is when we requested kestrel to gracefully shutdown, all the active connections would fail and stall out rather than complete as intended.

NOTE: I tried requesting Kestrel to shutdown via an external thread rather than on a request and get the same result. This leads me to believe that the deadlock exists in all scenario's if you try to gracefully shutdown Kestrel.

Is there another way that I missed to ask Kestrel to drain all its connections properly and exit?

cphillips83 · 2017-03-18T00:53:49Z

To be perfectly clear. I want to be able to signal Kestrel to exit and have it handle any active connections it has left before returning back to the command prompt. Even if those active connections take upwards of 60-120 seconds as a worst case scenario.

cphillips83 · 2017-03-20T14:27:47Z

@CesarBS I've been looking through this all weekend and read your diagnoses probably 10 times. While it makes sense what you are saying, I still fail to see how to resolve this. You've mentioned this is possibly an unsupported scenario, but the scenario is nothing more than wanting to stop Kestrel and fulfilling the remaining requests without taking new ones.

What I'm not understanding is why are things being disposed and locked when you request Kestrel to be stopped. Shouldn't Kestrel simply stop the listening socket and enter a drain mode until all connections are completed (or shutdown timeout lapses) before anything is even disposed?

At a higher level I imagine something like this (pseudo code).


public void Stop(){
	_running = false;
}

public void Run(){
	if(_running) return;

	_listenSocket.Start();

	_running = true;
	while(_running){
		ProcessActiveRequests();
		//sleep(0) or something?
	}
	_listenSocket.Stop();

	//request to stop
	var time = currentTime;
	while(time + _shutdownTime > currentTime && _activeRequests.Count > 0){
		ProcessActiveRequests();
		//sleep(0) or something?
	}

	if(_activeRequests.Count > 0){
		//log that we aborted {_activeRequests.Count} requests
	}


	//Dispose resources

}

cphillips83 · 2017-03-20T16:12:10Z

Since this is critical for me, I gave up on trying to determine how to handle this properly inside Kestrel and implemented my own version of drain. I'm putting this here in case anyone else needs a work around for the time being.

In program.cs, create and use a cancellation token source.

        public static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();

Pass the token to Run

        host.Run(CancellationTokenSource.Token);

While this code doesn't belong here, I'm putting it here for illustration purposes. In Startup.cs, create some static variables to track state.

        private static int _activeConnections;
        private static bool _stopRequest = false;

Add a method for Stop.

        public static void Stop()
        {
            _stopRequest = true;
            while (_activeConnections > 0)
            {
                System.Threading.Thread.Sleep(1000);
                Console.WriteLine($"Connections: {_activeConnections}");
            }

            Console.WriteLine("Stopping");
            Program.CancellationTokenSource.Cancel(false);
        }

And finally in Configure, add a custom Use to track connections and to return 500 when the system is shutting down for all new connection attempts.

I believe these work as a chain (hence next), so this should be first before all other UseXYZ.

            app.Use(async (http, next) =>
            {
                if (_stopRequest)
                {
                    var response = http.Response;
                    response.StatusCode = 500;
                    return;
                }

                Interlocked.Increment(ref _activeConnections);
                try
                {
                    await next();
                }
                finally
                {
                    Interlocked.Decrement(ref _activeConnections);
                }
            });

And finally, you can call Startup.Stop() anywhere you like and it will drain all active connections, return 500 to any new connections and then cleanly exit instantly without any dead locks.

avtc · 2017-03-21T11:59:30Z

It works even without adding CancellationTokenSource in Program.
You just need to subscribe to applicationLifetime.ApplicationStopping in Configure.

public void Configure(IApplicationBuilder app, 
            IHostingEnvironment env, 
            ILoggerFactory loggerFactory, 
            IApplicationLifetime applicationLifetime
        )
...
applicationLifetime.ApplicationStopping.Register(Stop);
...

Also, in case of using some sort of load balancer, like docker swarm, it could be handy to redirect to the same url, so request will be processed by other running nodes:

                if (_stopRequest)
                {
                    var response = context.Response;
                    response.StatusCode = 308;
                    response.Headers.Add("Location", $"{context.Request.Scheme}://{context.Request.Host}{context.Request.Path}{context.Request.QueryString}");
                    return;
                }

avtc · 2017-03-24T11:28:14Z

Created middleware based on a proposed workaround.
https://github.com/avtc/GracefullShutdown

davidfowl · 2017-03-24T17:23:02Z

I think this will solve the problem:

aspnet/Hosting#947

Lets track the issue over there

cesarblum · 2017-03-24T18:05:51Z

@cphillips83 Kestrel supports graceful shutdown, but by default it gives active connections a short period of time to complete their work (5 seconds). You can increase the shutdown timeout by setting KestrelServerOptions.ShutdownTimeout to a value suitable to your requirements.

algirdask · 2017-09-08T12:11:52Z

I cannot find KestrelServerOptions.ShutdownTimeout in 2.0 release. Was it removed/moved somewhere else? API documentation in web is still present

Tratcher · 2017-09-08T12:53:16Z

It was moved to IWebHostBuilder: https://github.com/aspnet/Hosting/blob/03bdb40f8a07427bbd83c89154c02da2ec92aea9/src/Microsoft.AspNetCore.Hosting.Abstractions/HostingAbstractionsWebHostBuilderExtensions.cs#L168

FrederickBrier · 2018-02-27T20:30:54Z

I downloaded and was reading the WebHostBuilderTests in Hosting to try and understand if there was a similar mechanism that avtc created with his GracefulShutdown extension. I need to save state once the Kestrel web server is shutdown and has no active connections. It seems that the Hosting is a generic IServer lifecycle management, and not doing active connection counting. Is there a similar hook or singleton in the DI that I can reference to determine if Kestrel is quiescent? Thank you.

muratg added this to the Discussions milestone Mar 20, 2017

davidfowl closed this as completed Mar 24, 2017

davidfowl mentioned this issue Mar 24, 2017

Graceful Server Shutdown aspnet/Hosting#947

Closed

davidfowl mentioned this issue Apr 12, 2017

WebHost Start/Run/StopAsync and IServer.Start/StopAsync aspnet/Hosting#995

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Graceful Shutdown Empty Response #1501

Graceful Shutdown Empty Response #1501

cphillips83 commented Mar 17, 2017

cesarblum commented Mar 17, 2017

Uh oh!

cesarblum commented Mar 17, 2017 •

edited

Loading

Uh oh!

cphillips83 commented Mar 18, 2017

Uh oh!

cphillips83 commented Mar 18, 2017

Uh oh!

cphillips83 commented Mar 20, 2017

Uh oh!

cphillips83 commented Mar 20, 2017

Uh oh!

avtc commented Mar 21, 2017

Uh oh!

avtc commented Mar 24, 2017

Uh oh!

davidfowl commented Mar 24, 2017

Uh oh!

cesarblum commented Mar 24, 2017

Uh oh!

algirdask commented Sep 8, 2017

Uh oh!

Tratcher commented Sep 8, 2017

Uh oh!

FrederickBrier commented Feb 27, 2018

Uh oh!

Graceful Shutdown Empty Response #1501

Graceful Shutdown Empty Response #1501

Comments

cphillips83 commented Mar 17, 2017

cesarblum commented Mar 17, 2017

Uh oh!

cesarblum commented Mar 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cphillips83 commented Mar 18, 2017

Uh oh!

cphillips83 commented Mar 18, 2017

Uh oh!

cphillips83 commented Mar 20, 2017

Uh oh!

cphillips83 commented Mar 20, 2017

Uh oh!

avtc commented Mar 21, 2017

Uh oh!

avtc commented Mar 24, 2017

Uh oh!

davidfowl commented Mar 24, 2017

Uh oh!

cesarblum commented Mar 24, 2017

Uh oh!

algirdask commented Sep 8, 2017

Uh oh!

Tratcher commented Sep 8, 2017

Uh oh!

FrederickBrier commented Feb 27, 2018

Uh oh!

cesarblum commented Mar 17, 2017 •

edited

Loading