Skip to content

ASP.NET Core - Performance - Load Testing issue Microsoft.AspNetCore.Server.Kestrel[13] Connection id "xyz", Request id "abc" An unhandled exception was thrown by the application. #19476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sanasz91mdev opened this issue Mar 1, 2020 · 33 comments
Assignees
Labels
api-needs-work API needs work before it is approved, it is NOT ready for implementation area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions bug This issue describes a behavior which is not expected - a bug. feature-kestrel
Milestone

Comments

@sanasz91mdev
Copy link

sanasz91mdev commented Mar 1, 2020

Hello,

testing an ASP.NET Core API under load gives following error:

**fail:** _Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLTTTNCMN23D", Request id "0HLTTTNCMN23D:00000002": An unhandled exception was thrown by the application.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content._

Load test settings: 4000 concurrent requests coming each second for a duration of 1 minute

I am using JMeter for load testing. This issue does not arise for 2000 concurrent users.

@analogrelay
Copy link
Contributor

We don't see this in our load testing (with many more concurrent requests). Can you provide a runnable application and steps for reproducing this?

@analogrelay analogrelay added this to the Discussions milestone Mar 2, 2020
@sanasz91mdev
Copy link
Author

sanasz91mdev commented Mar 9, 2020

@anurse it was a simple API calling a stored procedure and returning response.
Do you need code for this?

@davidfowl
Copy link
Member

@sanasz91mdev we can't take action because we can't reproduce it. Can you provide the application and the error you saw?

@sanasz91mdev
Copy link
Author

sanasz91mdev commented Mar 9, 2020

@davidfowl .. provide your email so that i can add you to access to my code?

Check question for error:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HLTTTNCMN23D", Request id "0HLTTTNCMN23D:00000002": An unhandled exception was thrown by the application.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.

@Roman-Blinkov
Copy link

I saw this exception in logs. In my case no load was applied.

@sanasz91mdev Lets try to replace stored procedure with await Task.Delay(100) and do public simple hello-world instead of complex app.

@xlievo
Copy link

xlievo commented Mar 15, 2020

@davidfowl can you have a look? thank you :)
same question!
simple hello-world app

os: FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine

Load test settings: 1000 concurrent users for 1 minute

This exception will occur for external network access, and there will be no pressure problem for local access :)

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLU8P7S8VQMH", Request id "0HLU8P7S8VQMH:00000001": An unhandled exception was thrown by the application.
System.InvalidOperationException: The examined position cannot be less than the previously examined position.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_InvalidExaminedPosition()
   at System.IO.Pipelines.Pipe.AdvanceReader(BufferSegment consumedSegment, Int32 consumedIndex, BufferSegment examinedSegment, Int32 examinedIndex)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.<ReadAsyncInternal>g__ResetReadingState|9_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FormPipeReader.ReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at BusinessController.Call() in /root/workspace/log/Common/Common.cs:line 613
   at lambda_method(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

@xlievo
Copy link

xlievo commented Mar 15, 2020

The only answer I can think of is: Internet bandwidth is full, and local access does not need bandwidth
Will this exception be thrown when the server bandwidth is full?
But I'm not sure!

@davidfowl
Copy link
Member

Sorry that doesn’t help. You need to provide at a bare minimum a repro of what the application is doing and how you’re running the load test

@sanasz91mdev
Copy link
Author

application is just calling a stored procedure and Load testing is being done by JMeter over same network @davidfowl

@rclarkezaner
Copy link

rclarkezaner commented Apr 13, 2020

I am getting this too.

Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Ubuntu 18.04.2
Apache/2.4.29

I had wondered if this error was related to the size of the request.

It is not. The crash here the content-length is 1694976

I have had successful requests more than 3x that size

Here is some data captured by my UseExceptionHandler(), in case it helps someone

The examined position cannot be less than the previously examined position.

   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_InvalidExaminedPosition()
   at System.IO.Pipelines.Pipe.AdvanceReader(BufferSegment consumedSegment, Int32 consumedIndex, BufferSegment examinedSegment, Int32 examinedIndex)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.<ReadAsyncInternal>g__ResetReadingState|9_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FormPipeReader.ReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList``1 factories)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.TryCreateAsync(ActionContext actionContext, IList``1 factories)
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageBinderFactory.<>c__DisplayClass2_0.<<CreatePropertyBinder>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.BindArgumentsCoreAsync()
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)`
Method: POST
Url: http://app.example.com/
getRequest: The examined position cannot be less than the previously examined position. getting Form
Request.Headers:
Cache-Control: max-age=0
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Cookie: _ga=GA1.2.105213241.1586789971; _gid=GA1.2.1505485800.1586789971; .AspNetCore.Antiforgery.o4DuEIf4Zck=CfDJ8GTEBDREdqBAl7oEJcwj9RDOYrAXKfGtBrSVximKJXggJziZDCe2AAyvoPeSdKCkUZAkO4Mm7LYluYEOECmWqxYartqysCxjE0wfl_yHCYZBLE03eTm2PHH0OEae0QSVhgIa0G3uB8LvEuNqfJZRcJk
Host: example.com
Referer: https://app.example.com/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Upgrade-Insecure-Requests: 1
Origin: https://app.example.com
Content-Length: 1694976
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: example.com
X-Forwarded-Server: example.com

@davidfowl
Copy link
Member

@rclarkezaner Can you reproduce this after a single request? Can you reproduce it reliably? Does it happen with form posts only? (I see the error is caused by a post request here).

@rclarkezaner
Copy link

@davidfowl Unfortunately I cannot reproduce it. I only see it happen on form posts. It's an online application form where customers enter biodata and upload a selfie. Sometimes it will happen several times in a row to the same customer making me think it's caused by something in the request body.

@davidfowl davidfowl removed this from the Discussions milestone Apr 13, 2020
@davidfowl
Copy link
Member

It's definitely a bug. Sounds like we need to stress form parsing some more to flesh it out (that component was re-written in 3.x)

cc @jkotalik @Tratcher

@Tratcher
Copy link
Member

@rclarkezaner can you share the body? That's the relevant part here.

@rclarkezaner
Copy link

@Tratcher I would like to share the body but cannot figure out how to access it in my error handler.

I have an open SO question on this

This error occurs pretty frequently, so once I have the capture part solved it won't take long to get some samples.

@Tratcher
Copy link
Member

This stack trace implies there's an issue in Kestrel rather than the form reader.

   at System.IO.Pipelines.Pipe.DefaultPipeReader.AdvanceTo(SequencePosition consumed)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.<ReadAsyncInternal>g__ResetReadingState|9_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)

The BadHttpRequestException: Unexpected end of request content from the original post points to possibly the same place, though the BadHttpRequestException wouldn't have been thrown if you hit the other exception given above.

if (_readResult.IsCompleted)
{
ResetReadingState();
ThrowUnexpectedEndOfRequestContent();
}

ResetReadingState is called for timeouts, early end of bodies, and cancellation tokens. It doesn't look like any cancellation tokens are used by default in this call stack. Early end of body matches the other callstack. If you are able to reproduce this with Debug or Trace level logs it might help narrow it down.

@analogrelay analogrelay added this to the Next sprint planning milestone Apr 13, 2020
@halter73
Copy link
Member

Before Http1ContentLengthMessageBody.ReadAsync() throws, it calls AdvanceTo(readResult.Buffer.Start) on the underlying connection PipeReader (a DefaultPipeReader) to put it back in a usable state.

This usually works, but an InvalidOperationException is thrown if the last time the PipeReader was advanced, more data was examined than consumed. Since Http1ContentLengthMessageBody only passes the consumed SequencePosition to AdvanceTo, the DefaultPipeReader complains "The examined position cannot be less than the previously examined position."

I think to fix this we'll have to keep track of the last examined SequencePosition passed to Http1ContentLengthMessageBody.AdvanceTo(consumed, examined), and replay it if it wasn't the same as consumed when resetting the reading state.

@rclarkezaner
Copy link

I moved the application off of Apache/Ubuntu and on to IIS/Windows, and I have not seen the "The examined position cannot be.." error since.

@Tratcher
Copy link
Member

@mohitvaid that's a different error. A 408 is a timeout waiting for the request body.

@mohitvaid
Copy link

@mohitvaid that's a different error. A 408 is a timeout waiting for the request body.

Will post in relevant issue.

@BrennanConroy BrennanConroy added the bug This issue describes a behavior which is not expected - a bug. label May 11, 2020
@JoshuaNitschke
Copy link

JoshuaNitschke commented May 15, 2020

I have a smilar issue. I can't seem to recreate it in my lower environments, even when creating more load.

This happens after a signal-r event tells a 10 or fewer clients (Angular SPA) to refresh their page because data has changed. Nobody is complaining about things not working. I looked at the proxy server logs, and I discovered that the errors are always coming from the same two IP addresses, where it is likely the browser is being left open on the page and possibly the computer is asleep. If i have them restart the browsers, the issue goes away, and then it gradually comes back.

I'm not sure what more I can do to troubleshoot, but happy to try anything you guys have in mind.

[13:17:38 ERR] An unhandled exception has occurred while executing the request.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
[13:17:38 ERR] An unhandled exception has occurred while executing the request.
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
[13:17:38 ERR] Unexpected end of request content.
[13:17:38 INF] Request finished in 20286.2345ms 500 application/json

@halter73
Copy link
Member

Nobody is complaining about things not working.

That's because the client aborted mid-request, so it's not expecting a response. Kestrel has to throw from ReadAsync in this situation. Otherwise, it would be indistinguishable from a full request body.

If the complaint is about the error log, that's just because middleware never caught the exception. It would be up to the BodyModelBinder to catch the exception and log something at a lower level. That's not this issue though. If the log level matters to you, I'd file a new issue @JoshuaNitschke.

@JoshuaNitschke
Copy link

@halter73 I see, thanks for clearly explaining it to me. I wasn't sure what to make of the error and this was the only issue I could find that seemed related, and the issue seemed a bit nebulous so I thought I'd submit. I'm don't know why the client is aborting mid request, but I will focus my energy on troubleshooting the SPA. Sorry if I wasted your time!

@Mathew-Kochekkan
Copy link

I have an application in which a source is writing data continuously into the writer side of a pipe and the reader side is reading and processing the data, and I am getting a similar error to the one being discussed.

Basically what is being done is to read a chunk of data from the reader and then process it.
After processing, depending on the information in the chunk, a certain length of data has to be cleared from the pipe. So I advance the PipeReader by calling AdvanceTo(buffer.Start) and then clearing the required length by reading from the pipe and then repeat the process.

I am always getting error at AdvanceTo(buffer.Start) but not exactly at the same point. Sometimes it occurs after 100 KB of data is read, and sometimes after 200 KB is read.

If I slow down the reader by artificially introducing a Timer.Delay for a second in my code, the error is not occurring at all.

If when creating the pipe I set in the pipeoptions - pauseWriterThreshold: long.MaxValue, then also the error is not occurring.

Is it a race condition?

@davidfowl
Copy link
Member

davidfowl commented May 29, 2020

@Mathew-Kochekkan

I have an application in which a source is writing data continuously into the writer side of a pipe and the reader side is reading and processing the data, and I am getting a similar error to the one being discussed.

Does it have anything to do with reading the request body in an ASP.NET Core application or just using pipelines? If it's the latter then it's unrelated to this issue.

PS: If you're using pipelines though you should read this guide https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines

@Mathew-Kochekkan
Copy link

@Mathew-Kochekkan

I have an application in which a source is writing data continuously into the writer side of a pipe and the reader side is reading and processing the data, and I am getting a similar error to the one being discussed.

Does it have anything to do with reading the request body in an ASP.NET Core application or just using pipelines? If it's the latter then it's unrelated to this issue.

PS: If you're using pipelines though you should read this guide https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines

I have been using Pipelines for sometime now and am familiar with the link. Thanks anyway. I am pretty impressed with Pipelines and it is very useful in my line of work where I am developing DICOM protocols for an imaging application and also protocols to interface Clinical Analyzers.

No, I am not reading a request body in ASP.NET Core.

I am running my application as a .net core 3.1 console application.

It is part of a larger application and what I have described is the bare essentials of how and where the issue is occurring and my observations on how it is mitigated.

I came to this page when I was searching for the error message 'The examined position cannot be less than the previously examined position.' which was being thrown and saw similar exceptions in the discussion.

Please is there anywhere else where I should discuss this?

@davidfowl
Copy link
Member

Please is there anywhere else where I should discuss this?

donet/runtime for general pipelines issues.

@Mathew-Kochekkan
Copy link

Please is there anywhere else where I should discuss this?

donet/runtime for general pipelines issues.

Thank you. Will try to make a repo application and discuss it there.

@Mathew-Kochekkan
Copy link

Mathew-Kochekkan commented May 29, 2020

Please is there anywhere else where I should discuss this?

donet/runtime for general pipelines issues.

Created dotnet/runtime#37176. Thanks once again

@benzhi2011
Copy link

benzhi2011 commented Jul 1, 2020

I came across this issue too, it is a core 2.2 .net framework 4.7.2 app which running on IIS.
The app has been working for few years, but it doesn't run stable recently. Every 4, 5 days it running into trouble and I have to restart the app pool to make it work again. I enable the stdout log on the IIS server and can see the same error, but before this error others have got, I can see the config load error. There is only one app running on the IIS, how can is lock itself? What can I do to solve this problem? It is a critical production app.

Application startup exception: System.Configuration.ConfigurationErrorsException: An error occurred loading a configuration file: The process cannot access the file 'xxxxxxxx.exe.Config' because it is being used by another process. (xxxxxx.exe.Config) ---> System.IO.IOException: The process cannot access the file 'xxxxxxxx.exe.Config' because it is being used by another process.

Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HM0TB8BTMOQM", Request id "0HM0TB8BTMOQM:00000002": An unhandled exception was thrown by the application.

@sk-pub
Copy link

sk-pub commented Jul 7, 2020

For those, who experience this issue, try to do
ThreadPool.SetMinThreads(threads, threads)
With threads value from 10 to 300 and above.

@ghost
Copy link

ghost commented Jul 24, 2020

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

@halter73 halter73 added the api-needs-work API needs work before it is approved, it is NOT ready for implementation label Sep 10, 2020
@BrennanConroy
Copy link
Member

Closed via #25799

@ghost ghost locked as resolved and limited conversation to collaborators Oct 11, 2020
@amcasey amcasey added area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions and removed area-runtime labels Jun 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
api-needs-work API needs work before it is approved, it is NOT ready for implementation area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions bug This issue describes a behavior which is not expected - a bug. feature-kestrel
Projects
None yet
Development

No branches or pull requests