-
Notifications
You must be signed in to change notification settings - Fork 10.3k
ASP.NET Core - Performance - Load Testing issue Microsoft.AspNetCore.Server.Kestrel[13] Connection id "xyz", Request id "abc" An unhandled exception was thrown by the application. #19476
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
We don't see this in our load testing (with many more concurrent requests). Can you provide a runnable application and steps for reproducing this? |
@anurse it was a simple API calling a stored procedure and returning response. |
@sanasz91mdev we can't take action because we can't reproduce it. Can you provide the application and the error you saw? |
@davidfowl .. provide your email so that i can add you to access to my code? Check question for error:
|
I saw this exception in logs. In my case no load was applied. @sanasz91mdev Lets try to replace stored procedure with await Task.Delay(100) and do public simple hello-world instead of complex app. |
@davidfowl can you have a look? thank you :) os: FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine Load test settings: 1000 concurrent users for 1 minute This exception will occur for external network access, and there will be no pressure problem for local access :)
|
The only answer I can think of is: Internet bandwidth is full, and local access does not need bandwidth |
Sorry that doesn’t help. You need to provide at a bare minimum a repro of what the application is doing and how you’re running the load test |
application is just calling a stored procedure and Load testing is being done by JMeter over same network @davidfowl |
I am getting this too. Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App] I had wondered if this error was related to the size of the request. It is not. The crash here the content-length is 1694976 I have had successful requests more than 3x that size Here is some data captured by my UseExceptionHandler(), in case it helps someone
|
@rclarkezaner Can you reproduce this after a single request? Can you reproduce it reliably? Does it happen with form posts only? (I see the error is caused by a post request here). |
@davidfowl Unfortunately I cannot reproduce it. I only see it happen on form posts. It's an online application form where customers enter biodata and upload a selfie. Sometimes it will happen several times in a row to the same customer making me think it's caused by something in the request body. |
@rclarkezaner can you share the body? That's the relevant part here. |
@Tratcher I would like to share the body but cannot figure out how to access it in my error handler. I have an open SO question on this This error occurs pretty frequently, so once I have the capture part solved it won't take long to get some samples. |
This stack trace implies there's an issue in Kestrel rather than the form reader.
The aspnetcore/src/Servers/Kestrel/Core/src/Internal/Http/Http1ContentLengthMessageBody.cs Lines 91 to 95 in e3d3da3
|
Before This usually works, but an InvalidOperationException is thrown if the last time the PipeReader was advanced, more data was examined than consumed. Since Http1ContentLengthMessageBody only passes the consumed SequencePosition to AdvanceTo, the DefaultPipeReader complains "The examined position cannot be less than the previously examined position." I think to fix this we'll have to keep track of the last examined SequencePosition passed to |
I moved the application off of Apache/Ubuntu and on to IIS/Windows, and I have not seen the "The examined position cannot be.." error since. |
@mohitvaid that's a different error. A 408 is a timeout waiting for the request body. |
Will post in relevant issue. |
I have a smilar issue. I can't seem to recreate it in my lower environments, even when creating more load. This happens after a signal-r event tells a 10 or fewer clients (Angular SPA) to refresh their page because data has changed. Nobody is complaining about things not working. I looked at the proxy server logs, and I discovered that the errors are always coming from the same two IP addresses, where it is likely the browser is being left open on the page and possibly the computer is asleep. If i have them restart the browsers, the issue goes away, and then it gradually comes back. I'm not sure what more I can do to troubleshoot, but happy to try anything you guys have in mind.
|
That's because the client aborted mid-request, so it's not expecting a response. Kestrel has to throw from ReadAsync in this situation. Otherwise, it would be indistinguishable from a full request body. If the complaint is about the error log, that's just because middleware never caught the exception. It would be up to the BodyModelBinder to catch the exception and log something at a lower level. That's not this issue though. If the log level matters to you, I'd file a new issue @JoshuaNitschke. |
@halter73 I see, thanks for clearly explaining it to me. I wasn't sure what to make of the error and this was the only issue I could find that seemed related, and the issue seemed a bit nebulous so I thought I'd submit. I'm don't know why the client is aborting mid request, but I will focus my energy on troubleshooting the SPA. Sorry if I wasted your time! |
I have an application in which a source is writing data continuously into the writer side of a pipe and the reader side is reading and processing the data, and I am getting a similar error to the one being discussed. Basically what is being done is to read a chunk of data from the reader and then process it. I am always getting error at AdvanceTo(buffer.Start) but not exactly at the same point. Sometimes it occurs after 100 KB of data is read, and sometimes after 200 KB is read. If I slow down the reader by artificially introducing a Timer.Delay for a second in my code, the error is not occurring at all. If when creating the pipe I set in the pipeoptions - pauseWriterThreshold: long.MaxValue, then also the error is not occurring. Is it a race condition? |
Does it have anything to do with reading the request body in an ASP.NET Core application or just using pipelines? If it's the latter then it's unrelated to this issue. PS: If you're using pipelines though you should read this guide https://docs.microsoft.com/en-us/dotnet/standard/io/pipelines |
I have been using Pipelines for sometime now and am familiar with the link. Thanks anyway. I am pretty impressed with Pipelines and it is very useful in my line of work where I am developing DICOM protocols for an imaging application and also protocols to interface Clinical Analyzers. No, I am not reading a request body in ASP.NET Core. I am running my application as a .net core 3.1 console application. It is part of a larger application and what I have described is the bare essentials of how and where the issue is occurring and my observations on how it is mitigated. I came to this page when I was searching for the error message 'The examined position cannot be less than the previously examined position.' which was being thrown and saw similar exceptions in the discussion. Please is there anywhere else where I should discuss this? |
|
Thank you. Will try to make a repo application and discuss it there. |
Created dotnet/runtime#37176. Thanks once again |
I came across this issue too, it is a core 2.2 .net framework 4.7.2 app which running on IIS. Application startup exception: System.Configuration.ConfigurationErrorsException: An error occurred loading a configuration file: The process cannot access the file 'xxxxxxxx.exe.Config' because it is being used by another process. (xxxxxx.exe.Config) ---> System.IO.IOException: The process cannot access the file 'xxxxxxxx.exe.Config' because it is being used by another process. Microsoft.AspNetCore.Server.Kestrel[13] |
For those, who experience this issue, try to do |
Thanks for contacting us. |
Closed via #25799 |
Uh oh!
There was an error while loading. Please reload this page.
Hello,
testing an ASP.NET Core API under load gives following error:
Load test settings: 4000 concurrent requests coming each second for a duration of 1 minute
I am using JMeter for load testing. This issue does not arise for 2000 concurrent users.
The text was updated successfully, but these errors were encountered: