Asp.Net Core API controller memory leak

[yarong-lifemap]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

We have identified an issue with Asp.Net core API controllers, specifically related to memory allocation which does not get cleared.

Our original use case was an API controller that reads a JSON file and returns data from it. We noted that memory use was growing every time we called the API controller.

We have generated a simplified example to reproduce this issue. Here, we use the following method to generate a 50 million character string and puts it into an object that is not used at all.

[HttpGet("[action]")]
public IActionResult Test1(int testNumber)
{
    var result = new string((char)(new Random().NextInt64(1, 65535)), 50_000_000);
    var mem = Process.GetCurrentProcess().PagedMemorySize64;
    return Ok($"TEST #{testNumber} ; Mem: {mem:N0}");
}

However, the memory still seems to be allocated and retained for this object for a very long duration (GC is run but the memory doesn't seem to be released). Interestingly, at some point, this hits a peak memory allocation, which differs based on the size of the string created.

The problem is also true when deserializing JSON files (which is what we started with) and returning a subset of information from the result object. In that case, the memory allocation for the complete object doesn't seem to go away, causing a server to get overloaded.

In .Net 7, memory is never released, it seems. In .Net 6, some of the memory is released, some of the time, but we still don't understand why it's even stored in memory at all for any duration, since the object is not used for any purpose, and once the API call has completed, should be released.

You can test this issue with the attached solution. Once the project is run, monitor your Visual Studio's Diagnostic Tools window (Process Memory). The API calls also return some information about memory, but the graph represents the issue better IMHO.

Expected Behavior

Objects are released from memory after method completes execution and the API response is returned

Steps To Reproduce

TestSolution.zip

Run the TestWebAppCore project. Click on the links. Watch the memory not get released, but hit a top limit at some point per test.

Exceptions (if any)

No response

.NET Version

.Net 7.0.100 and .Net 6.0.400

Anything else?

[davidfowl]

I haven't looked, but allocating a single 50MB/100MB/500MB string per request is a recipe for bad performance, see https://learn.microsoft.com/en-us/aspnet/core/performance/memory?view=aspnetcore-6.0.

Though this is bad, it should just make the GC run way more often. It's likely that you are also running into this #45037.

[yarong-lifemap]

While GC.Collect very slowly reduces the memory, I don't believe it's a viable solution. In our original use case, which loads data from a JSON file and returns a subset of that information, it takes minutes and several GC collection cycles to eventually get rid of all the used-up memory.

What I'm proposing is this: The memory should be cleared immediately. Assuming we have 1000 users using our site, and each consumes 10MB of data via an API controller within a given minute or two, why would the expectation be that the machine is required to have 10GB of RAM (1000 x 10MB) to that it can hold those objects in memory?

[davidfowl]

This isn't how the GC works. It's never worked this way but I'm not convinced this application you've shared is replicating you're actually experiencing. If it is, then the application needs to change to not allocate giant strings that end up on the large object heap. I've linked a high level overview that should help understand why this doesn't work well. For a deeper dive on how to think about memory in a .NET application, read https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md.

It's a deep dive, but answers all of the questions that you will eventually have about memory in .NET.

[oferze]

It happens even in a small API endpoint footprint. In the demo app, if you add:

        [HttpGet("[action]")]
        public IActionResult GetMem(int testNumber)
        {
            var mem = Process.GetCurrentProcess().PagedMemorySize64;
            return Ok($"Mem: {mem:N0}");
        }

and call just this endpoint, you'll see that the memory keeps going up.

I also tried your advice on Twitter to test it not inside Visual Studio and it still happens (Azure App Service, published as "Release", target win-x64).

Seems like ASP.NET API on .NET 7 has an issue of not cleaning up after the endpoint return.

Either it's a serious bug, or we are missing something basic (hopefully the latter).

[davidfowl]

Is there a specific issue you are having where you are seeing out of memory issues? Did you look at any of the references I posted? Is the garbage collector running? I really recommend reading:

• https://learn.microsoft.com/en-us/aspnet/core/performance/memory?view=aspnetcore-6.0.
• https://learn.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak
• https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md.

I know time is short and you're trying to solve an issue but:

1. I don't see an issue based on any evidence presented so far so I'd like you to collect more data so we can get a better understanding.
2. Get a better understanding of how the GC works based on reading some of those articles.

If you want a TL;DR, the GC doesn't just collect when you think it will, it'll do so when it thinks it should.

While GC.Collect very slowly reduces the memory, I don't believe it's a viable solution.

The fact that this works tells me you don't really have a memory problem. At least, not one that I see as yet.

[ChristianPejrup]

Hi @yarong-lifemap
If i remember my .NET memory days correct (and .NET Core and Full Framework use roughly the same GC) then i think you problem is the size of the string you are allocating.

Any object bigger than 85.000 bytes will be allocated on the large object heap LOH

The garbage collection of LOH is different from the normal garbage collection in the sense that it only runs whenever .NET determines that its needed (eg. when you are running out of memory), this is because compacting memory consumes CPU cycles so if you constantly use the CPU to compact the memory then that CPU cannot also be used to serve requests (basically it's a performance optimization).

I would guess that you can reproduce this memory leak in any version of .NET and .NET Core.

I'm not going to be the judge of "if you really need this massive size json (i would however recommend that you gzip (browsers support this out of the box) the response before returning it to your consumer)".

Assuming that you need Mb size memory allocation to solve the problem you are facing I would recommend that you take a look at MemoryPool which class that can help you do memory allocations and reuse of them which will move the GC responsibility from .NET Runtime into your code (before starting to use memory pooling i can recommend this article which highlights some of the pitfalls of the technique)

[KSemenenko]

I did test

MemTest.zip

For just calls:

Total Allocated:203119
Total Memory:72812

Call GC.Collect() after requests:

Total Allocated:202525
Total Memory:-127

I also somehow subconsciously expected that there would be no alocations.

But still string is an object and it is logical that it needs memory.
And after calling GC.Collect() everything falls into place.

[oferze]

@KSemenenko thank you for the test app.

Few points I wish to address:

1. One shouldn't manually run GC.Collect(): https://learn.microsoft.com/en-us/aspnet/core/performance/memory?view=aspnetcore-7.0#call-gccollect. Yes, it's good for debugging (as we're doing right now) but the point of the issue is to show that under normal .NET 7 API usage, objects seem to not be collected automatically.

2. Even your app shows a slow increase in memory consumption. Indeed, it only returns a short string, hence many hits are needed and the memory usage increases slowly. Without GC.Collect(), it seems like the memory doesn't get cleaned. Our demo (the project @yarong-lifemap attached above) uses the 50MB+ objects just to show the scenario quicker.

3. This is also answering @davidfowl's question: We do have a real world scenario of returning ~4MB JSON content in our API, and we don't believe it's an unreasonable case. Even 50MB should be able to return with reasonably frequent collection.

We think it's a regression with .NET 7 since .NET 6 somehow does it better.

[oferze]

@davidfowl we have a fair knowledge of the GC (albeit we'll read the articles you've linked to and no doubt we'll learn more from them) and of course we know the GENs, the LOH, and that GC will collect when it needs and not when we expect.

I'm not convinced this application you've shared is replicating you're actually experiencing

We do return 4-6MB JSON objects in our real app as I mentioned, but I guess an API should return much larger strings and not leak. The demo app does reflect and is even more "pure" because our "real" app also deserializes from a GZip stream and does other things before returning that JSON. The demo app in this issue just returns a big string with nothing else, no IDisposable objects used, very clean.

Now, we'd love to hear that it's a normal behavior and that the memory will only grow based on what's available on the machine and will collect and that we shouldn't worry, but we saw that in our very early steps of migrating a big app from .NET Framework to .NET Core - those early steps are meant to validate that our skeleton is sound before we dive into lots of implementation details - and we fear that once we go to production, the memory will quickly bloat and crash the app.

[KSemenenko]

@oferze This is an interesting thing, look, you make a 50MB line, it is stored in the memory.
the method finished - done.

Now you have a 50mb string in memory.
And now you have to wait for GC to remove it. But this is an expensive operation and it will not happen immediately.
I just tried to do a version of the tests with threads, but the steel is only worse.

So on the one hand this is understandable behavior.
On the other hand, when you need to transfer a 250 MB document or 50 MB JSON, what is the right way to manage memory here?

especially if you have a MininalAPI service, with 500 RAM in the docker, which just has to transfer data from stream to http respnce stream.

[davidfowl]

By using streams. When you use the result types with objects, it doesn’t first buffer 50Mb into memory because they would be horrible for performance. Instead, it writes your 50Mb in 16K (configurable) chunks asynchronously to the response stream.

[KSemenenko]

@davidfowl so, just in Responce.Body.WriteBytes() and periodically call Flush?

[oferze]

@davidfowl to summarise, what's the right way to return 5MB Jsons in API responses without .NET increasing memory and not cleaning after itself? this continues @KSemenenko's question.