Huge memory consumption under load

[tatsuhiro-t]

I did benchmark against several HTTP/2 and SPDY servers, including node-http2.
Results are https://github.com/tatsuhiro-t/nghttp2/wiki/ServerBenchmarkRoundH210

node-http2 seems consumes huge memory, 1GB, under benchmar test. Under the same load, node-spdy uses ~300MB of memory and process the requests much faster.

I use examples/server.js and some modifications to cache "/index.html" rather then server.js. The exact source code is available in the page at the above link.

Do you think this is an issue in node-http2? Or are there any settings to change the behavior?

[molnarg]

First of all, thanks for doing this benchmark!

I didn't do any memory profiling yet, but I believe this should be a problem in node-http2 itself (and not some config issue). It should release unneded memory but I guess there are memory leaks here and there.

I'll try to track down the problem in the near future!

[snorp]

It looks like Connection never deletes anything from _streamIds, so AFAICT that would mean Stream instances are never cleaned up.

[molnarg]

Yeah you're right they are not deleted. The reason behind that is that streams still should react to frames after they becomes CLOSED, and currently this is implemented by keeping the stream alive in closed state indefinitely. This should be implemented without keeping the stream objects alive.

[tatsuhiro-t]

I think the requirements in HTTP/2 spec for closed streams make implementors headache.
nghttp2 currently ignores the frames for closed stream which has been purged from memory.

[nwgh]

Firefox also gleefully ignores frames on closed streams.

The only time that ignoring those frames could really get us into "trouble" is when re-prioritizing, but I think that's an acceptable tradeoff to make (especially given that the spec explicitly says that once a stream is closed, you can't rely on the other end having its full priority tree intact)

I think that's an acceptable tradeoff to make.