OutOfMemoryError - Memory leak with example project (reproduce scenario)

[kzimnicki-n26]

Description

The memory leak is a real thing in spring-cloud-gateway. In production for our machines heap memory is growing:

Before I had a problem with Direct memory but then changed netty flag (noUnsafe) to true:
System.setProperty("io.netty.noUnsafe", "true");
at least I can monitor a Heap Memory.

At the beginning I thought this is a netty problem, had so many issues with https, but it looks like somehow gateway is not deallocation the memory. It doesn't matter if this is just simple rest post with json body (growing slower) or file upload (growing faster). I even created some small code to upload file to endpoint using (io.projectreactor.ipc:reactor-netty:jar:0.7.2.RELEASE) but then everything was correct.

It looks like the code is copying data from the request to proxy request and it not releasing it properly:

	return proxyRequest.sendHeaders() //I shouldn't need this
					.send(request.getBody()
							.map(DataBuffer::asByteBuffer)
							.map(Unpooled::wrappedBuffer));

Let me know if I you have any questions.

How to reproduce:

I've created small project with extracted "NettyRoutingFilter" logic.
https://github.com/kzimnicki/netty-leak/blob/master/src/main/java/reproduce/controller/TestController.java

It's just uploading the 1mb file to the controller (FileController)

The project is here:
https://github.com/kzimnicki/netty-leak

I've tried to keep it simple:

How to run:
mvn clean install exec:java -Dexec.mainClass="reproduce.Application"

You can run it from IDEA then it will be easier to monitor it from jvisualvm.

Memory:

I'm using jvisualvm to monitor the memory:

After some point of in time it just start throwing OutOfMemoryException.

Memory Leak information (netty):

#1:
	io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:518)
	io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:88)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	java.lang.Thread.run(Thread.java:748)
#2:
	io.netty.buffer.AdvancedLeakAwareByteBuf.readRetainedSlice(AdvancedLeakAwareByteBuf.java:106)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305)
	io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:101)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	java.lang.Thread.run(Thread.java:748)
#3:
	Hint: 'reactor.left.httpCodec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	java.lang.Thread.run(Thread.java:748)
#4:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	java.lang.Thread.run(Thread.java:748)
Created at:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
	io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	java.lang.Thread.run(Thread.java:748)

Exception:

io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:283) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:463) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:183) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1041) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at reactor.ipc.netty.http.HttpOperations.lambda$sendHeaders$0(HttpOperations.java:130) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:106) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.core.publisher.Mono.subscribe(Mono.java:3008) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
	at reactor.core.publisher.Mono.subscribe(Mono.java:3008) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
	at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:356) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:383) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:479) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_144]
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[na:1.8.0_144]
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[na:1.8.0_144]
	at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_144]
	at java.lang.StackTraceElement.toString(StackTraceElement.java:172) ~[na:1.8.0_144]
	at io.netty.util.ResourceLeakDetector$Record.toString(ResourceLeakDetector.java:602) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.ResourceLeakDetector$DefaultResourceLeak.toString(ResourceLeakDetector.java:489) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.ResourceLeakDetector.reportLeak(ResourceLeakDetector.java:295) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.ResourceLeakDetector.track0(ResourceLeakDetector.java:262) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:245) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.buffer.AbstractByteBufAllocator.toLeakAwareBuffer(AbstractByteBufAllocator.java:48) ~[netty-buffer-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:314) ~[netty-buffer-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:166) ~[netty-buffer-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:157) ~[netty-buffer-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115) ~[netty-buffer-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:92) ~[netty-codec-http-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:283) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:463) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:183) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1041) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]

[spencergibb]

Two things. 1) We don't recommend running milestones in production. 2) I know there has been quite a bit of work in reactor-netty, can you try with snapshots rather than the milestone?

[spencergibb]

@smaldini any thoughts?

[kzimnicki-n26]

Hi @spencergibb

Thank you for response.

With the updated version it's even worse. Leak still exists and some SocketTimeoutException popup (never happened before). I've reduced number of endpoint hits from 2k to 1k to finish test.

Logs from test:

https://gist.github.com/kzimnicki-n26/f6091213ae912ee39403b72ab421a81d

Dependency

 <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-gateway</artifactId>
            <version>2.0.0.BUILD-SNAPSHOT</version>
        </dependency>

Dependency Tree from example project:

[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ test ---
[INFO] Deleting /Users/krzysztofzimicki/dev/netty-leak/target
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ test ---
[INFO] test:test:jar:1.0-SNAPSHOT
[INFO] +- org.springframework.cloud:spring-cloud-starter-gateway:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |  +- org.springframework.cloud:spring-cloud-gateway-core:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  |  +- org.springframework.boot:spring-boot:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  |  |  \- org.springframework:spring-context:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |  |  |  |     +- org.springframework:spring-aop:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |  |  |  |     \- org.springframework:spring-expression:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |  |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |  |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  |  +- javax.annotation:javax.annotation-api:jar:1.3.1:compile
[INFO] |  |  |  +- org.springframework:spring-core:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |  |  |  |  \- org.springframework:spring-jcl:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |  |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  |  \- org.springframework:spring-tuple:jar:1.0.0.RELEASE:compile
[INFO] |  |     \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
[INFO] |  |        +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |        \- org.objenesis:objenesis:jar:2.1:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-webflux:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |     +- org.springframework.boot:spring-boot-starter-json:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |     |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.2:compile
[INFO] |     |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.2:compile
[INFO] |     |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.2:compile
[INFO] |     +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:2.0.0.BUILD-SNAPSHOT:compile
[INFO] |     |  \- io.projectreactor.ipc:reactor-netty:jar:0.7.3.BUILD-SNAPSHOT:compile
[INFO] |     |     +- io.netty:netty-codec-http:jar:4.1.20.Final:compile
[INFO] |     |     |  \- io.netty:netty-codec:jar:4.1.20.Final:compile
[INFO] |     |     +- io.netty:netty-handler:jar:4.1.20.Final:compile
[INFO] |     |     |  +- io.netty:netty-buffer:jar:4.1.20.Final:compile
[INFO] |     |     |  \- io.netty:netty-transport:jar:4.1.20.Final:compile
[INFO] |     |     |     \- io.netty:netty-resolver:jar:4.1.20.Final:compile
[INFO] |     |     +- io.netty:netty-handler-proxy:jar:4.1.20.Final:compile
[INFO] |     |     |  \- io.netty:netty-codec-socks:jar:4.1.20.Final:compile
[INFO] |     |     \- io.netty:netty-transport-native-epoll:jar:4.1.20.Final:compile
[INFO] |     |        +- io.netty:netty-common:jar:4.1.20.Final:compile
[INFO] |     |        \- io.netty:netty-transport-native-unix-common:jar:4.1.20.Final:compile
[INFO] |     +- org.hibernate.validator:hibernate-validator:jar:6.0.7.Final:compile
[INFO] |     |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |     |  +- org.jboss.logging:jboss-logging:jar:3.3.0.Final:compile
[INFO] |     |  \- com.fasterxml:classmate:jar:1.3.1:compile
[INFO] |     +- org.springframework:spring-web:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |     |  \- org.springframework:spring-beans:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |     +- org.springframework:spring-webflux:jar:5.0.3.BUILD-SNAPSHOT:compile
[INFO] |     |  \- io.projectreactor:reactor-core:jar:3.1.3.RELEASE:compile
[INFO] |     |     \- org.reactivestreams:reactive-streams:jar:1.0.2:compile
[INFO] |     \- org.synchronoss.cloud:nio-multipart-parser:jar:1.1.0:compile
[INFO] |        \- org.synchronoss.cloud:nio-stream-storage:jar:1.1.3:compile
[INFO] +- com.mashape.unirest:unirest-java:jar:1.4.9:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  \- commons-codec:commons-codec:jar:1.9:compile
[INFO] |  +- org.apache.httpcomponents:httpasyncclient:jar:4.1.1:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore-nio:jar:4.4.4:compile
[INFO] |  +- org.apache.httpcomponents:httpmime:jar:4.5.2:compile
[INFO] |  \- org.json:json:jar:20160212:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.0.13:compile
[INFO] |  +- ch.qos.logback:logback-core:jar:1.0.13:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] \- com.fasterxml.jackson.core:jackson-databind:jar:2.9.2:compile
[INFO]    +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO]    \- com.fasterxml.jackson.core:jackson-core:jar:2.9.2:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29.677 s
[INFO] Finished at: 2018-01-23T11:49:24+01:00
[INFO] Final Memory: 21M/192M
[INFO] ------------------------------------------------------------------------

Memory graphs

[smaldini]

Thanks for the well documented and detailed report @kzimnicki-n26! We are investigating this with the reactor team. Unfortunately our latest reactor BOM just released but we'll see if we can fix this in gateway directly. /cc @violetagg

[spencergibb]

After changing to

            return proxyRequest.sendHeaders() //I shouldn't need this
                    .send(request.getBody().map(dataBuffer ->
                            ((NettyDataBuffer)dataBuffer).getNativeBuffer()));

I see regular sawtooth behavior.

[kzimnicki-n26]

Solution looks good on my side.
I will try to prepare fix for our production gateway and send some real traffic.

[kzimnicki-n26]

The fix was solving some of the problems, but on production with real traffic memory leak still exists:

With the production dump we could finally see that the problem exists in the micrometer library:

After removing micrometer from dependency:

 <exclusion>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
</exclusion>

The heap memory looks more stable:

[jkschneider]

@kzimnicki-n26 I believe that leak was solved for Micrometer 1.0.0 and by 1.0.3 substantial improvements were made to the heap footprint of timers and distribution summaries.