Jetty 12.0.8 seems to leak connection when it encounters earlyEOF

[LarsKrogJensen]

Jetty version(s)
12.0.8

Jetty Environment
ee10

Java version/vendor (use: java -version)
java 21.0.2 2024-01-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30, mixed mode, sharing)

OS type/version
Ubuntu 22.04

Description
We have seen a clear increase of requests from mobile clients where jetty fails to handle/parse the request body and causes HttpConnection$RequestHandler.earlyEOF to trigger.
That in itself is perhaps not alarming, but it seems that this causes Jetty to leak requests and connection, we can see a steady increase in our metrics of QoSHandler's active requests. Eventually QoS hits the max active requests and starts rejecting requests.

We have a very strong correlation between these errors and leaking connections/requests.

Other sites where we do not see these errors we do not see any leakage.

How to reproduce?
I am afraid I have not been able to reproduce to make jetty trigger HttpParser.earlyEOF, not sure how to simulate a bad client :(

Stacktrace
message: "Error while closing the output stream in order to commit response"
stacktrace:

org.glassfish.jersey.server.ServerRuntime$Responder: org.eclipse.jetty.http.BadMessageException: 400: Early EOF
	at o.e.j.s.internal.HttpConnection$RequestHandler.earlyEOF(HttpConnection.java:1051)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1729)
	at org.eclipse.jetty.server.internal.HttpConnection.parseRequestBuffer(HttpConnection.java:568)
	at org.eclipse.jetty.server.internal.HttpConnection.parseAndFillForContent(HttpConnection.java:493)
	at o.e.j.s.i.HttpConnection$HttpStreamOverHTTP1.read(HttpConnection.java:1346)
	at org.eclipse.jetty.server.HttpStream$Wrapper.read(HttpStream.java:161)
	at o.e.j.s.internal.HttpChannelState$ChannelRequest.read(HttpChannelState.java:888)
	at org.eclipse.jetty.server.Request$Wrapper.read(Request.java:791)
	at o.e.j.server.handler.EventsHandler$EventsRequest.read(EventsHandler.java:376)
	at org.eclipse.jetty.server.Request$Wrapper.read(Request.java:791)
	at o.eclipse.jetty.ee10.servlet.AsyncContentProducer.readChunk(AsyncContentProducer.java:319)
	at o.eclipse.jetty.ee10.servlet.AsyncContentProducer.produceChunk(AsyncContentProducer.java:305)
	at o.eclipse.jetty.ee10.servlet.AsyncContentProducer.isReady(AsyncContentProducer.java:243)
	at o.e.jetty.ee10.servlet.BlockingContentProducer.nextChunk(BlockingContentProducer.java:108)
	at org.eclipse.jetty.ee10.servlet.HttpInput.read(HttpInput.java:244)
	at org.eclipse.jetty.ee10.servlet.HttpInput.read(HttpInput.java:225)
	at o.g.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
	at o.g.j.m.i.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:276)
	at c.f.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:547)
	at c.f.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:137)
	at c.f.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:266)
	at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1863)
	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1262)
	at o.g.j.j.internal.jackson.jaxrs.base.ProviderBase._createParser(ProviderBase.java:843)
	at o.g.j.j.internal.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:787)
	at o.g.j.m.i.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:235)
	at o.g.j.m.i.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:214)
	at o.g.j.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:134)
	at o.g.j.s.i.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49)
	at o.g.j.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:134)
	at o.g.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
	at o.g.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:657)
	at org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290)
	at o.g.j.s.i.i.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73)
	at o.g.j.s.i.i.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:56)
	at o.g.j.s.spi.internal.ParamValueFactoryWithSource.apply(ParamValueFactoryWithSource.java:50)
	at o.g.j.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:68)
	at o.g.j.s.m.i.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:109)
	at o.g.j.s.m.i.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
	at o.g.j.s.m.i.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at o.g.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at o.g.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at o.g.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:263)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at o.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:242)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:312)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:736)
	at o.e.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1614)
	at c.k.p.f.rest.filter.RequestContextBindingFilter.doFilter(RequestContextBindingFilter.java:63)
	at org.eclipse.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
	at o.eclipse.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1586)
	at o.e.j.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1547)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:819)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:431)
	at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:464)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:851)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at org.eclipse.jetty.server.handler.ResourceHandler.handle(ResourceHandler.java:164)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at o.eclipse.jetty.server.handler.CrossOriginHandler.handle(CrossOriginHandler.java:375)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at o.eclipse.jetty.server.handler.ConditionalHandler.nextHandler(ConditionalHandler.java:421)
	at org.eclipse.jetty.server.handler.QoSHandler.handleWithPermit(QoSHandler.java:266)
	at org.eclipse.jetty.server.handler.QoSHandler.onConditionsMet(QoSHandler.java:191)
	at o.eclipse.jetty.server.handler.ConditionalHandler.handle(ConditionalHandler.java:378)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at c.k.p.frontend.ratelimiter.PunterThrottlingHandler.handle(PunterThrottlingHandler.java:36)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at c.k.k.jetty.KelogsTracingHandler.handle(KelogsTracingHandler.java:53)
	at org.eclipse.jetty.server.Handler$Wrapper.handle(Handler.java:740)
	at org.eclipse.jetty.server.handler.EventsHandler.handle(EventsHandler.java:81)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:597)
	at org.eclipse.jetty.server.Server.handle(Server.java:179)
	at o.e.j.s.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:619)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
	at o.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at j.util.concurrent.ThreadPerTaskExecutor$TaskRunner.run(ThreadPerTaskExecutor.java:314)
	at java.lang.VirtualThread.run(VirtualThread.java:309)

[LarsKrogJensen]

Just to add to what I found; from a heap dump I can see Eclipse MAT is (this log is from heap-hero that seems to be built on MAT)

2,971 instances of "org.eclipse.jetty.io.SocketChannelEndPoint", loaded by "jdk.internal.loader.ClassLoaders$AppClassLoader @ 0x6ac006a40" occupy 361,225,312 (75.72%) bytes. These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by ""

At the point of heap dump there is certainly not that many real active requests.

[sbordet]

@LarsKrogJensen can you share the heap dump?

[LarsKrogJensen]

I don't think so as it contains various sensitive data

[LarsKrogJensen]

If I could figure out a way to reproduce earlyEOF I would in a separate standalone app it would ofc be fine.

How do you verify this scenario? any tooling

[LarsKrogJensen]

not sure sharing screenshots help

[sbordet]

@LarsKrogJensen

If you can write a reproducer, will be great.

Otherwise, please detail exactly how you configured your application.

What Handlers, in what order, their configuration, etc. so that we can try to reproduce the issue as closely as possible as your configuration.

[LarsKrogJensen]

I will for sure continue trying to create a reproducer, already spent many hours without success. I raised this issue cause I was hoping you would be able to spot something in the code that explains it.

@sbordet
Do you have any unit/integration test that simulates the earlyEOF that I can learn from?

Application is using jetty and jersey to provide a rest api, otherwise it is framework-free™.
Using virtual threads as described in https://webtide.com/jetty-12-virtual-threads-support/

Handler chain obtained from Server.dump():

GZipHandler
StatisticsHandler
CustomTracingHandler (simple Handler.Wrapper implementation)
CrossOriginHandler
CustomRateLimitHandler (simple Handler.Wrapper using Bucket4j to rate limit with redis as backend)
QoSHandler (max requests 4k, max suspend 10sec)
ServlerContextHandler
ConstraintSecurityHandler (limits allowed methods)
ServletHandler (running Jersey ServletContainer)

[LarsKrogJensen]

@sbordet I sanitized the heap dump with https://github.com/paypal/heap-dump-tool and got clearance to share it with you privately. Dump is 500M and ziped 50M.

[sbordet]

@LarsKrogJensen email it to sbordet@webtide.com, let's see if it's not too big.

[sbordet]

@LarsKrogJensen I wrote a test case with Jersey that tries to reproduce your issue.

I get a stack trace that is almost identical to yours, but everything works fine: the connection is closed on the server, and the server endPoint removed.

I analyzed the heap dump, and there are few things I noted:

• The idle timeout is 30 minutes. This is typically too long, and the reason those connections stay around for such a long time.
• Many of the Jetty objects that manage responses are in a state where they have not written any response.
• You are using virtual threads, and 3 of them are pinned into NIO select(). This is not good.
• The other 4 virtual threads instances in the heap dump have no carrier, and they are blocked on some lock.

We suspect your problems have to do with virtual threads.
We apparently cannot reproduce them because it's not your full system, and you may make calls to code that is not virtual thread safe (and pins the carriers).

That would explain why the QoSHandler would not decrement the permits: the virtual thread entered to serve the request, got blocked and was never resumed (which also explains why the Jetty output objects have not written any response).
Considering that 3 carriers are pinned, and you seem to have configured a parallelism of 4, you have only 1 carrier left (if any at all -- maybe we just did not find where it is pinned).

If you want to go to the bottom of this issue, you should take a thread dump when your system is again in a state similar to the one captured with the heap dump, and let's see what it shows. Then it's going to be a chase to understand what virtual threads are doing, as explained in the Observing virtual threads section of JEP 444: set jdk.trackAllThreads=true and use jcmd <pid> Thread.dump_to_file -format=json <file>.

Otherwise, I suggest that you disable virtual threads, and try your system again, and see if you have the same problems.

Let us know.
We are very interested in knowing interactions of Jetty with virtual threads, which sometimes are very unexpected.

[LarsKrogJensen]

Thanks for thorough explanations, I will continue to try to create a standalone reproducer on my side.

VThread pinning is indeed a pita, have been monitoring it with jfr recordings to examine any > 20ms pinnings and found nothing. So I am a bit surprised of this finding and need to dig deeper.

I did a jcmd thread dump under similar scenario, but not at same time as the head dump. Will send it to you.

Will keep this issue open for a few more days so that I can do a bit more investigations.

[LarsKrogJensen]

Forgot to mention, we been running with jetty 12 and virtual threads for a couple of months now and it has been performing really, really well, even under really heavy load.

[sbordet]

we been running with jetty 12 and virtual threads for a couple of months now and it has been performing really, really well, even under really heavy load

Do you have a comparison with platform threads?
Also platform threads run really, really well?

Can you detail what do you mean by "well"? Less latency, more throughput, less memory used, less CPU, etc?
What exactly is better with virtual threads?