Using Jetty with Spring Webclient causes thread leak

[thomas-br]

Hi,

I am currently looking into using the jetty httpclient to replace netty for my Spring Webclient which is responsible for external calls. As there is a hard limit when these external requests shall be cancelled, I am using a timeout in the Webclient execution call:

ResponseEntity<String> response = request.retrieve()
    .toEntity(String.class)
    .timeout(Duration.ofMillis(timeoutMillis))
    .block();

I am using Jetty as ClientHttpConnector, simplified:

HttpClient httpClient = new HttpClient(SSL_CONTEXT_FACTORY);
WebClient.builder()
    .clientConnector(new JettyClientHttpConnector(httpClient))
...

In a load test scenario where all calls are cancelled before it can finish, I am observing a thread leak on the Jetty HttpClient Threads:

Even after a long cooldown they are not coming back. And at the peak new requests are failing due to no free resources.

Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:804)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:660)

[sbordet]

Can you take a thread dump to see what are those threads doing?

[thomas-br]

Can you take a thread dump to see what are those threads doing?

Sure. You can find the thread dump here

[thomas-br]

@sbordet Could you get any insight out of it?

[sbordet]

@thomas-br sorry for the late reply, I was in vacation.
Looking at the dumps now.

[sbordet]

From your logs, there are 439 different instances of HttpClient.

That is not good, as there should be only 1.
Each instance will require selector threads, schedulers, etc. so the 439 of them will consume a lot of threads.

I suspect there is something really wrong in the way you setup your application.

[thomas-br]

@sbordet
Thanks for your reply. Okay, yes indeed I am creating a HttpClient for each request.

On the one handside same procedure does not cause such a thread behavior with Netty HttpClient (see graph below).

On the other, if you say ideally a singleton instance of the HttpClient should be used sounds generally fair. But the issue I see is, that proxy-configuration can only be applied globally on the HttpClient but which may differ in my case per request. So I do not see an option how to do that thread-safe per request (parallel) without new instances.

[rstoyanchev]

When creating the JettyClientHttpConnector you can also pass a JettyResourceFactory to re-use the same resources across different client instances. Or you can also configure HttpClient yourself with the same resources, if you see what that constructor does. The convenience of the JettyClientHttpConnector is that it's designed to be declared in Spring configuration and automatically start and stop the resources with the Spring ApplicationContext lifecycle.

[sbordet]

But the issue I see is, that proxy-configuration can only be applied globally on the HttpClient but which may differ in my case per request.

Can you detail this?

Typically when using a proxy, you want to say that requests to a particular origin should be proxied - see for example how they are configured in a browser.

Do you have a case where, for the same origin, you want to use a proxy for one request, but not for another?

[thomas-br]

But the issue I see is, that proxy-configuration can only be applied globally on the HttpClient but which may differ in my case per request.

Can you detail this?

Typically when using a proxy, you want to say that requests to a particular origin should be proxied - see for example how they are configured in a browser.

Do you have a case where, for the same origin, you want to use a proxy for one request, but not for another?

Yes sure. So basically the use case would be that the service itself would not know any request information on application startup, but everything is basically dynamic configuration per "job". So it get's at runtime the task to execute the following request with the passed configuration. We cannot suppose any similarity on, for example, same hosts.

Imagine the first request to example.com is configured to don't use any proxy, second requests is configured to use a certain proxy and third one to the same host / url can even have a complete different proxy setting in place. Same applies for the expected timeout behavior, which is also another problem with a single HttpClientInstance. So every job is completely independent of any other, and should behave stateless without any interference.

Still for testing purposes, I have tried a singleton instance of HttpClient with a static proxy-configuration. Yes this solves the threading problem initially reported. But during the same load test I have seen another issue. Load test setup is the following:

• Attempted HTTP-Request is requesting a large file, e.g. 20MB
• On the Spring Webclient configuration we are setting a timeout (see coding above) of (for simplicity) 1s, thus expecting all requests to be cancelled after that time
• 10 Threads in parallel, thus new requests after each cancellation, all to the same host (for simplicity)

Thread / Memory behavior seems to be okay, but from a certain point I receive the following error:

Max requests queued per destination 1024 exceeded for HttpDestination[https://<target>.com]@7da592a2,queue=1024,pool=DuplexConnectionPool@43c321e[c=0/64/64,a=64,i=0]

Which is within the load period acceptable. But after stopping the load, all WebClient requests are at latest done after 1s. Still this message stays for more than 30min, thus making the instance unusable in such a scenario.

It seems although the WebClient request is cancelled, internally something seems to be queued still. Why is this the case? Why doesn't the triggered timeout by Spring WebClient on the Mono leads to a cleanup of everything related of the initial HttpClient request?

[sbordet]

It's the first time that I encounter the need of different proxy configurations for different requests for the same origin server (e.g. browsers do not support this).

Jetty's HttpClient currently does not support this because the proxy configuration is per-HttpClient, not per-request.
You may want to open an issue to https://github.com/eclipse/jetty.project/issues for this.

The workaround is to use different HttpClient instances each with different proxy configurations.
You can even get by using a HttpClient per request, but you need to create, configure, start the HttpClient instance, then send the request, and when the request is complete you must stop the HttpClient instance to shutdown the thread pool and all the other components. One HttpClient per request will impact performance heavily - you can perhaps improve by creating a new HttpClient instance only if you have a proxy configuration (or a new proxy configuration).

About the load test you describe in the above comment, I am not sure I understand "Still this message stays for more than 30min".
From the HttpClient point of view, if you abort a request, it's gone and won't be retried.
Also the Jetty's Reactive HttpClient wrapper does not retry requests.
Can you verify there is no automatic retry mechanism in Spring WebClient?

[thomas-br]

Okay. Accepting that the proxy configuration is bound per HttpClient level.

For the other issue:

About the load test you describe in the above comment, I am not sure I understand "Still this message stays for more than 30min".
From the HttpClient point of view, if you abort a request, it's gone and won't be retried.
Also the Jetty's Reactive HttpClient wrapper does not retry requests.
Can you verify there is no automatic retry mechanism in Spring WebClient?

It seems to be a deadlock. I am not talking about any retry behavior. But more on the mechanism when the Mono of a Webclient request is cancelled by the timeout property. It seems when the timeout on Mono level occurs, the request or better say the resources inside the Jetty HttpClient are not terminated and freed properly.

My observation is that after heavy load of the same requests performed to the same host, which are all terminated by the Mono timeout, will produce a deadlock of leaked queued requests. In my example all WebClient request tasks are finished. But still the HttpDestination queue within the DuplexConnectionPool stays full and blocks new request to the same host until the application restarts. I have observed the instance now for a couple of hours, the queue is not getting cleared.

[sbordet]

Any chance that you can write a simple reproducer?

[thomas-br]

Any chance that you can write a simple reproducer?

Sure, will probably find some time next week :)

[thomas-br]

Hi @sbordet

Simple reproducing project can be found here

Simply insert in the URI constant an arbitrary large file which takes more than 1sec to load.
Background is that we want to run in the timeout case.

Fire a lot of requests, for example with jmeter. I used 10 threads in parallel.
After a few minutes you see the exception described above coming in which shows the block of the host of your file, which persists even after cooldown.