HttpComponentsClientHttpRequestFactory.setBufferRequestBody(false) does not stop OutOfMemoryError [SPR-11977]

[spring-projects-issues]

Syam Sarangadharan opened SPR-11977 and commented

I am using HttpComponentsClientHttpRequestFactory to establish HttpConnection pooling. I am getting OutOfMemoryError. To avoid this as suggested I used HttpComponentsClientHttpRequestFactory.setBufferRequestBody(false) to prevent the payload getting buffered. But still I get the same OutOfMemoryError.

Here is the code snippet.

public String send(String url, String request) throws RestClientException {

	// Create the HttpComponentsClientHttpRequestFactory
	HttpComponentsClientHttpRequestFactory preconfiguredHTTPInstance = new HttpComponentsClientHttpRequestFactory();

	// Set BufferRequestBody as false, default is true.
	preconfiguredHTTPInstance.setBufferRequestBody(false);
	preconfiguredHTTPInstance.setHttpClient(httpClient);

	RestTemplate restTemplate = new RestTemplate();
	restTemplate.setRequestFactory(preconfiguredHTTPInstance);
	restTemplate.getMessageConverters().add(new StringHttpMessageConverter());

	String responseString = restTemplate.postForObject(url, request, String.class);

	return responseString;
}

---

Affects: 4.0.6

Issue Links:

• Inconsistent support for RequestCallback in HttpComponentsClientHttpRequestFactory [SPR-11981] #16597 Inconsistent support for RequestCallback in HttpComponentsClientHttpRequestFactory

[spring-projects-issues]

Stéphane Nicoll commented

Could you please post the OOM stacktrace? Thanks.

[spring-projects-issues]

Stéphane Nicoll commented

I did quite an extensive debugging and profiling session on your code using a 30MB file as input. It turns out that the streaming of the request works just fine and the memory hotspot is on the request object holding those 30MB.

I tried to change your original code to stream the request as well instead of holding it in memory. This turns out to be much harder than I thought so we need to work on that.

Option 1

public String send(String url, String request) throws RestClientException {
    // Create the HttpComponentsClientHttpRequestFactory
    HttpComponentsClientHttpRequestFactory preconfiguredHTTPInstance = new HttpComponentsClientHttpRequestFactory();
 
    // Set BufferRequestBody as false, default is true.
    preconfiguredHTTPInstance.setBufferRequestBody(false);
    preconfiguredHTTPInstance.setHttpClient(httpClient);
 
    RestTemplate restTemplate = new RestTemplate();
    restTemplate.setRequestFactory(preconfiguredHTTPInstance);
    restTemplate.getMessageConverters().add(new StringHttpMessageConverter());
 
    RequestCallback requestCallback = new RequestCallback() {
	@Override
	public void doWithRequest(ClientHttpRequest request) throws IOException {
		((StreamingHttpOutputMessage) request).setBody(new StreamingHttpOutputMessage.Body() {
			@Override
			public void writeTo(OutputStream outputStream) throws IOException {
				IOUtils.copy(content.getInputStream(), outputStream);
			}
		});
	}
    }; 
    HttpMessageConverterExtractor<String> responseExtractor =
            new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());
    String responseString =  restTemplate.execute(url, HttpMethod.POST, requestCallback, responseExtractor);
 
    return responseString;
}

Option 2: switch to the simple counter part (that supports the regular callback)

// Create the HttpComponentsClientHttpRequestFactory
SimpleClientHttpRequestFactory preconfiguredHTTPInstance = new SimpleClientHttpRequestFactory();

// Set BufferRequestBody as false, default is true.
preconfiguredHTTPInstance.setBufferRequestBody(false);

RestTemplate restTemplate = new RestTemplate();
restTemplate.setRequestFactory(preconfiguredHTTPInstance);
restTemplate.getMessageConverters().add(new StringHttpMessageConverter()); 
RequestCallback requestCallback = new RequestCallback() {
	@Override
	public void doWithRequest(ClientHttpRequest request) throws IOException {
		IOUtils.copy(content.getInputStream(), request.getBody());
	}
}; 
HttpMessageConverterExtractor<String> responseExtractor =
		new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());
return restTemplate.execute(url, HttpMethod.POST, requestCallback, responseExtractor);

Could you please try this out and let me know if that fixes your issue? Thanks.

[spring-projects-issues]

Syam Sarangadharan commented

I am trying to post the string data to the url. I am not trying to read from a file and post the data. Please clarify.

[spring-projects-issues]

Stéphane Nicoll commented

Well, that's exactly what I am saying! My tests show that storing the data in a String is the reason of the OutOfMemoryError. If you read that data from a stream instead, the memory usage is perfectly fine.

I would like to confirm that I haven't missed anything. Please look at the code in my previous comment: if that fits with your use case, then I'll resolve this issue as invalid.