Get errors java.net.SocketTimeoutException: for PUT /latest/api/token request, while generating pre signed url, which cause latency at least one second.

[boris-ait]

Describe the bug

We encounter some latency while generating a pre-signed URL for some files in S3 bucket.
We generate it by java microservice which runs in EKS cluster. The cluster and bucket are in the same region.
It uses AWS java SDK, 2.16.4 version.
The microservice uses the Spring Boot framework to handle HTTP requests from the clients.
It's run in openjdk:11-jre-slim container.
We are creating S3Presigner with InstanceProfileCredentialsProvider once the microservice is up( with Bean annotation), and then it generates a pre-signed URL once it gets POST request from any client (currently using Postman to simulate the client). In general, it takes less than 300 ms to get a response from the service. However, sometimes it can take 5 seconds or more. We attached APM to the service to get requests span.
Trace in APM shows that when latency happens in the request we get an error:
java.net.SocketTimeoutException: Read timed out
for PUT /latest/api/token request, which as I understand is a request to IMDS to retrieve credentials from EC2. And only that request takes 1 second.

Questions:
How to prevent those errors?
Are we using the right way the aws sdk for generating pre signed URL?
Do we need to close presigner connection? If yes and if you know how it should be done in Spring Boot Bean?

Attached print screens of code and trace span from APM:

Region: eu-central-1

Expected Behavior

Not to get errors that cause latency while generating pre-signed URL

Current Behavior

The current behavior described above

Log with debug enabled:

debug.log

Full error caught by APM, happens while SDK sends PUT request to http://169.254.169.254/latest/api/token

java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.io.BufferedInputStream.fill(Unknown Source)
at java.base/java.io.BufferedInputStream.read1(Unknown Source)
at java.base/java.io.BufferedInputStream.read(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:114)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getToken(EC2MetadataUtils.java:442)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getToken(InstanceProfileCredentialsProvider.java:83)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getCredentialsEndpointProvider(InstanceProfileCredentialsProvider.java:69)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:74)
at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
at software.amazon.awssdk.utils.cache.OneCallerBlocks.prefetch(OneCallerBlocks.java:38)
at software.amazon.awssdk.utils.cache.CachedSupplier.prefetchCache(CachedSupplier.java:116)
at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:91)
at java.base/java.util.Optional.map(Unknown Source)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.createExecutionContext(DefaultS3Presigner.java:301)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.presign(DefaultS3Presigner.java:269)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.presignGetObject(DefaultS3Presigner.java:192)
at scr.translation.bl.TranslationBL.CreatePresignedURL(TranslationBL.java:78)
at scr.translation.controller.TranslationController.CreatePresignedURLs(TranslationController.java:45)
at jdk.internal.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Appl

Steps to Reproduce

try to use the same way to generate a pre-signed URL. The issue reproduced after some time the endpoint is idle. The next request could take much less than a second (80-90 ms)

Context

Latency while generating a pre-signed URL for files in S3

Your Environment

• AWS Java SDK version used: openjdk 11
• JDK version used: 2.16.4
• Operating System and version: EKS v1.15 (inside docker container openjdk:11-jre-slim)

[debora-ito]

Hi @boris-ait we have had reports of high latency when using the IMDSv2 endpoint, which is called by the InstanceProfileCredentialsProvider. Previous reports include #1667 and aws/aws-sdk-java#2276

The issue is still being investigated by the EC2 team.

To answer your questions:

• There's no way to prevent this if you are using an SDK version after the change to IMDSv2. We reduced the ReadTimeout and ConnectTimeout to 1 second so requests can fail faster.
• About closing the presigner connection, yes, it's a good practice to call presigner.close() to release the resources once the client is not needed. This is mentioned in the S3Presigner documentation.

To avoid duplicated issues, I'm marking this to close soon. We can keep track of this issue in #1667. Let us know if you have any more questions.

[github-actions]

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[boris-ait]

Adding dev team. Doron, Liron, FYI.

…

On Tue, Mar 23, 2021 at 3:44 AM Debora N. Ito ***@***.***> wrote: Hi @boris-ait <https://github.com/boris-ait> we have had reports of high latency when using the IMDSv2 <https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/> endpoint, which is called by the InstanceProfileCredentialsProvider. Previous reports include #1667 <#1667> and aws/aws-sdk-java#2276 <aws/aws-sdk-java#2276> The issue is still being investigated by the EC2 team. To answer your questions: - There's no way to prevent this if you are using an SDK version after the change to IMDSv2. We reduced the ReadTimeout and ConnectTimeout to 1 second so requests can fail faster. - About closing the presigner connection, yes, it's a good practice to call presigner.close() to release the resources once the client is not needed. This is mentioned in the S3Presigner documentation <http://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/presigner/S3Presigner.html> . To avoid duplicated issues, I'm marking this to close soon. We can keep track of this issue in #1667 <#1667>. Let us know if you have any more questions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2339 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANEHYOZXN7KZCVYXINLID7DTE7W7NANCNFSM4ZNRYJVQ> .

-- *Best regards,* *Boris Paperny, **DevOps Engineer* Email: [email protected] Web: http://www.Automat-IT.com <http://www.automat-it.com/> <https://mailtrack.io/trace/link/6a95f2ffa66876d51dec2faedb631947ad05c63b?url=http%3A%2F%2Fwww.Automat-IT.com&userId=2840505&signature=d33c6ec7e9a5c4e1> *‘THE’ DevOps <https://mailtrack.io/trace/link/c9621480b42ee99511f5af151979e86ca35dca19?url=http%3A%2F%2F%2F&userId=2840505&signature=73f3683a19e4b85a> Company!*