Skip to content

Get errors java.net.SocketTimeoutException: for PUT /latest/api/token request, while generating pre signed url, which cause latency at least one second. #2339

Closed
@boris-ait

Description

@boris-ait

Describe the bug

We encounter some latency while generating a pre-signed URL for some files in S3 bucket.
We generate it by java microservice which runs in EKS cluster. The cluster and bucket are in the same region.
It uses AWS java SDK, 2.16.4 version.
The microservice uses the Spring Boot framework to handle HTTP requests from the clients.
It's run in openjdk:11-jre-slim container.
We are creating S3Presigner with InstanceProfileCredentialsProvider once the microservice is up( with Bean annotation), and then it generates a pre-signed URL once it gets POST request from any client (currently using Postman to simulate the client). In general, it takes less than 300 ms to get a response from the service. However, sometimes it can take 5 seconds or more. We attached APM to the service to get requests span.
Trace in APM shows that when latency happens in the request we get an error:
java.net.SocketTimeoutException: Read timed out
for PUT /latest/api/token request, which as I understand is a request to IMDS to retrieve credentials from EC2. And only that request takes 1 second.

Questions:
How to prevent those errors?
Are we using the right way the aws sdk for generating pre signed URL?
Do we need to close presigner connection? If yes and if you know how it should be done in Spring Boot Bean?

Attached print screens of code and trace span from APM:
APM+trace
MicrosoftTeams-image+(3)
MicrosoftTeams-image+(1)

Region: eu-central-1

Expected Behavior

Not to get errors that cause latency while generating pre-signed URL

Current Behavior

The current behavior described above

Log with debug enabled:

debug.log

Full error caught by APM, happens while SDK sends PUT request to http://169.254.169.254/latest/api/token

java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.io.BufferedInputStream.fill(Unknown Source)
at java.base/java.io.BufferedInputStream.read1(Unknown Source)
at java.base/java.io.BufferedInputStream.read(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:114)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getToken(EC2MetadataUtils.java:442)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getToken(InstanceProfileCredentialsProvider.java:83)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getCredentialsEndpointProvider(InstanceProfileCredentialsProvider.java:69)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:74)
at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
at software.amazon.awssdk.utils.cache.OneCallerBlocks.prefetch(OneCallerBlocks.java:38)
at software.amazon.awssdk.utils.cache.CachedSupplier.prefetchCache(CachedSupplier.java:116)
at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:91)
at java.base/java.util.Optional.map(Unknown Source)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.createExecutionContext(DefaultS3Presigner.java:301)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.presign(DefaultS3Presigner.java:269)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.presignGetObject(DefaultS3Presigner.java:192)
at scr.translation.bl.TranslationBL.CreatePresignedURL(TranslationBL.java:78)
at scr.translation.controller.TranslationController.CreatePresignedURLs(TranslationController.java:45)
at jdk.internal.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Appl

Steps to Reproduce

try to use the same way to generate a pre-signed URL. The issue reproduced after some time the endpoint is idle. The next request could take much less than a second (80-90 ms)

Context

Latency while generating a pre-signed URL for files in S3

Your Environment

  • AWS Java SDK version used: openjdk 11
  • JDK version used: 2.16.4
  • Operating System and version: EKS v1.15 (inside docker container openjdk:11-jre-slim)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.closed-for-stalenessresponse-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.service-apiThis issue is due to a problem in a service API, not the SDK implementation.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions