Description
Describe the bug
We encounter some latency while generating a pre-signed URL for some files in S3 bucket.
We generate it by java microservice which runs in EKS cluster. The cluster and bucket are in the same region.
It uses AWS java SDK, 2.16.4 version.
The microservice uses the Spring Boot framework to handle HTTP requests from the clients.
It's run in openjdk:11-jre-slim container.
We are creating S3Presigner with InstanceProfileCredentialsProvider once the microservice is up( with Bean annotation), and then it generates a pre-signed URL once it gets POST request from any client (currently using Postman to simulate the client). In general, it takes less than 300 ms to get a response from the service. However, sometimes it can take 5 seconds or more. We attached APM to the service to get requests span.
Trace in APM shows that when latency happens in the request we get an error:
java.net.SocketTimeoutException: Read timed out
for PUT /latest/api/token request, which as I understand is a request to IMDS to retrieve credentials from EC2. And only that request takes 1 second.
Questions:
How to prevent those errors?
Are we using the right way the aws sdk for generating pre signed URL?
Do we need to close presigner connection? If yes and if you know how it should be done in Spring Boot Bean?
Attached print screens of code and trace span from APM:
Region: eu-central-1
Expected Behavior
Not to get errors that cause latency while generating pre-signed URL
Current Behavior
The current behavior described above
Log with debug enabled:
Full error caught by APM, happens while SDK sends PUT request to http://169.254.169.254/latest/api/token
java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.net.SocketInputStream.read(Unknown Source)
at java.base/java.io.BufferedInputStream.fill(Unknown Source)
at java.base/java.io.BufferedInputStream.read1(Unknown Source)
at java.base/java.io.BufferedInputStream.read(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:114)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getToken(EC2MetadataUtils.java:442)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getToken(InstanceProfileCredentialsProvider.java:83)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getCredentialsEndpointProvider(InstanceProfileCredentialsProvider.java:69)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:74)
at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
at software.amazon.awssdk.utils.cache.OneCallerBlocks.prefetch(OneCallerBlocks.java:38)
at software.amazon.awssdk.utils.cache.CachedSupplier.prefetchCache(CachedSupplier.java:116)
at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:91)
at java.base/java.util.Optional.map(Unknown Source)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.createExecutionContext(DefaultS3Presigner.java:301)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.presign(DefaultS3Presigner.java:269)
at software.amazon.awssdk.services.s3.internal.presigner.DefaultS3Presigner.presignGetObject(DefaultS3Presigner.java:192)
at scr.translation.bl.TranslationBL.CreatePresignedURL(TranslationBL.java:78)
at scr.translation.controller.TranslationController.CreatePresignedURLs(TranslationController.java:45)
at jdk.internal.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Appl
Steps to Reproduce
try to use the same way to generate a pre-signed URL. The issue reproduced after some time the endpoint is idle. The next request could take much less than a second (80-90 ms)
Context
Latency while generating a pre-signed URL for files in S3
Your Environment
- AWS Java SDK version used: openjdk 11
- JDK version used: 2.16.4
- Operating System and version: EKS v1.15 (inside docker container openjdk:11-jre-slim)