Intermittent issue on sdk v2 - Unable to load credentials from service endpoint

[striker50]

Describe the bug

I am running a Spark application on an EMR 5.30.1 cluster. Seeing intermittent credential access issue after upgrading AWS SDK version from 1.11.297 to 2.17.11.
We upgraded from v1 to v2 as v1 doesn't have stable support for configuring custom VPC endpoints - As recommended here aws/aws-sdk-java#2135 (comment)

On moving to sdk v2, the VPC endpoint access is working fine but we are seeing INTERMITTENT SQS sendMessage() failures because of credential access issue due to connection timed out.
Following https://docs.amazonaws.cn/en_us/sdk-for-java/latest/developer-guide/migration-client-credentials.html I also enabled async credential refresher using below code during SQS client initialization. But the issue still occurs

Please advise on how to fix the intermittent credential access issue on aws sdk v2. We get the credentials from InstanceProfileCredentialsProvider.

Expected Behavior

Consistent credential access for the SQSClient when sending messages

Current Behavior

code snippet:

URI endpointURI = new URI(sqsVPCEndpoint);
InstanceProfileCredentialsProvider provider = InstanceProfileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(true)
.build();
SqsClient sqs = SqsClient.builder().region(Region.of(awsRegion)).endpointOverride(endpointURI).
credentialsProvider(provider).build();

Error message:
java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint. at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) ...... at java.lang.Thread.run(Thread.java:750) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint. at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:110) at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132) at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89) at java.util.Optional.map(Optional.java:215) at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146) at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.sqs.DefaultSqsClient.sendMessage(DefaultSqsClient.java:1528) ....... ....... Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1228) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1207) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) at software.amazon.awssdk.regions.internal.util.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:45) at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:112) at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91) at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:79) ... 21 more

Reproduction Steps

code snippet for AWS SDK v2:

URI endpointURI = new URI(sqsVPCEndpoint);
InstanceProfileCredentialsProvider provider = InstanceProfileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(true)
.build();
SqsClient sqs = SqsClient.builder().region(Region.of(awsRegion)).endpointOverride(endpointURI).
credentialsProvider(provider).build();

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.11

JDK version used

1.8

Operating System and version

EMR clusters

[yasminetalby]

Hello @striker50

Thank you very much for your submission.

Here, the unable to connect to service endpoint indicates that the InstanceProfileCredentialsProvider attempted to refresh the credentials but could not connect to the service endpoint before timeout.

After some research, I have found some other users issue submission reporting high latency of credential refresh : see
This latency issue seems to be present in later version of v1 (started in 1.11.678 because it's when the new Instance Metadata Service v2 was released, we have seen various reports of increased latency on the service side), this is why I believed this wasn’t an issue you have encountered prior to upgrading from 1.11.297 to 2.17.11.

Since you are using EMR, you can adjust the hop limit using the modify-instance-metadata-options command if you need to make it larger. You can find more information on the use of IMDSv2 as well as hop-limit configuration here.

Note: This seems to be a bug submission question for the AWS Java SDK V2 rather than a bug submission for AWS Java SDK V1. To facilitate other user guidance search, I will update the label accordingly and transfer the submission to the appropriate repository.

Best,

Yasmine

[striker50]

@yasminetalby We created a new EMR cluster with below configs but still seeing the same issue



Please guide if there is some misconfiguration here. This is on EMR 5.32.1. We need this configuration to be applicable for core and task nodes as well
Here is the code snippet. Should I make any change to code/IAM role policy?

URI endpointURI = new URI(sqsVPCEndpoint);
InstanceProfileCredentialsProvider provider = InstanceProfileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(true)
.build();
SqsClient sqs = SqsClient.builder().region(Region.of(awsRegion)).endpointOverride(endpointURI).
credentialsProvider(provider).build();

[skumarstrike02]

@yasminetalby Any update on this issue ? we are blocked on deploying our services to prod due to this issue. Can you please escalate it.

[yasminetalby]

Hello @skumarstrike02 , @striker50 ,

Apologies for the delay.
To confirm the origin of the behavior: Could you please provide the full stack trace associated with the issue as well as the verbose wirelog?

Please make sure to remove any sensitive information.

It seems that the EC2 team behind the Metadata Service and are still investigating the latency issue mentioned above.

Best,

Yasmine

[yasminetalby]

I was also wondering it you have enabled the SDK metrics?
Do you notice high CredentialsFetchDuration? Or higher latency of a specific request?

Best,

Yasmine

[skumarstrike02]

HI @yasminetalby We ran the service with updated logging settings, below are the logs around the error :
2022-10-12 02:21:28 DEBUG software.amazon.awssdk.request:84 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=<VPC ENDPOINT>, encodedPath=, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[]) 2022-10-12 02:21:28 DEBUG software.amazon.awssdk.request:84 - Received successful response: 200 2022-10-12 02:21:28 DEBUG software.amazon.awssdk.request:84 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=<VPC ENDPOINT>, encodedPath=, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[]) 2022-10-12 02:21:28 ERROR <code error message> java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint. at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) ...... at java.lang.Thread.run(Thread.java:750) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint. at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:110) at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132) at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89) at java.util.Optional.map(Optional.java:215) at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146) at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.sqs.DefaultSqsClient.sendMessage(DefaultSqsClient.java:1528) .... at java.lang.Iterable.forEach(Iterable.java:75) .... at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1228) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1207) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) at software.amazon.awssdk.regions.internal.util.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:45) at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:112) at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91) at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider$InstanceProviderCredentialsEndpointProvider.endpoint(InstanceProfileCredentialsProvider.java:150) at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:112) at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91) at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:79) ... 21 more 2022-10-12 02:21:28 DEBUG software.amazon.awssdk.request:84 - Received successful response: 200