Intermittent failure for getting IAM credential in k8s pod with Kube2IAM role

[shambhand]

Description

Observed Intermittent failure for getting IAM credential in k8s pod with Kube2IAM role

StackTrace:

com.amazonaws.SdkClientException : Failed to connect to service endpoint: 
		com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
		com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
		com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
		com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
		com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:47)
		com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
		com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
		com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
		com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
		com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:111)
		com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1257)
		com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:833)
		com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:783)
		com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
		com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
		com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
		com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
		com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
		com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
		com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:5136)
		com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:5103)
		com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeScan(AmazonDynamoDBClient.java:3273)
		com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.scan(AmazonDynamoDBClient.java:3239)

Environment

• AWS Java SDK version used: 1.11.788
• JDK version used: {"lang":"jvm","lang_version":"11.0.8","jvm_vendor":"AdoptOpenJDK","jvm_version":"11.0.8+10","java_class_version":"55.0"}
• Operating System and version: Docker base Image: adoptopenjdk/openjdk11:alpine-jre
• Kube2IAM: jtblin/kube2iam:0.10.11
• Kubernetes: AWS v1.15.10-EKS

[debora-ito]

Failed to connect to service endpoint indicates the InstanceProfileCredentialsProvider attempted to fetch the credentials but could not connect to the IMDS endpoint before timeout. This can be caused by service latency or network latency, for example.

Currently the SDK does not retry IMDS credentials fetching, you can add a custom retry logic. You can also enable async refreshing of IMDS credentials and see if the condition improve: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html#refresh-credentials

[shambhand]

Thanks @debora-ito for response. I am new here so just trying understand:

• Is not that possible to provide the "custom retry logic" as part of SDK?
• How can I increase timeout?

Can you provide me some favour to understand the thread closing statement mentioned in the documentation.

I have a spring boot web app and creating DynamoDb client as singleton bean on startup. After adding InstanceProfileCredentialsProvider refreshAsync=true Do I need to call close()?

Could you check below code snippet, based on my understanding:
Existing: So far we have been creating dynamoDb instance as below(relying on the SDK internal Default credential chain to call the InstanceProfileCredentialsProvider)

@Bean
protected AmazonDynamoDB initialiseDynamoDB(DynamoDBConfig dynamoDBConfig) {

    final var endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(dynamoDBConfig.getEndpoint(), dynamoDBConfig.getRegion());
    return AmazonDynamoDBClient.builder()
            .withEndpointConfiguration(endpointConfiguration)
            .build();

}

Changed: With the use of refreshAsync=true it will look as below(note: I did not call the close() method ):

@Bean
protected AmazonDynamoDB initialiseDynamoDB(DynamoDBConfig dynamoDBConfig) {

    final var instanceProfileCredentialsProvider = new InstanceProfileCredentialsProvider(true);

    final var endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(dynamoDBConfig.getEndpoint(), dynamoDBConfig.getRegion());
    return AmazonDynamoDBClient.builder()
            .withCredentials(instanceProfileCredentialsProvider)
            .withEndpointConfiguration(endpointConfiguration)
            .build();

}

Please confirm does above making sense and correct to you?

[debora-ito]

Is not that possible to provide the "custom retry logic" as part of SDK?

Yes it's possible, you can set a RetryPolicy in the ClientConfiguration.
Take a look at this blog post to see an example: https://aws.amazon.com/blogs/database/tuning-aws-java-sdk-http-request-settings-for-latency-aware-amazon-dynamodb-applications/

How can I increase timeout?

This connection timeout is not configurable. We have a feature request to make it configurable, you can add your thumbs up: #2365

The code seems good, calling close() after you're done with the credentials provider would release the background thread so your application can have more resources if needed.

[shambhand]

Thanks @debora-ito 👍

[ffeltrinelli]

Hi! I talked about this issue and described our custom solution in this article.