Error while using aws-secretsmanager with aws-java-sdk-sts

[jainhitesh9998]

Hi,
I'm working to deploy spring boot applicaiton with aws-secretsmanager-jdbc integrated. It works fine with the the usual flow where I set AWS_ACCESS_KEY_ID and and VALUE in env variable. when i'm trying to deploy the app in eks, ive created a service role with access to a particular secret, and included aws-java-sdk-sts to assume the role which is injected in the pod.
when i'm not adding the sdk-sts to pom, it tries to assume the role of the managed nodes, which doesn't have access to secrets. whenever I add the above library, it is able to assume the proper role but gives the below stack trace.

Caused by: java.lang.StackOverflowError: null
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
        at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1369)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1338)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1327)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:488)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:460)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73) ```

is there a way to use AWS_WEB_IDENTITY_TOKEN_FILE instead of aws creds to access the secrets

[ktzevelekidis]

Hello,
I am facing a similar issue. In my case, even when I add the sts module on the classpath (in my application's build.gradle), it still does not use the correct credentials provider (i.e. WebIdentityTokenFileCredentialsProvider). It uses the node's IAM role.

Did you manage to find a solution?

I was thinking to download the library, add the sts dependency in the pom.xml, build it and then use (hoping that something will change).

[jainhitesh9998]

I've checked this library, it uses a pretty old version of aws as of now which doesn't support web identity token so even including sts module won't belp in this case. This needs an update to use compatible aws sdk version. As of now I've not found a solution for this. I'll be mostly using at sealed secrets for the time being. May also look into modifications needed in this code to support sts

[ktzevelekidis]

I added this dependency in my build.gradle:

• com.amazonaws:aws-java-sdk-sts:1.11.1004

and it worked for me. However, I get an exception the first time the application starts:

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

As a consequence the application restarts and then it works.

[jainhitesh9998]

So it's using the node's iam role instead of the service account injected into pod right. I didn't want the node to have dB access so decided to dropped it. Was running short of time so had to look into alternative approaches

[ktzevelekidis]

No, it is using the service account role and not the node's role. The only issue is that in the first boot, I get the aforementioned exception, and a restart takes place. After this it works as expected. I will investigate further and I will inform you for my findings. Anyway, it would be good if the library will be migrated to aws sdk v2. I saw that there is an open PR for the migration from November, but not yet reviewed.

[jainhitesh9998]

@ktzevelekidis you mentioned initially that it was picking nodes iam role can you clarify more. For me when I add sts module which supports it, the pod just throws a stack trace. It looked to me like circular dependency error but it never managed to work

[ktzevelekidis]

Maybe I confused you. So, let me explain.

If the sts dependency is not present in the classpath (i.e. not inside build.gradle for my case), it will use the node's IAM role, which is something we do not want, as you also said. When we add the sts dependency, it uses the service account role. In order to achieve this with this specific library, it seems that you need to add the v1 of this sts dependency (i.e. com.amazonaws:aws-java-sdk-sts:1.11.1004).
I am using version 1.0.6 of the aws-secretsmanager-jdbc library

I hope it is more clear now!

[willtong1234]

Thank you for opening this issue - we are looking into it.

[KarakiBusiness]

I'm getting the same error - when I use the DefaultAWSCredentialsProviderChain locally, and I provide only the assumed role via the AWS_ROLE_ARN and AWS_ROLE_SESSION_NAME , I get the error, but when I also give it an access key and secret, it works as expected. I'm testing a deploy to ECS now and getting this error.

[galvezlj]

still not working for service role?

[itlicious]

We have the same issue on EKS. AWS_ROLE_ARN is set through a service-account. But it still uses the assumed role of the node.