(cli): (cdk fails with "Profile <profile name> did not include credential process)

[kirnberger1980]

Describe the bug

We want to use the CDK CLI with Gitlab Runner, which is running on EKS in Account A. cdk deploy is executed for Account B (Cross-Account Deployment).

cdk deploy --profile AccountB

CLI Command fails with:

Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile <AccountB> did not include credential process
....
Could not assume arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1, proceeding anyway.

We are using IRSA (IAM Role for Service Accounts), further called Role A, for the Gitlab Runner in Account A.
In Account B there is an IAM Role (Role B), which can be assumed by Role A. RoleB can assume all cdk-provisioned IAM Roles in that Account (created by cdk bootstrap).

Profile file (stored in /configmaps/awsconfig):

    [default]
    region = eu-central-1
    role_arn = arn:aws:iam::AccountA:role/RoleA
    web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
    output = json


    [profile AccountB]
    role_arn = arn:aws:iam::AccountB:role/RoleB
    source_profile = default
    output = json
    region = eu-central-1

We have set environment variables AWS_PROFILE and AWS_CONFIG_FILE:

AWS_PROFILE=AccountB
AWS_CONFIG_FILE=/configmaps/awsconfig

AWS CLI is working fine with above profile file and the environment variables:

aws sts get-caller-identity
{
    "UserId": "AROA2SFT2XXXXXXXXX:botocore-session-168600000",
    "Account": "00000000000",
    "Arn": "arn:aws:sts::00000000000:assumed-role/RoleB/botocore-session-168600000"
}

Assuming the cdk-lookup Role from RoleB with AWS CLI works fine.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1 --role-session-name test123
{
    "Credentials": {
        "AccessKeyId": "<somekey>",
        "SecretAccessKey": "<somesecret>",
        "SessionToken": "<nice jwt>",
        "Expiration": "2023-06-06T15:29:36+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "<some otherkey:test123",
        "Arn": "arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1"
    }
}

By the way: cdk also seems not to respect AWS_PROFILE environment variable as it is using the default role (it could be, that the session token by IRSA is used), when NOT using --profile.

Expected Behavior

We expect that AWS CDK CLI is acting identically to AWS CLI when using AWS profiles for Cross-Account Deployments. Environment Variable AWS_PROFILE should be respected.

Current Behavior

The CDK CLI fails that the cdk-Lookup-Role cannot be assumed, because "Profile did not include credential process". When you don't specify --profile, then default-Profile (or session token created by OIDC Provider of IAM) is used.

Reproduction Steps

You need to run a pod in EKS:

• configure IRSA (RoleA) for the pod in the same account (Account A)
• configure a RoleB in Account B, which can be assumed by RoleA
• give permissions to RoleB to assume cdk-provisioned roles
• configure an AWS_CONFIG_FILE like described above.
• install CDK, NODE etc on the pod
• Configure env variables AWS_PROFILE and AWS_CONFIG_FILE
• Run cdk deploy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.78.0

Framework Version

2.78.0

Node.js Version

v18.16.0

OS

node:18-alpine

Language

Typescript

Language Version

3.9.7

Other information

No response

[pahud]

Thanks for reporting.

1. Can you run this in the pod from Account A?

aws --profile AccountB sts get-caller-identity

2. Can you share a minimal CDK code that provisions this environment so I can try reproduce it in my account?

[kirnberger1980]

Thanks for the quick reply.

1. Here is the result of the aws cli response:

{
    "UserId": "AXXXXXXXXXX:botocore-session-1686146634",
    "Account": "<AccountB>",
    "Arn": "arn:aws:sts::<AccountB>:assumed-role/RoleB/botocore-session-1686146634"
}

This is, what we expected, as setting env variable AWS_PROFILE or using --profile should return the identical response.

2. Here is some sample code tested, where we could reproduce the issue:

bin-File:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdkStack } from '../lib/cdk-stack';

const app = new cdk.App();
new CdkStack(app, 'CdkStack', {
  env: { account: "<AccountB>",region: "eu-central-1" }
});

lib-File:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

  const vpc = cdk.aws_ec2.Vpc.fromLookup(this, "MyVPC", {
    vpcName: "MyVpcName"
  });

  new cdk.aws_ec2.SecurityGroup(this, "MyNewSecurityGroup", {
    vpc,
    description: "This will secure my resources"
  });
  }
}

Kind Regards,
Marcell