eks: After upgrading to 2.80, imported cluster resources fail to deploy

[okoskine]

Describe the bug

After upgrading CDK to 2.80, the EKS cluster creation role no longer can be assumed by separate handlers in stacks that import the EKS cluster as explained in #25674.

The suggested fix of importing the whole kubectlProvider results in Modifying service token is not allowed. error when trying to deploy the stack.

Expected Behavior

Kubernetes resources in the importing stack should still deploy in 2.80+ after adjusting the cluster importing as mentioned in #25674

Current Behavior

cdk8s chart in the importing stack fails to deploy after the upgrade.

StackNameRedacted: creating CloudFormation changeset...
2:20:09 PM | UPDATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | KubeClusterCommonK8sChartA8489958
Modifying service token is not allowed.

❌ StackNameRedacted failed: Error: The stack named StackNameRedacted failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed.
at FullCloudFormationDeployment.monitorDeployment (/workdir/node_modules/aws-cdk/lib/index.js:397:10236)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/workdir/node_modules/aws-cdk/lib/index.js:400:149977)
at async /workdir/node_modules/aws-cdk/lib/index.js:400:135508

❌ Deployment failed: Error: The stack named StackNameRedacted failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed.
at FullCloudFormationDeployment.monitorDeployment (/workdir/node_modules/aws-cdk/lib/index.js:397:10236)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/workdir/node_modules/aws-cdk/lib/index.js:400:149977)
at async /workdir/node_modules/aws-cdk/lib/index.js:400:135508

The stack named StackNameRedacted failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed.

Reproduction Steps

Export (app A, stack A)

const kubectlProvider = cluster.stack.node
  .findChild('@aws-cdk--aws-eks.KubectlProvider') as eks.KubectlProvider

new CfnOutput(scope, 'KubectlProviderRole', {
  exportName: 'KubectlRoleArn',
  value: kubectlProvider.roleArn,
});

new CfnOutput(scope, 'KubectlProviderHandlerRole', {
  exportName: 'KubectlHandlerRoleArn',
  value: kubectlProvider.handlerRole.roleArn,
});

const kubectlHandler = kubectlProvider.node.findChild('Handler') as lambda.IFunction;

new CfnOutput(scope, 'KubectlProviderHandler', {
  exportName: 'KubectlHandlerArn',
  value: kubectlHandler.functionArn,
});

// tried also
// new CfnOutput(scope, 'KubectlProviderHandler', {
//   exportName: 'KubectlHandlerArn',
//   value: kubectlProvider.serviceToken,
// });

Import (app B, stack B)

  const kubectlProvider = eks.KubectlProvider.fromKubectlProviderAttributes(
    scope,
    'KubectlProvider',
    {
      functionArn: Fn.importValue('KubectlHandlerArn'),
      handlerRole: iam.Role.fromRoleArn(
        scope,
        'HandlerRole',
        Fn.importValue('KubectlHandlerRoleArn')
      ),
      kubectlRoleArn: Fn.importValue('KubectlRoleArn'),
    }
  );

  const openIdConnectProvider = iam.OpenIdConnectProvider.fromOpenIdConnectProviderArn(
    scope,
    'KubeOidcProvider',
    Fn.importValue('KubernetesOidcProviderArn')
  );

  const kubectlSecurityGroupId = Fn.importValue('KubernetesControlPlaneSGId');
  return eks.Cluster.fromClusterAttributes(scope, 'KubeCluster', {
    clusterName,
    kubectlPrivateSubnetIds: Fn.split(',', Fn.importValue('KubernetesPrivateSubnetIds')),
    kubectlSecurityGroupId: kubectlSecurityGroupId,
    clusterSecurityGroupId: kubectlSecurityGroupId,
    vpc,
    openIdConnectProvider,
    kubectlLayer: new KubectlV25Layer(scope, 'kubectl'),
    kubectlProvider: kubectlProvider,
  });

Then trying to deploy the (already existing) stack B fails.

Possible Solution

No response

Additional Information/Context

Both of the stacks have been in use for a long time and the only change to the cluster importing was to replace kubectlRoleArn with kubectlProvider. They are part of bigger ckd apps and this problem affects multiple stacks in multiple importing apps.

CDK CLI Version

2.81.0 (build bd920f2)

Framework Version

No response

Node.js Version

v18.14.2

OS

Linux

Language

Typescript

Language Version

TypeScript (3.9.10)

Other information

Came across the following earlier issue with similar symptoms from back when eks was experimental in CDK: #6129.

[peterwoodworth]

Could you post the cdk diff of the stack that fails to deploy please? Full stack examples that I can copy + paste would be helpful too. I'm not sure why the service token would be changing, which it looks like is causing the failure

[okoskine]

Here's a cdk diff of the stack:

Stack StackB
Resources
[-] AWS::CloudFormation::Stack StackBKubeClusterB9F6EB70-KubectlProvider.NestedStack/StackBKubeClusterB9F6EB70-KubectlProvider.NestedStackResource StackBKubeClusterB9F6EB70KubectlProviderNestedStackStackBKubeClusterB9F6EB70KubectlProviderNestedStackResourceFC006381 destroy
[~] Custom::AWSCDK-EKS-KubernetesResource KubeCluster/CommonK8sChart/Resource KubeClusterCommonK8sChartA8489958 
 ├─ [~] RoleArn
 │   └─ [~] .Fn::ImportValue:
 │       ├─ [-] KubernetesMastersRoleArn
 │       └─ [+] KubectlRoleArn
 └─ [~] ServiceToken
     ├─ [-] Removed: .Fn::GetAtt
     └─ [+] Added: .Fn::ImportValue

KubernetesMastersRoleArn was the name of the old output from the main EKS stack for the kubectl role (cluster.kubectlRole.roleArn).

In more detail from synthesized output the ServiceToken used to be:

      ServiceToken:
        Fn::GetAtt:
          - StackBKubeClusterB9F6EB70KubectlProviderNestedStackStackBKubeClusterB9F6EB70KubectlProviderNestedStackResourceFC006381
          - Outputs.StackBStackBKubeClusterB9F6EB70KubectlProviderframeworkonEvent2980E169Arn

And now is:

      ServiceToken:
        Fn::ImportValue: KubectlHandlerArn

To me this seems to be because ImportedKubectlProvider uses the handler arn as the service token (

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/kubectl-provider.ts

Line 226 in 5837373

this.serviceToken = props.functionArn;

).

If you need any more information please ask. I can also try to reproduce this minimally but it might take some time.

[peterwoodworth]

What are you doing in stack B to create resources? I'm interested in reproducing this, and being able to find a workaround, but I'm not super sure what exactly you're doing or what changed in stack B, the snippet posted only imports resources

[peterwoodworth]

The service token changing may have a workaround, or there could be an alternate workaround available, I'm not super sure how exactly to reach this state though.

[okoskine]

Ok, I reproduced this with simple stacks. Steps:

• deploy A with 2.79.0
• deploy B (2.79.0 variant) with 2.79.0
• deploy A with 2.80.0
• deploy B (2.80.0 variant) with 2.80.0 fails with error:

1:47:25 PM | UPDATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | ImportedClustermanifestmanifestD43ADCFB
Modifying service token is not allowed.
 ❌  KubetestStackB failed: Error: The stack named KubetestStackB failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed.
    at FullCloudFormationDeployment.monitorDeployment (/home/onni/import1/node_modules/aws-cdk/lib/index.js:397:10236)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/home/onni/import1/node_modules/aws-cdk/lib/index.js:400:149585)
    at async /home/onni/import1/node_modules/aws-cdk/lib/index.js:400:135508
 ❌ Deployment failed: Error: The stack named KubetestStackB failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed.
    at FullCloudFormationDeployment.monitorDeployment (/home/onni/import1/node_modules/aws-cdk/lib/index.js:397:10236)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/home/onni/import1/node_modules/aws-cdk/lib/index.js:400:149585)
    at async /home/onni/import1/node_modules/aws-cdk/lib/index.js:400:135508

Stacks:
A

export class KubetestStackA extends Stack {
  output(name: string, value: string): void {
    new CfnOutput(this, name, { exportName: name, value: value });
  }
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    const cluster = new Cluster(this, 'TempCluster', {
      version: KubernetesVersion.V1_25,
      kubectlLayer: new KubectlV25Layer(this, 'kubectl'),
      clusterName: "TempCluster",
    });

    this.output('TempOidcProvider', cluster.openIdConnectProvider.openIdConnectProviderArn)
    this.output('TempSg', cluster.clusterSecurityGroupId)
    this.output('TempKubectlRole', cluster.kubectlRole?.roleArn as string)
    this.output('TempKubectlLambdaRole', cluster.kubectlLambdaRole?.roleArn as string)

    const kubectlProvider = this.node.findChild('@aws-cdk--aws-eks.KubectlProvider') as KubectlProvider;
    const kubectlHandler = kubectlProvider.node.findChild('Handler') as IFunction;
    this.output('TempKubectlHandler', kubectlHandler.functionArn)
  }
}

B 2.79.0 variant

export class KubetestStackB extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const vpc = Vpc.fromLookup(this, 'Vpc', { vpcName: 'KubetestStackA/TempCluster/DefaultVpc' });
    const openIdConnectProvider = OpenIdConnectProvider.fromOpenIdConnectProviderArn(this,'KubeOidcProvider', Fn.importValue('TempOidcProvider'));
    const sgId = Fn.importValue('TempSg');

    const cluster = Cluster.fromClusterAttributes(this, 'ImportedCluster', {
      clusterName: 'TempCluster',
      clusterSecurityGroupId: sgId,
      vpc,
      openIdConnectProvider,
      kubectlLayer: new KubectlV25Layer(this, 'kubectl'),
      kubectlRoleArn: Fn.importValue('TempKubectlRole'),
    });

    cluster.addManifest("manifest", {
      apiVersion: 'v1',
      kind: 'Namespace',
      metadata: {
        name: 'temp-namespace',
      },
    })
  }
}

B 2.80.0 variant

export class KubetestStackB extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const vpc = Vpc.fromLookup(this, 'Vpc', { vpcName: 'KubetestStackA/TempCluster/DefaultVpc' });
    const openIdConnectProvider = OpenIdConnectProvider.fromOpenIdConnectProviderArn(this,'KubeOidcProvider', Fn.importValue('TempOidcProvider'));
    const sgId = Fn.importValue('TempSg');

    const kubectlProvider = KubectlProvider.fromKubectlProviderAttributes(
        this,
        'KubectlProvider',
        {
          functionArn: Fn.importValue('TempKubectlHandler'),
          handlerRole: Role.fromRoleArn(
              this,
              'HandlerRole',
              Fn.importValue('TempKubectlLambdaRole')
          ),
          kubectlRoleArn: Fn.importValue('TempKubectlRole'),
        }
    );

    const cluster = Cluster.fromClusterAttributes(this, 'ImportedCluster', {
      clusterName: 'TempCluster',
      clusterSecurityGroupId: sgId,
      vpc,
      openIdConnectProvider,
      kubectlLayer: new KubectlV25Layer(this, 'kubectl'),
      kubectlProvider: kubectlProvider,
    });

    cluster.addManifest("manifest", {
      apiVersion: 'v1',
      kind: 'Namespace',
      metadata: {
        name: 'temp-namespace',
        labels: {
          'test-label': 'test-value'
        }
      },
    })
  }
}

[pahud]

To me this seems to be because ImportedKubectlProvider uses the handler arn as the service token (

@okoskine Yes I think you are right. This might be the root cause. I am investigating this too. Will update here if I find anything and create a PR if necessary.