asset publishing: docker image publishing to ECR can fail, and the cloudformation deploy proceeds anyway

[joelcox22]

Describe the bug

I found a case where the docker push to ECR failed, but CDK still attempted to deploy the cloudformation stack anyway.

Relevant section from my build logs (with project/account specific details redacted)

Successfully built a7709aca15c1
07:29:19  Successfully tagged cdkasset-a1db03a68e945daf6d8219c89a7b83695b4d0f10a3405a27fbf50b113701a601:latest
07:29:19  REDACTED:  success: Built a1db03a68e945daf6d8219c89a7b83695b4d0f10a3405a27fbf50b113701a601:REDACTED-ap-southeast-2
07:29:19  REDACTED:  start: Publishing a1db03a68e945daf6d8219c89a7b83695b4d0f10a3405a27fbf50b113701a601:REDACTED-ap-southeast-2
07:29:19  The push refers to repository [REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com/cdk-hnb659fds-container-assets-REDACTED-ap-southeast-2]
07:29:19  f9939a7e8903: Preparing
07:29:19  c2574e1d63a2: Preparing
07:29:19  33ace43f4309: Preparing
07:29:19  6850d95bfb09: Preparing
07:29:19  402fa4eeaa46: Preparing
07:29:19  bd7a8dceda87: Preparing
07:29:19  2b8f8a3dbcfd: Preparing
07:29:19  1988723c2a96: Preparing
07:29:19  bb84bf0418d7: Preparing
07:29:19  2e818a82a23f: Preparing
07:29:19  e09bb0b54218: Preparing
07:29:19  feef05f055c9: Preparing
07:29:19  bd7a8dceda87: Waiting
07:29:19  feef05f055c9: Waiting
07:29:19  e09bb0b54218: Waiting
07:29:19  bb84bf0418d7: Waiting
07:29:19  2b8f8a3dbcfd: Waiting
07:29:19  1988723c2a96: Waiting
07:29:19  33ace43f4309: Retrying in 5 seconds
07:29:19  6850d95bfb09: Retrying in 5 seconds
07:29:19  402fa4eeaa46: Retrying in 5 seconds
07:29:19  f9939a7e8903: Retrying in 5 seconds
07:29:19  c2574e1d63a2: Retrying in 5 seconds
07:29:20  33ace43f4309: Retrying in 4 seconds
07:29:20  6850d95bfb09: Retrying in 4 seconds
07:29:20  402fa4eeaa46: Retrying in 4 seconds
07:29:20  f9939a7e8903: Retrying in 4 seconds
07:29:20  c2574e1d63a2: Retrying in 4 seconds
07:29:21  33ace43f4309: Retrying in 3 seconds
07:29:21  6850d95bfb09: Retrying in 3 seconds
07:29:21  402fa4eeaa46: Retrying in 3 seconds
07:29:21  f9939a7e8903: Retrying in 3 seconds
07:29:21  c2574e1d63a2: Retrying in 3 seconds
07:29:22  33ace43f4309: Retrying in 2 seconds
07:29:22  6850d95bfb09: Retrying in 2 seconds
07:29:22  402fa4eeaa46: Retrying in 2 seconds
07:29:22  f9939a7e8903: Retrying in 2 seconds
07:29:22  c2574e1d63a2: Retrying in 2 seconds
07:29:23  33ace43f4309: Retrying in 1 second
07:29:23  6850d95bfb09: Retrying in 1 second
07:29:23  402fa4eeaa46: Retrying in 1 second
07:29:23  f9939a7e8903: Retrying in 1 second
07:29:23  c2574e1d63a2: Retrying in 1 second
07:29:24  402fa4eeaa46: Retrying in 10 seconds
07:29:24  6850d95bfb09: Retrying in 10 seconds
07:29:24  c2574e1d63a2: Retrying in 10 seconds
07:29:24  f9939a7e8903: Retrying in 10 seconds
07:29:24  33ace43f4309: Retrying in 10 seconds
07:29:25  402fa4eeaa46: Retrying in 9 seconds
07:29:25  6850d95bfb09: Retrying in 9 seconds
07:29:25  c2574e1d63a2: Retrying in 9 seconds
07:29:25  f9939a7e8903: Retrying in 9 seconds
07:29:25  33ace43f4309: Retrying in 9 seconds
07:29:26  402fa4eeaa46: Retrying in 8 seconds
07:29:26  6850d95bfb09: Retrying in 8 seconds
07:29:26  c2574e1d63a2: Retrying in 8 seconds
07:29:26  33ace43f4309: Retrying in 8 seconds
07:29:26  f9939a7e8903: Retrying in 8 seconds
07:29:27  402fa4eeaa46: Retrying in 7 seconds
07:29:27  6850d95bfb09: Retrying in 7 seconds
07:29:27  c2574e1d63a2: Retrying in 7 seconds
07:29:27  33ace43f4309: Retrying in 7 seconds
07:29:27  f9939a7e8903: Retrying in 7 seconds
07:29:28  402fa4eeaa46: Retrying in 6 seconds
07:29:28  6850d95bfb09: Retrying in 6 seconds
07:29:28  c2574e1d63a2: Retrying in 6 seconds
07:29:28  33ace43f4309: Retrying in 6 seconds
07:29:28  f9939a7e8903: Retrying in 6 seconds
07:29:29  402fa4eeaa46: Retrying in 5 seconds
07:29:29  6850d95bfb09: Retrying in 5 seconds
07:29:29  c2574e1d63a2: Retrying in 5 seconds
07:29:29  33ace43f4309: Retrying in 5 seconds
07:29:29  f9939a7e8903: Retrying in 5 seconds
07:29:30  402fa4eeaa46: Retrying in 4 seconds
07:29:30  6850d95bfb09: Retrying in 4 seconds
07:29:30  c2574e1d63a2: Retrying in 4 seconds
07:29:30  33ace43f4309: Retrying in 4 seconds
07:29:30  f9939a7e8903: Retrying in 4 seconds
07:29:31  402fa4eeaa46: Retrying in 3 seconds
07:29:31  6850d95bfb09: Retrying in 3 seconds
07:29:31  c2574e1d63a2: Retrying in 3 seconds
07:29:31  33ace43f4309: Retrying in 3 seconds
07:29:31  f9939a7e8903: Retrying in 3 seconds
07:29:32  402fa4eeaa46: Retrying in 2 seconds
07:29:32  6850d95bfb09: Retrying in 2 seconds
07:29:32  c2574e1d63a2: Retrying in 2 seconds
07:29:32  33ace43f4309: Retrying in 2 seconds
07:29:32  f9939a7e8903: Retrying in 2 seconds
07:29:33  402fa4eeaa46: Retrying in 1 second
07:29:33  6850d95bfb09: Retrying in 1 second
07:29:33  c2574e1d63a2: Retrying in 1 second
07:29:33  33ace43f4309: Retrying in 1 second
07:29:33  f9939a7e8903: Retrying in 1 second
07:29:34  f9939a7e8903: Retrying in 15 seconds
07:29:34  33ace43f4309: Retrying in 15 seconds
07:29:34  402fa4eeaa46: Retrying in 15 seconds
07:29:34  6850d95bfb09: Retrying in 15 seconds
07:29:34  c2574e1d63a2: Retrying in 15 seconds
07:29:35  f9939a7e8903: Retrying in 14 seconds
07:29:35  33ace43f4309: Retrying in 14 seconds
07:29:35  402fa4eeaa46: Retrying in 14 seconds
07:29:35  6850d95bfb09: Retrying in 14 seconds
07:29:35  c2574e1d63a2: Retrying in 14 seconds
07:29:36  f9939a7e8903: Retrying in 13 seconds
07:29:36  33ace43f4309: Retrying in 13 seconds
07:29:36  6850d95bfb09: Retrying in 13 seconds
07:29:36  402fa4eeaa46: Retrying in 13 seconds
07:29:36  c2574e1d63a2: Retrying in 13 seconds
07:29:37  f9939a7e8903: Retrying in 12 seconds
07:29:37  33ace43f4309: Retrying in 12 seconds
07:29:37  402fa4eeaa46: Retrying in 12 seconds
07:29:37  6850d95bfb09: Retrying in 12 seconds
07:29:37  c2574e1d63a2: Retrying in 12 seconds
07:29:38  f9939a7e8903: Retrying in 11 seconds
07:29:38  33ace43f4309: Retrying in 11 seconds
07:29:38  6850d95bfb09: Retrying in 11 seconds
07:29:38  402fa4eeaa46: Retrying in 11 seconds
07:29:38  c2574e1d63a2: Retrying in 11 seconds
07:29:39  f9939a7e8903: Retrying in 10 seconds
07:29:39  33ace43f4309: Retrying in 10 seconds
07:29:39  402fa4eeaa46: Retrying in 10 seconds
07:29:39  6850d95bfb09: Retrying in 10 seconds
07:29:39  c2574e1d63a2: Retrying in 10 seconds
07:29:40  f9939a7e8903: Retrying in 9 seconds
07:29:40  33ace43f4309: Retrying in 9 seconds
07:29:40  402fa4eeaa46: Retrying in 9 seconds
07:29:40  6850d95bfb09: Retrying in 9 seconds
07:29:40  c2574e1d63a2: Retrying in 9 seconds
07:29:41  f9939a7e8903: Retrying in 8 seconds
07:29:41  33ace43f4309: Retrying in 8 seconds
07:29:41  402fa4eeaa46: Retrying in 8 seconds
07:29:41  6850d95bfb09: Retrying in 8 seconds
07:29:41  c2574e1d63a2: Retrying in 8 seconds
07:29:42  f9939a7e8903: Retrying in 7 seconds
07:29:42  33ace43f4309: Retrying in 7 seconds
07:29:42  402fa4eeaa46: Retrying in 7 seconds
07:29:42  6850d95bfb09: Retrying in 7 seconds
07:29:42  c2574e1d63a2: Retrying in 7 seconds
07:29:43  f9939a7e8903: Retrying in 6 seconds
07:29:43  33ace43f4309: Retrying in 6 seconds
07:29:43  402fa4eeaa46: Retrying in 6 seconds
07:29:43  6850d95bfb09: Retrying in 6 seconds
07:29:43  c2574e1d63a2: Retrying in 6 seconds
07:29:44  f9939a7e8903: Retrying in 5 seconds
07:29:44  33ace43f4309: Retrying in 5 seconds
07:29:44  402fa4eeaa46: Retrying in 5 seconds
07:29:44  6850d95bfb09: Retrying in 5 seconds
07:29:44  c2574e1d63a2: Retrying in 5 seconds
07:29:45  f9939a7e8903: Retrying in 4 seconds
07:29:45  33ace43f4309: Retrying in 4 seconds
07:29:45  402fa4eeaa46: Retrying in 4 seconds
07:29:45  6850d95bfb09: Retrying in 4 seconds
07:29:45  c2574e1d63a2: Retrying in 4 seconds
07:29:46  f9939a7e8903: Retrying in 3 seconds
07:29:46  33ace43f4309: Retrying in 3 seconds
07:29:46  402fa4eeaa46: Retrying in 3 seconds
07:29:46  6850d95bfb09: Retrying in 3 seconds
07:29:46  c2574e1d63a2: Retrying in 3 seconds
07:29:47  f9939a7e8903: Retrying in 2 seconds
07:29:47  33ace43f4309: Retrying in 2 seconds
07:29:47  402fa4eeaa46: Retrying in 2 seconds
07:29:47  6850d95bfb09: Retrying in 2 seconds
07:29:47  c2574e1d63a2: Retrying in 2 seconds
07:29:48  f9939a7e8903: Retrying in 1 second
07:29:48  33ace43f4309: Retrying in 1 second
07:29:48  402fa4eeaa46: Retrying in 1 second
07:29:48  6850d95bfb09: Retrying in 1 second
07:29:48  c2574e1d63a2: Retrying in 1 second
07:29:49  f9939a7e8903: Retrying in 20 seconds
07:29:49  402fa4eeaa46: Retrying in 20 seconds
07:29:49  6850d95bfb09: Retrying in 20 seconds
07:29:49  33ace43f4309: Retrying in 20 seconds
07:29:49  c2574e1d63a2: Retrying in 20 seconds
07:29:50  f9939a7e8903: Retrying in 19 seconds
07:29:50  402fa4eeaa46: Retrying in 19 seconds
07:29:50  6850d95bfb09: Retrying in 19 seconds
07:29:50  33ace43f4309: Retrying in 19 seconds
07:29:50  c2574e1d63a2: Retrying in 19 seconds
07:29:51  f9939a7e8903: Retrying in 18 seconds
07:29:51  402fa4eeaa46: Retrying in 18 seconds
07:29:51  6850d95bfb09: Retrying in 18 seconds
07:29:51  c2574e1d63a2: Retrying in 18 seconds
07:29:51  33ace43f4309: Retrying in 18 seconds
07:29:52  f9939a7e8903: Retrying in 17 seconds
07:29:52  402fa4eeaa46: Retrying in 17 seconds
07:29:52  6850d95bfb09: Retrying in 17 seconds
07:29:52  33ace43f4309: Retrying in 17 seconds
07:29:52  c2574e1d63a2: Retrying in 17 seconds
07:29:53  f9939a7e8903: Retrying in 16 seconds
07:29:53  402fa4eeaa46: Retrying in 16 seconds
07:29:53  6850d95bfb09: Retrying in 16 seconds
07:29:53  33ace43f4309: Retrying in 16 seconds
07:29:53  c2574e1d63a2: Retrying in 16 seconds
07:29:54  f9939a7e8903: Retrying in 15 seconds
07:29:54  402fa4eeaa46: Retrying in 15 seconds
07:29:54  6850d95bfb09: Retrying in 15 seconds
07:29:54  33ace43f4309: Retrying in 15 seconds
07:29:54  c2574e1d63a2: Retrying in 15 seconds
07:29:55  f9939a7e8903: Retrying in 14 seconds
07:29:55  402fa4eeaa46: Retrying in 14 seconds
07:29:55  6850d95bfb09: Retrying in 14 seconds
07:29:55  33ace43f4309: Retrying in 14 seconds
07:29:55  c2574e1d63a2: Retrying in 14 seconds
07:29:56  f9939a7e8903: Retrying in 13 seconds
07:29:56  402fa4eeaa46: Retrying in 13 seconds
07:29:56  6850d95bfb09: Retrying in 13 seconds
07:29:56  33ace43f4309: Retrying in 13 seconds
07:29:56  c2574e1d63a2: Retrying in 13 seconds
07:29:57  f9939a7e8903: Retrying in 12 seconds
07:29:57  402fa4eeaa46: Retrying in 12 seconds
07:29:57  6850d95bfb09: Retrying in 12 seconds
07:29:57  33ace43f4309: Retrying in 12 seconds
07:29:57  c2574e1d63a2: Retrying in 12 seconds
07:29:58  f9939a7e8903: Retrying in 11 seconds
07:29:58  402fa4eeaa46: Retrying in 11 seconds
07:29:58  6850d95bfb09: Retrying in 11 seconds
07:29:58  33ace43f4309: Retrying in 11 seconds
07:29:58  c2574e1d63a2: Retrying in 11 seconds
07:29:59  f9939a7e8903: Retrying in 10 seconds
07:29:59  402fa4eeaa46: Retrying in 10 seconds
07:29:59  6850d95bfb09: Retrying in 10 seconds
07:29:59  33ace43f4309: Retrying in 10 seconds
07:29:59  c2574e1d63a2: Retrying in 10 seconds
07:30:00  f9939a7e8903: Retrying in 9 seconds
07:30:00  402fa4eeaa46: Retrying in 9 seconds
07:30:00  6850d95bfb09: Retrying in 9 seconds
07:30:00  33ace43f4309: Retrying in 9 seconds
07:30:00  c2574e1d63a2: Retrying in 9 seconds
07:30:01  f9939a7e8903: Retrying in 8 seconds
07:30:01  402fa4eeaa46: Retrying in 8 seconds
07:30:01  6850d95bfb09: Retrying in 8 seconds
07:30:01  33ace43f4309: Retrying in 8 seconds
07:30:01  c2574e1d63a2: Retrying in 8 seconds
07:30:02  f9939a7e8903: Retrying in 7 seconds
07:30:02  402fa4eeaa46: Retrying in 7 seconds
07:30:02  6850d95bfb09: Retrying in 7 seconds
07:30:02  33ace43f4309: Retrying in 7 seconds
07:30:02  c2574e1d63a2: Retrying in 7 seconds
07:30:03  f9939a7e8903: Retrying in 6 seconds
07:30:03  402fa4eeaa46: Retrying in 6 seconds
07:30:03  6850d95bfb09: Retrying in 6 seconds
07:30:03  33ace43f4309: Retrying in 6 seconds
07:30:03  c2574e1d63a2: Retrying in 6 seconds
07:30:04  f9939a7e8903: Retrying in 5 seconds
07:30:04  402fa4eeaa46: Retrying in 5 seconds
07:30:04  6850d95bfb09: Retrying in 5 seconds
07:30:04  33ace43f4309: Retrying in 5 seconds
07:30:04  c2574e1d63a2: Retrying in 5 seconds
07:30:05  f9939a7e8903: Retrying in 4 seconds
07:30:05  402fa4eeaa46: Retrying in 4 seconds
07:30:05  6850d95bfb09: Retrying in 4 seconds
07:30:05  33ace43f4309: Retrying in 4 seconds
07:30:05  c2574e1d63a2: Retrying in 4 seconds
07:30:06  f9939a7e8903: Retrying in 3 seconds
07:30:06  402fa4eeaa46: Retrying in 3 seconds
07:30:06  6850d95bfb09: Retrying in 3 seconds
07:30:06  33ace43f4309: Retrying in 3 seconds
07:30:06  c2574e1d63a2: Retrying in 3 seconds
07:30:07  f9939a7e8903: Retrying in 2 seconds
07:30:07  402fa4eeaa46: Retrying in 2 seconds
07:30:07  6850d95bfb09: Retrying in 2 seconds
07:30:07  33ace43f4309: Retrying in 2 seconds
07:30:07  c2574e1d63a2: Retrying in 2 seconds
07:30:08  f9939a7e8903: Retrying in 1 second
07:30:08  402fa4eeaa46: Retrying in 1 second
07:30:08  6850d95bfb09: Retrying in 1 second
07:30:08  33ace43f4309: Retrying in 1 second
07:30:08  c2574e1d63a2: Retrying in 1 second
07:30:09  2e818a82a23f: Waiting
07:30:09  EOF
07:30:09  
07:30:09  REDACTED:  fail: docker push REDACTED.dkr.ecr.ap-southeast-2.amazonaws.com/cdk-hnb659fds-container-assets-REDACTED-ap-southeast-2:a1db03a68e945daf6d8219c89a7b83695b4d0f10a3405a27fbf50b113701a601 exited with error code 1: EOF
07:30:09  
07:30:09  REDACTED: deploying... [1/1]
07:30:09  REDACTED: creating CloudFormation changeset...
07:30:28  REDACTED |  0/23 | 11:30:09 PM | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack                  | REDACTED User Initiated

Expected Behavior

I expect CDK should have failed with exit code 1 after the failed docker push, not continued anyway and attempted to deploy the cloudformation stack.

Current Behavior

CDK failed to push the asset container image to ECR, and then proceeded to deploy the cloudformation stack anyway.

Reproduction Steps

The ECR push failure was a one-off problem, so I can't reliably reproduce it - just logging this issue for awareness that some cases of asset publishing failures do not fail in the way I'd expect.

Possible Solution

Better error handling for docker image asset publishing failures.

Additional Information/Context

No response

CDK CLI Version

2.81.0

Framework Version

No response

Node.js Version

18

OS

Ubuntu

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

Thanks for the report. I am making it p2 bug and will look into this issue when we can reproduce it.

[tmokmss]

I also encountered this problem, and you can reproduce it just with an invalid Dockerfile.

CDK code:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';

export class DockerAssetFailStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new lambda.DockerImageFunction(this, 'Function', {
      code: lambda.DockerImageCode.fromImageAsset(__dirname),
    });
  }
}

and Dockerfile:

FROM alpine
# add an invalid file
ADD dummy ./

Then deployment continues even after the docker build error:

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref moby::kt7vrsavmcgftrxsf4dprq1wf: "/dummy": not found
DockerAssetFailStack:  start: Publishing f7668ad2c6b030a538e04703de7f92a0b1a958583dd689a32c2dc77af96257a1:current_account-current_region
The push refers to repository [xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/cdk-hnb659fds-container-assets-xxxxxx-ap-northeast-1]
An image does not exist locally with the tag: xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/cdk-hnb659fds-container-assets-xxxxxx-ap-northeast-1
DockerAssetFailStack:  fail: docker push xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/cdk-hnb659fds-container-assets-xxxxxx-ap-northeast-1:f7668ad2c6b030a538e04703de7f92a0b1a958583dd689a32c2dc77af96257a1 exited with error code 1: An image does not exist locally with the tag: xxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/cdk-hnb659fds-container-assets-xxxxxx-ap-northeast-1
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes

Also, I found that at least in CDK v2.64.0, this issue did not exist. I guess it was introduced somewhere between v2.64.0 - 2.81.0.

edit) I confirmed that it was introduced on v2.80.0.

[tmokmss]

pinging @rix0rrr as it may be relevant with #25536.

I guess we need another condition to check publisher.hasFailures somewhere during building/publishing assets, like below.

aws-cdk/packages/aws-cdk/lib/util/asset-publishing.ts

Lines 106 to 109 in 3bd973a

	await publisher.publish();
	if (publisher.hasFailures) {
	throw new Error('Failed to build one or more assets. See the error messages above for more information.');
	}