@aws-cdk: S3 buckets block stack deletion #26874
Description
Describe the bug
On stack deletion, buckets with RemovalPolicy.DESTROY often fail to delete due to continued writes to it, such as from zombie CloudFront distribution logging (the CloudFront distribution no longer exists in the console, but log data continues to populate the bucket rapidly).
This appears to be a resource accounting bug within AWS and not CDK-specific, but through the use of a DENY policy, CDK's existing auto-delete should be able to mitigate.
Expected Behavior
On stack deletion, the S3 buckets with RemovalPolicy.DESTROY should always be deleted.
Current Behavior
On stack deletion, the S3 buckets with RemovalPolicy.DESTROY often cannot be deleted after emptying due to a race condition with external writers, thereby blocking overall stack deletion.
Reproduction Steps
Errant behavior is non-deterministic.
- Using the AWS CloudFrontToS3 construct (@aws-solutions-constructs/aws-cloudfront-s3), deploy a stack.
- Delete the stack.
Possible Solution
The auto-delete lambda should add a policy to the target buckets to deny further PutObject calls prior to emptying the bucket.
Additional Information/Context
Defunct CloudFront distributions intermittently continue writing (often for more than 24 hours) to its log buckets even though the CloudFront distributions no longer appear in the AWS console, in the CLI, or in the stack resources. The zombie distributions have been observed to continue writing large numbers of small objects for days.
CDK CLI Version
2.93.0 (build 724bd01)
Framework Version
No response
Node.js Version
v18.17.1
OS
Amazon Linux 3.0 (ARM64)
Language
Typescript
Language Version
5.1.6
Other information
No response