[Bug]: aws_s3_bucket_lifecycle_configuration "couldn't find resource" since 5.86

[Dave-EMIS]

Terraform Core Version

1.10

AWS Provider Version

5.86.0

Affected Resource(s)

• aws_s3_bucket_lifecycle_configuration
• aws_s3_bucket

Expected Behavior

When creating an S3 bucket and lifecycle configuration policy there should be no error

Actual Behavior

Getting:

│ Error: creating S3 Bucket (my-bucket) Lifecycle Configuration
│ 
│   with module.bucket.aws_s3_bucket_lifecycle_configuration.xxx,
│   on ../modules/bucket/main.tf line 93, in resource "aws_s3_bucket_lifecycle_configuration" "xxx":
│   93: resource "aws_s3_bucket_lifecycle_configuration" "xxx" {
│ 
│ couldn't find resource

Relevant Error/Panic Output Snippet

Terraform Configuration Files

I've had failures with 2 different lifecylce configurations, I've shown them both here, but I don't think the rule's are the problem

resource "aws_s3_bucket" "xxx" {
  bucket = "my-bucket"
  tags = {
    Name = "name"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "xxx" {
  bucket = aws_s3_bucket.xxx.id

  rule {
    id = "Retain last 30 days"
    # Expires after 30 days
    expiration {
      days = 30
    }
    # Deleted 1 day after expiration
    noncurrent_version_expiration {
      noncurrent_days = 1
    }
    status = "Enabled"
  }
}

# OR

resource "aws_s3_bucket_lifecycle_configuration" "xxx" {
  bucket = aws_s3_bucket.xxx.id

  rule {
    id = "Archive to Glacier Deep Archive"
    transition {
      days          = 30
      storage_class = "DEEP_ARCHIVE"
    }
    status = "Enabled"
  }
}

Steps to Reproduce

⬇️ ⬇️ See my comment below with a minimal repo ⬇️ ⬇️

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @Dave-EMIS 👋 Thank you for taking the time to raise this! To clarify, are you using two aws_s3_bucket_lifecycle_configuration resources with the same S3 bucket, or were you more trying to provide two different examples in your sample configuration? If you are, that's not a supported configuration per the documentation, and may be part of the problem here.

Given that it sounds like this issue is fairly transient, are you able to reproduce with debug logging enabled and supply those logs (redacted as needed)? That information may be necessary in case whoever picks this up can't reproduce the issue.

[Dave-EMIS]

Hey @Dave-EMIS 👋 Thank you for taking the time to raise this! To clarify, are you using two aws_s3_bucket_lifecycle_configuration resources with the same S3 bucket, or were you more trying to provide two different examples in your sample configuration? If you are, that's not a supported configuration per the documentation, and may be part of the problem here.

Given that it sounds like this issue is fairly transient, are you able to reproduce with debug logging enabled and supply those logs (redacted as needed)? That information may be necessary in case whoever picks this up can't reproduce the issue.

Hi, no, there is only 1 configuration per bucket.
The 2 examples above are for different 2 buckets, but I just showed that I have the problem with different configurations.

[justinretzolk]

Thanks for the follow up here @Dave-EMIS, and apologies that I missed the pretty clear # OR in the example configuration 🤦. Are you able to supply debug logging around this? I tried to reproduce by creating and destroying 40 S3 buckets and associated lifecycles configurations repeatedly, and haven't yet been able to do so.

[Dave-EMIS]

It has only happened so far on GH actions, I'll do my best to repro it and see what I find.

[Dave-EMIS]

Hi @justinretzolk - I have recreated with the debug logs. I don't want to upload the entire 24MB into public (and keybase is blocked at work).

Looking around the error I see that there's a aws_s3_bucket_notification on both of the buckets I have seen this with.

The way I have it structured is:

| Modules
  |- S3 Bucket1 + Lifecycle1
  |- Lambda1 + Notification1
  |- S3 Bucket2 + Lifecycle2
  |- Lambda2 + Notification2

I don't think the fact that I am doing this twice is relevant, but I just wanted to make it clear, both Lifecycle Policies I am having trouble with are separate from each other, and it isn't just one big S3 or Lambda module.

Here's hopefully the relevant section of the logs:

2025-02-14T09:34:43.663Z [DEBUG] provider.terraform-provider-aws_v5.86.0_x5: HTTP Response Received: http.response.header.date="Fri, 14 Feb 2025 09:34:44 GMT" http.response.header.x_amz_request_id=3VPEBGD6FTSFQGHM rpc.service=S3 @module=aws http.duration=55 http.response.header.x_amz_id_2="NjusAnrbgMn6Izwwf6Tlq4oGlf/P2/OFbT+2orGhyadHK77vqmKj60Jzbro2ccXADSp0gwqO07c=" tf_aws.sdk=aws-sdk-go-v2 tf_provider_addr=registry.terraform.io/hashicorp/aws @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.62/logging/tf_logger.go:45
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CloudFunctionConfiguration><Id>tf-s3-lambda-20250214093443284000000005</Id><CloudFunction>arn:aws:lambda:eu-west-2:123456:function:dev-dave-s3-xxx-log-aggregator</CloudFunction><Event>s3:ObjectCreated:*</Event></CloudFunctionConfiguration></NotificationConfiguration>
   http.response.header.server=AmazonS3 rpc.method=GetBucketNotificationConfiguration tf_aws.signing_region="" tf_resource_type=aws_s3_bucket_notification aws.region=eu-west-2 http.status_code=200 rpc.system=aws-api tf_mux_provider="*schema.GRPCProviderServer" tf_req_id=01867e0c-b9d1-87b0-6760-4f0bc99fa949 tf_rpc=ApplyResourceChange timestamp=2025-02-14T09:34:43.663Z
2025-02-14T09:34:43.664Z [DEBUG] provider.terraform-provider-aws_v5.86.0_x5: HTTP Request Sent: @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.62/logging/tf_logger.go:45 http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 net.peer.name=dev-dave-s3-xxx-logging-raw.s3.eu-west-2.amazonaws.com tf_aws.signing_region="" http.request.header.amz_sdk_invocation_id=7f5fe3ed-5d00-4ac5-a3d6-e20d7924784d tf_mux_provider="*schema.GRPCProviderServer" tf_provider_addr=registry.terraform.io/hashicorp/aws http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************VHJP/20250214/eu-west-2/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_date=20250214T093443Z http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.10.5 (+https://www.terraform.io) terraform-provider-aws/5.86.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.23.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.75.3" tf_aws.sdk=aws-sdk-go-v2 @module=aws aws.region=eu-west-2 http.method=GET rpc.service=S3 rpc.system=aws-api tf_resource_type=aws_s3_bucket_notification tf_req_id=01867e0c-b9d1-87b0-6760-4f0bc99fa949 http.url=https://dev-dave-s3-xxx-logging-raw.s3.eu-west-2.amazonaws.com/?notification= tf_rpc=ApplyResourceChange http.request.body="" http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=25" http.request.header.x_amz_security_token="*****" rpc.method=GetBucketNotificationConfiguration timestamp=2025-02-14T09:34:43.663Z
2025-02-14T09:34:43.702Z [DEBUG] provider.terraform-provider-aws_v5.86.0_x5: HTTP Response Received: rpc.method=GetBucketLifecycleConfiguration tf_req_id=9e9f72fb-ccd8-e05a-b8d9-b2667448e82c @module=aws http.response.header.server=AmazonS3 http.response.header.x_amz_id_2=PYWPhf6Q0YP9J2rFYMXiTbGfGcIDKdbliD3hekkY5xrff0qYyZ7ZJQaoKLj451Hb8YNG5YbONT0= tf_aws.signing_region="" tf_rpc=ApplyResourceChange
  http.response.body=
  | <Error><Code>NoSuchLifecycleConfiguration</Code><Message>The lifecycle configuration does not exist</Message><BucketName>dev-dave-s3-xxx-lb-access-logs</BucketName><RequestId>3VP69YQ58ZRRMAMW</RequestId><HostId>PYWPhf6Q0YP9J2rFYMXiTbGfGcIDKdbliD3hekkY5xrff0qYyZ7ZJQaoKLj451Hb8YNG5YbONT0=</HostId></Error>
   http.response.header.date="Fri, 14 Feb 2025 09:34:44 GMT" aws.region=eu-west-2 http.duration=61 http.response_content_length=308 rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_s3_bucket_lifecycle_configuration http.response.header.content_type=application/xml http.response.header.x_amz_request_id=3VP69YQ58ZRRMAMW http.status_code=404 tf_mux_provider="*proto5server.Server" @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.62/logging/tf_logger.go:45 timestamp=2025-02-14T09:34:43.702Z
2025-02-14T09:34:43.703Z [DEBUG] provider.terraform-provider-aws_v5.86.0_x5: request failed with unretryable error https response error StatusCode: 404, RequestID: 3VP69YQ58ZRRMAMW, HostID: PYWPhf6Q0YP9J2rFYMXiTbGfGcIDKdbliD3hekkY5xrff0qYyZ7ZJQaoKLj451Hb8YNG5YbONT0=, api error NoSuchLifecycleConfiguration: The lifecycle configuration does not exist: aws.region=eu-west-2 rpc.service=S3 tf_provider_addr=registry.terraform.io/hashicorp/aws rpc.method=GetBucketLifecycleConfiguration rpc.system=aws-api @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.62/logging/tf_logger.go:45 tf_mux_provider="*proto5server.Server" tf_req_id=9e9f72fb-ccd8-e05a-b8d9-b2667448e82c tf_resource_type=aws_s3_bucket_lifecycle_configuration @module=aws tf_aws.sdk=aws-sdk-go-v2 tf_rpc=ApplyResourceChange timestamp=2025-02-14T09:34:43.702Z
2025-02-14T09:34:43.703Z [ERROR] provider.terraform-provider-aws_v5.86.0_x5: Response contains error diagnostic: diagnostic_detail="couldn't find resource" diagnostic_severity=ERROR diagnostic_summary="creating S3 Bucket (dev-dave-s3-xxx-lb-access-logs) Lifecycle Configuration" tf_proto_version=5.8 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=9e9f72fb-ccd8-e05a-b8d9-b2667448e82c tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.26.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto tf_resource_type=aws_s3_bucket_lifecycle_configuration timestamp=2025-02-14T09:34:43.703Z
2025-02-14T09:34:43.704Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2025-02-14T09:34:43.704Z [ERROR] vertex "module.alb.aws_s3_bucket_lifecycle_configuration.lb_access_logs" error: creating S3 Bucket (dev-dave-s3-xxx-lb-access-logs) Lifecycle Configuration
2025-02-14T09:34:43.711Z [DEBUG] provider.terraform-provider-aws_v5.86.0_x5: HTTP Response Received: @module=aws http.response.header.date="Fri, 14 Feb 2025 09:34:44 GMT" http.status_code=200 @caller=github.com/hashicorp/aws-sdk-go-base/v2@v2.0.0-beta.62/logging/tf_logger.go:45 http.duration=47
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CloudFunctionConfiguration><Id>tf-s3-lambda-20250214093443284000000005</Id><CloudFunction>arn:aws:lambda:eu-west-2:123456:function:dev-dave-s3-xxx-log-aggregator</CloudFunction><Event>s3:ObjectCreated:*</Event></CloudFunctionConfiguration></NotificationConfiguration>
   http.response.header.server=AmazonS3 tf_aws.signing_region="" tf_aws.sdk=aws-sdk-go-v2 tf_mux_provider="*schema.GRPCProviderServer" tf_resource_type=aws_s3_bucket_notification aws.region=eu-west-2 http.response.header.x_amz_id_2="BkmUjjO1MBsIZp+O7M7TU1jd/snft1mCqnX3ut23iKunUjA+sxvhBVYHp04IyJarmKdng+b8n5Y=" rpc.system=aws-api tf_req_id=01867e0c-b9d1-87b0-6760-4f0bc99fa949 tf_rpc=ApplyResourceChange http.response.header.x_amz_request_id=3VP4JNVPY42TQ963 rpc.method=GetBucketNotificationConfiguration rpc.service=S3 tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2025-02-14T09:34:43.711Z
2025-02-14T09:34:43.712Z [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value for module.log_aggregator_lambda.aws_s3_bucket_notification.bucket_notification, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .lambda_function[0].filter_prefix: was null, but now cty.StringVal("")
      - .lambda_function[0].filter_suffix: was null, but now cty.StringVal("")
2025-02-14T09:34:43.713Z [DEBUG] State storage *remote.State declined to persist a state snapshot

If you need more info, please reach out and I'll work around the issues blocking a full upload :)

[Dave-EMIS]

OK, I have a minimal repro

repro.tar.gz

Run this a few times with:

terraform destroy -auto-approve && \
  rm .terraform -rf && \
  rm tf.log && \
  rm terraform.tfstate && \
  terraform init && \
  terraform plan && \
  terraform apply -auto-approve

[justinretzolk]

Thanks for the follow up here @Dave-EMIS 👍. I tried to reproduce using your minimal reproduction and still haven't been able to. I'm going to let someone from the team pick this up to look at it before I ask you to do anything else regarding debug logging, since it seems like it'd be a bit of a hassle.