[Bug]: Bucket lifecycle rules get dangerously mangled when earlier entries are deleted

[purajit]

Terraform Core Version

1.11.1

AWS Provider Version

5.90.0

Affected Resource(s)

• aws_s3_bucket_lifecycle_configuration

Expected Behavior

This problem definitely does not exist of 5.80.0, but does on 5.90.0. I haven't bisected to check the middle versions. I believe it is related to all the default value changes recently

Let's say you start with the follow configuration:

resource "aws_s3_bucket" "this" {
  bucket = "test-<name>-lifecycle-policy-20250312"
}

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    id     = "del asdf1"
    status = "Enabled"

    filter {  # the issue happens even without prefix, keeping it to show how disastrous the result is
      prefix = "asdf1"
    }
    expiration {
      days = 1
    }
  }
  
  rule {
    id     = "expire delete markers"
    status = "Enabled"

    expiration {
      expired_object_delete_marker = true
    }
  }
}

Now, let's remove the first rule, so that the lifecycle configuration looks like

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    id     = "expire delete markers"
    status = "Enabled"

    expiration {
      expired_object_delete_marker = true
    }
  }
}

Here's the output of the apply:

  # aws_s3_bucket_lifecycle_configuration.this will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "this" {
        id                                     = "test-purj-lifecycle-policy-20250312"
        # (3 unchanged attributes hidden)

      ~ rule {
          ~ id     = "del asdf1" -> "expire delete markers"
          + prefix = (known after apply)
            # (1 unchanged attribute hidden)

          ~ expiration {
              ~ expired_object_delete_marker = false -> true
                # (1 unchanged attribute hidden)
            }

          - filter {
              - prefix = "asdf1" -> null
            }
        }
      - rule {
          - id     = "expire delete markers" -> null
          - status = "Enabled" -> null
            # (1 unchanged attribute hidden)

          - expiration {
              - days                         = 0 -> null
              - expired_object_delete_marker = true -> null
            }
        }
    }

This is a bad change - notice that rule[0] has not been fully cleared out - notably, the expiration.days entry should be nullified (set to 0), but it isn't, meaning the final rule is a garbled combination of the two rules. Applying results in an error, but only after changes have already been made:

╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to aws_s3_bucket_lifecycle_configuration.this, provider
│ "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value:
│ .rule[0].expiration[0].expired_object_delete_marker: was cty.True, but now cty.False.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Indeed, checking the UI, I see

before

after

This is an incredibly dangerous change - essentially, we initially set an expiration of 1 day for all objects under a prefix, and now, without asking for it, we've unexpectedly set an expiration of 1 day for the entire bucket, and on top of that, we've completely undone the Delete expired object markers rule, even though we'd expect to have it based on the terraform definition and the rule name. This happened to two of our buckets, and we had to do a sprawling data recovery.

Actual Behavior

Rules should resemble the final state we ask for. Here is the plan from provider version 5.80.0, which is correct (though ultimately having rules blocks indexed by id rather than array index would be ideal for diffs, if possible):

  # aws_s3_bucket_lifecycle_configuration.this will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "this" {
        id                                     = "test-purj-lifecycle-policy-20250312"
        # (3 unchanged attributes hidden)

      ~ rule {
          ~ id     = "del asdf1" -> "expire delete markers"
            # (2 unchanged attributes hidden)

          ~ expiration {
              ~ days                         = 1 -> 0
              ~ expired_object_delete_marker = false -> true
                # (1 unchanged attribute hidden)
            }

          ~ filter {
              - prefix                   = "asdf1" -> null
                # (2 unchanged attributes hidden)
            }
        }
      - rule {
          - id     = "expire delete markers" -> null
          - status = "Enabled" -> null
            # (1 unchanged attribute hidden)

          - expiration {
              - days                         = 0 -> null
              - expired_object_delete_marker = true -> null
                # (1 unchanged attribute hidden)
            }

          - filter {
                # (3 unchanged attributes hidden)
            }
        }
    }

Steps to Reproduce

terraform apply
# remove aws_s3_bucket_lifecycle_configuration.this.rules[0]
terraform apply

Would you like to implement a fix?

None

[github-actions]

Community Guidelines

This comment is added to every new Issue to provide quick reference to how the Terraform AWS Provider is maintained. Please review the information below, and thank you for contributing to the community that keeps the provider thriving! 🚀

Voting for Prioritization

• Please vote on this Issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize it.
• Please see our prioritization guide for additional information on how the maintainers handle prioritization.
• Please do not leave +1 or other comments that do not add relevant new information or questions; they generate extra noise for others following the Issue and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.

[github-actions]

Warning

This Issue has been closed, meaning that any additional comments are much easier for the maintainers to miss. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.95.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.