aws_volume_attachment Error waiting for Volume

[jrobison-sb]

Terraform Version

Terraform v0.11.7

• provider.aws v1.20.0

Affected Resource(s)

aws_volume_attachment

Actual Behavior

When running a terraform destroy, in which you want to destroy an aws_instance and it's aws_volume_attachment, terraform (or the AWS API) will attempt to destroy the volume attachment first, then then instance second. This often fails because Error waiting for Volume (vol-010b3d979b1027867) to detach from Instance.

Expected Behavior

If an aws_volume_attachment was going to be destroyed, and the associated aws_instance was also going to be destroyed, it would make sense to destroy the instance first and then the aws_volume_attachment second, which would never run any risk of timing out.

Steps to Reproduce

1. Create an aws_ebs_volume, an aws_instance, and a aws_volume_attachment.
2. terraform destroy

[YektaLeblebici]

I can confirm this for our infra too. On latest Terraform, destroying aws_volume_attachment and aws_instance together fails with this timeout error.

[alexaandru]

Running into this as well.

[Hashfyre]

➜ tfa
data.aws_availability_zones.az: Refreshing state...
data.aws_route53_zone.int-orgaws: Refreshing state...
data.aws_route53_zone.orgaws: Refreshing state...
data.aws_vpc.stage-org: Refreshing state...
data.aws_ami.stage-mongodb: Refreshing state...
data.aws_subnet.stage-mongodb: Refreshing state...
data.aws_security_group.stage-mongodb: Refreshing state...
aws_instance.stage-mongodb: Refreshing state... (ID: i-xxxxxxxxxxxxxxxxxxx)
aws_route53_record.stage-mongodb-int: Refreshing state... (ID: ACCID-stage-mongodb-int_A)
aws_ebs_volume.stage-mongodb-volume-a: Refreshing state... (ID: vol-xxxxxxxxxxx)
aws_volume_attachment.stage-mongodb-volume-attachment-a: Refreshing state... (ID: vai-xxxxxxxxx)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - aws_ebs_volume.stage-mongodb-volume-a

  - aws_instance.stage-mongodb

  - aws_route53_record.stage-mongodb-int

  - aws_volume_attachment.stage-mongodb-volume-attachment-a


Plan: 0 to add, 0 to change, 4 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_volume_attachment.stage-mongodb-volume-attachment-a: Destroying... (ID: vai-xxxxxxxxxxx)
aws_route53_record.stage-mongodb-int: Destroying... (ID: ACCID-stage-mongodb-int_A)
aws_volume_attachment.stage-mongodb-volume-attachment-a: Still destroying... (ID: vai-385500658, 10s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 10s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 20s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 30s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 40s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 50s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m0s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m10s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m20s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m30s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m40s elapsed)
aws_route53_record.stage-mongodb-int: Destruction complete after 1m40s

Error: Error applying plan:

1 error(s) occurred:

* aws_volume_attachment.stage-mongodb-volume-attachment-a (destroy): 1 error(s) occurred:

* aws_volume_attachment.stage-mongodb-volume-attachment-a: Error waiting for Volume (vol-xxxxxxxxxxxxx) to detach from Instance: i-xxxxxxxxxxxxx

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[queglay]

this is also a problem if ebs volume ids are replaced in the template, you'll get a similar error. if a volume can't be detached, it would be good to have a variable that did both these things-
shutdown instances if volumes can't be detached
startup instances after volume detachments and attachments are complete.

this would also help when doing terraform destroy.

I've attached an example below from changing the aws_volume_attachement name that causes the same problem.

Error: Error applying plan:

5 error(s) occurred:

• module.softnas.aws_volume_attachment.ebs_att[0] (destroy): 1 error(s) occurred:

• aws_volume_attachment.ebs_att.0: Error waiting for Volume (vol-0dfdb289f3a00f63c) to detach from Instance: i-055c1f1bd7d1b417c

• module.softnas.aws_cloudformation_stack.SoftNAS1Stack: 1 error(s) occurred:

• aws_cloudformation_stack.SoftNAS1Stack: Creating CloudFormation stack failed: AlreadyExistsException: Stack [FCB-SoftNAS1Stack] already exists
  status code: 400, request id: 303555c9-287f-11e9-b20c-4d54279ee980

• module.softnas.aws_volume_attachment.ebs_att[1] (destroy): 1 error(s) occurred:

• aws_volume_attachment.ebs_att.1: Error waiting for Volume (vol-071219d59b8153d6a) to detach from Instance: i-055c1f1bd7d1b417c

• module.softnas.aws_volume_attachment.ebs_att[3] (destroy): 1 error(s) occurred:

• aws_volume_attachment.ebs_att.3: Error waiting for Volume (vol-0c0703a9c333cef86) to detach from Instance: i-055c1f1bd7d1b417c

• module.softnas.aws_volume_attachment.ebs_att[2] (destroy): 1 error(s) occurred:

• aws_volume_attachment.ebs_att.2: Error waiting for Volume (vol-0a21cf466c807a753) to detach from Instance: i-055c1f1bd7d1b417c

[mikaelforeseeti]

I ran into this today, is there a workaround? I tried running terraform destroy --target=module.machines.ep_instance but it still tries to destroy the volume_attachment object first.

[paulrowlands]

FYI, I ran into this when implementing Windows servers with MSSQL.

I found that adding a null_resource which has a depends_on for the instance and a remote-exec provisioner which includes a 'when = "destroy"' option and executes a Powershell command stop the MSSQLSERVER service on the host.

After this, terraform destroy will destroy the attachment.

You could replace the MSSQL service with relevant shell commands to stop services, unmount disks, etc.

[yum-dev]

Even I'm facing this issue.

Terraform v0.12.1

• provider.aws v2.14.0

[DrHashi]

Terraform v0.12.3

• provider.aws v2.18.0

Still failing...

[kingindanord]

I have the same issue

Terraform v0.12.5

• provider.aws v2.20.0

[twarnock]

We are facing this issue as well.

Terraform v0.12.5
provider.aws v2.22

[AndreiF8]

Facing same issue

Terraform v0.11.14

• provider.aws v2.25.0

[pitthecat]

We have the same problem

Terraform 0.12.7 and provider AWS 2.25.0