`task` Create Bucket fails 409

[dacbd]

was taking a look at: iterative/blog-tpi-bees and my impatience resulted in an error... 🔥

git clone git@github.com:iterative/blog-tpi-bees.git
cd blog-tpi-bees/
vim main.tf
terraform init
terraform apply
# 3 minutes later I decide that is too long and Ctrl-C
terraform destroy
vim main.tf #change to a different region
terraform apply
# ERR

Relevant terraform outputs:

terraform init

dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding iterative/iterative versions matching ">= 0.9.9"...
- Installing iterative/iterative v0.10.9...
- Installed iterative/iterative v0.10.9 (self-signed, key ID 910A9EF2910B1E37)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

1st terraform apply

dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform apply
2022-04-26T16:23:06.850-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:23:06.881-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:06.881-0700
2022-04-26T16:23:06.905-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:23:06.937-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:06.937-0700
2022-04-26T16:23:06.958-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:06 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:23:06.958-0700
2022-04-26T16:23:06.961-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:23:06.993-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:06.993-0700
2022-04-26T16:23:07.016-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:07 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:23:07.016-0700

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # iterative_task.tpi-examples-basic will be created
  + resource "iterative_task" "tpi-examples-basic" {
      + addresses       = (known after apply)
      + cloud           = "aws"
      + disk_size       = 30
      + events          = (known after apply)
      + id              = (known after apply)
      + image           = "ubuntu"
      + logs            = (known after apply)
      + machine         = "t3.small"
      + name            = "tpi-examples-basic "
      + parallelism     = 1
      + region          = "us-east-2"
      + script          = <<-EOT
            #!/bin/bash
            sudo apt update
            sudo apt install -y python3-pip
            pip3 install -r requirements.txt
            python3 src/train.py 
        EOT
      + spot            = 0
      + ssh_private_key = (sensitive value)
      + ssh_public_key  = (sensitive value)
      + status          = (known after apply)
      + timeout         = 86400

      + storage {
          + output  = "shared"
          + workdir = "."
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

2022-04-26T16:23:10.644-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:23:10.678-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:10.678-0700
2022-04-26T16:23:10.702-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:23:10.702-0700
iterative_task.tpi-examples-basic: Creating...
2022-04-26T16:23:10.704-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "addresses" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700
2022-04-26T16:23:10.704-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "events" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700
2022-04-26T16:23:10.704-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "logs" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700
2022-04-26T16:23:10.704-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "status" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700
TPI [INFO] Creation may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait.
iterative_task.tpi-examples-basic: Still creating... [10s elapsed]
iterative_task.tpi-examples-basic: Still creating... [20s elapsed]
iterative_task.tpi-examples-basic: Still creating... [30s elapsed]
iterative_task.tpi-examples-basic: Still creating... [40s elapsed]
iterative_task.tpi-examples-basic: Still creating... [50s elapsed]
iterative_task.tpi-examples-basic: Still creating... [1m0s elapsed]
iterative_task.tpi-examples-basic: Still creating... [1m10s elapsed]
iterative_task.tpi-examples-basic: Still creating... [1m20s elapsed]
iterative_task.tpi-examples-basic: Still creating... [1m30s elapsed]
iterative_task.tpi-examples-basic: Still creating... [1m40s elapsed]
iterative_task.tpi-examples-basic: Still creating... [1m50s elapsed]
iterative_task.tpi-examples-basic: Still creating... [2m0s elapsed]
iterative_task.tpi-examples-basic: Still creating... [2m10s elapsed]
iterative_task.tpi-examples-basic: Still creating... [2m20s elapsed]
iterative_task.tpi-examples-basic: Still creating... [2m30s elapsed]
iterative_task.tpi-examples-basic: Still creating... [2m40s elapsed]
iterative_task.tpi-examples-basic: Still creating... [2m50s elapsed]
iterative_task.tpi-examples-basic: Still creating... [3m0s elapsed]
^CStopping operation...

Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...

2022-04-26T16:26:18.289-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_038.png: Failed to copy: RequestCanceled: request context canceled
caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700
2022-04-26T16:26:18.289-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_011.png: Failed to copy: RequestCanceled: request context canceled
caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700
2022-04-26T16:26:18.289-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_004.png: Failed to copy: RequestCanceled: request context canceled
caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700
2022-04-26T16:26:18.289-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_010.png: Failed to copy: RequestCanceled: request context canceled
caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700
╷
│ Error: execution halted
│ 
│ 
╵
╷
│ Error: execution halted
│ 
│ 
╵
╷
│ Error: RequestCanceled: request context canceled
│ caused by: context canceled
│ 
│   with iterative_task.tpi-examples-basic,
│   on main.tf line 5, in resource "iterative_task" "tpi-examples-basic":
│    5: resource "iterative_task" "tpi-examples-basic" {
│ 
╵
╷
│ Error: operation error Auto Scaling: DeleteAutoScalingGroup, https response error StatusCode: 0, RequestID: , canceled, context canceled
│ 
│   with iterative_task.tpi-examples-basic,
│   on main.tf line 5, in resource "iterative_task" "tpi-examples-basic":
│    5: resource "iterative_task" "tpi-examples-basic" {
│ 
╵

terraform destroy

dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform destroy
2022-04-26T16:26:23.544-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:26:23.577-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:23.577-0700
2022-04-26T16:26:23.602-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:26:23.633-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:23.633-0700
2022-04-26T16:26:23.655-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:23 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:26:23.655-0700
2022-04-26T16:26:23.658-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:26:23.690-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:23.690-0700
iterative_task.tpi-examples-basic: Refreshing state... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry]
TPI [INFO] Reading resources... (this may happen several times)                             
TPI [INFO] [1/9] Reading DefaultVPC...                                                      
TPI [INFO] [2/9] Reading DefaultVPCSubnet...                                                
TPI [INFO] [3/9] Reading Image...                                                           
TPI [INFO] [4/9] Reading Bucket...                                                          
TPI [INFO] [5/9] Reading SecurityGroup...                                                   
TPI [INFO] [6/9] Reading KeyPair...                                                         
TPI [INFO] [7/9] Reading Credentials...                                                     
TPI [INFO] [8/9] Reading LaunchTemplate...                                                  
TPI [INFO] [9/9] Reading AutoScalingGroup...                                                
TPI [INFO] Read completed                                                                   
TPI [INFO] Reading resources... (this may happen several times)                             
TPI [INFO] [1/9] Reading DefaultVPC...                                                      
TPI [INFO] [2/9] Reading DefaultVPCSubnet...                                                
TPI [INFO] [3/9] Reading Image...                                                           
TPI [INFO] [4/9] Reading Bucket...                                                          
TPI [INFO] [5/9] Reading SecurityGroup...                                                   
TPI [INFO] [6/9] Reading KeyPair...                                                         
TPI [INFO] [7/9] Reading Credentials...                                                     
TPI [INFO] [8/9] Reading LaunchTemplate...                                                  
TPI [INFO] [9/9] Reading AutoScalingGroup...                                                
TPI [INFO] Read completed                                                                   
TPI [INFO] Reading resources... (this may happen several times)                             
TPI [INFO] [1/9] Reading DefaultVPC...                                                      
TPI [INFO] [2/9] Reading DefaultVPCSubnet...                                                
TPI [INFO] [3/9] Reading Image...                                                           
TPI [INFO] [4/9] Reading Bucket...                                                          
TPI [INFO] [5/9] Reading SecurityGroup...                                                   
TPI [INFO] [6/9] Reading KeyPair...                                                         
TPI [INFO] [7/9] Reading Credentials...                                                     
TPI [INFO] [8/9] Reading LaunchTemplate...                                                  
TPI [INFO] [9/9] Reading AutoScalingGroup...                                                
TPI [INFO] Read completed                                                                   
TPI [INFO] aws t3.small in us-east-2                                                        
TPI [INFO]                                                                                  
TPI [INFO] Status: queued •                                                               
2022-04-26T16:26:33.846-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:33 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:26:33.846-0700
2022-04-26T16:26:33.854-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:26:33.893-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:33.893-0700

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # iterative_task.tpi-examples-basic will be destroyed
  - resource "iterative_task" "tpi-examples-basic" {
      - addresses       = [] -> null
      - cloud           = "aws" -> null
      - disk_size       = 30 -> null
      - events          = [] -> null
      - id              = "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry" -> null
      - image           = "ubuntu" -> null
      - logs            = [] -> null
      - machine         = "t3.small" -> null
      - name            = "tpi-examples-basic " -> null
      - parallelism     = 1 -> null
      - region          = "us-east-2" -> null
      - script          = <<-EOT
            #!/bin/bash
            sudo apt update
            sudo apt install -y python3-pip
            pip3 install -r requirements.txt
            python3 src/train.py 
        EOT -> null
      - spot            = 0 -> null
      - ssh_private_key = (sensitive value)
      - ssh_public_key  = (sensitive value)
      - status          = {
          - "running" = 0
        } -> null
      - timeout         = 86400 -> null

      - storage {
          - output  = "shared" -> null
          - workdir = "." -> null
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

2022-04-26T16:26:37.305-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:26:37.340-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:37.338-0700
iterative_task.tpi-examples-basic: Destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry]
TPI [INFO] Destruction may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait.
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 10s elapsed]
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 20s elapsed]
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 30s elapsed]
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 40s elapsed]
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 50s elapsed]
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 1m0s elapsed]
iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 1m10s elapsed]
iterative_task.tpi-examples-basic: Destruction complete after 1m14s

Destroy complete! Resources: 1 destroyed.

2nd terraform apply

dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform apply
2022-04-26T16:28:28.907-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:28:28.942-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:28.942-0700
2022-04-26T16:28:28.965-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:28:28.996-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:28.995-0700
2022-04-26T16:28:29.019-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:29 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:28:29.019-0700
2022-04-26T16:28:29.022-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:28:29.053-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:29.053-0700
2022-04-26T16:28:29.075-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:29 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:28:29.075-0700

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # iterative_task.tpi-examples-basic will be created
  + resource "iterative_task" "tpi-examples-basic" {
      + addresses       = (known after apply)
      + cloud           = "aws"
      + disk_size       = 30
      + events          = (known after apply)
      + id              = (known after apply)
      + image           = "ubuntu"
      + logs            = (known after apply)
      + machine         = "t3.small"
      + name            = "tpi-examples-basic "
      + parallelism     = 1
      + region          = "us-west"
      + script          = <<-EOT
            #!/bin/bash
            sudo apt update
            sudo apt install -y python3-pip
            pip3 install -r requirements.txt
            python3 src/train.py 
        EOT
      + spot            = 0
      + ssh_private_key = (sensitive value)
      + ssh_public_key  = (sensitive value)
      + status          = (known after apply)
      + timeout         = 86400

      + storage {
          + output  = "shared"
          + workdir = "."
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

2022-04-26T16:28:30.881-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:28:30.912-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:30.912-0700
2022-04-26T16:28:30.935-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:28:30.935-0700
iterative_task.tpi-examples-basic: Creating...
2022-04-26T16:28:30.939-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "events" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700
2022-04-26T16:28:30.939-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "addresses" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700
2022-04-26T16:28:30.939-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "status" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700
2022-04-26T16:28:30.939-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "logs" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700
TPI [INFO] Creation may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait.
╷
│ Error: operation error S3: CreateBucket, https response error StatusCode: 409, RequestID: 59A2YHE2FD93V5VE, HostID: ZhiCW18MkagKK0DmFj2pSZvrkLAANKr3KtFG28hidkuqXbSbe4lO0OUeBhNjxEae5Oh3zojGPts=, api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
│ 
│   with iterative_task.tpi-examples-basic,
│   on main.tf line 5, in resource "iterative_task" "tpi-examples-basic":
│    5: resource "iterative_task" "tpi-examples-basic" {
│ 
╵
╷
│ Error: failed to create
│ 
│   with iterative_task.tpi-examples-basic,
│   on main.tf line 5, in resource "iterative_task" "tpi-examples-basic":
│    5: resource "iterative_task" "tpi-examples-basic" {
│ 
╵

`TF_LOG_PROVIDER=TRACE terraform apply`

dabarnes@ubuntu:~/tests/blog-tpi-bees$ TF_LOG_PROVIDER=TRACE terraform apply
2022-04-26T16:35:20.724-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:35:20.731-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9]
2022-04-26T16:35:20.732-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55570
2022-04-26T16:35:20.732-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9
2022-04-26T16:35:20.756-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:20.756-0700
2022-04-26T16:35:20.764-0700 [DEBUG] provider: using plugin: version=5
2022-04-26T16:35:20.764-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin181832840 network=unix timestamp=2022-04-26T16:35:20.763-0700
2022-04-26T16:35:20.775-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:20.777-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-04-26T16:35:20.779-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55570
2022-04-26T16:35:20.779-0700 [DEBUG] provider: plugin exited
2022-04-26T16:35:20.780-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:35:20.787-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9]
2022-04-26T16:35:20.788-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55578
2022-04-26T16:35:20.788-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9
2022-04-26T16:35:20.812-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:20.811-0700
2022-04-26T16:35:20.820-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin2892143703 network=unix timestamp=2022-04-26T16:35:20.819-0700
2022-04-26T16:35:20.820-0700 [DEBUG] provider: using plugin: version=5
2022-04-26T16:35:20.831-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:20.835-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:20 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:35:20.835-0700
2022-04-26T16:35:20.836-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-04-26T16:35:20.838-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55578
2022-04-26T16:35:20.838-0700 [DEBUG] provider: plugin exited
2022-04-26T16:35:20.838-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:35:20.846-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9]
2022-04-26T16:35:20.846-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55587
2022-04-26T16:35:20.846-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9
2022-04-26T16:35:20.870-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:20.870-0700
2022-04-26T16:35:20.878-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin3554886002 network=unix timestamp=2022-04-26T16:35:20.878-0700
2022-04-26T16:35:20.878-0700 [DEBUG] provider: using plugin: version=5
2022-04-26T16:35:20.890-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:20.893-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:20 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:35:20.893-0700
2022-04-26T16:35:20.895-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-04-26T16:35:20.897-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55587
2022-04-26T16:35:20.897-0700 [DEBUG] provider: plugin exited

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # iterative_task.tpi-examples-basic will be created
  + resource "iterative_task" "tpi-examples-basic" {
      + addresses       = (known after apply)
      + cloud           = "aws"
      + disk_size       = 30
      + events          = (known after apply)
      + id              = (known after apply)
      + image           = "ubuntu"
      + logs            = (known after apply)
      + machine         = "t3.small"
      + name            = "tpi-examples-basic "
      + parallelism     = 1
      + region          = "us-west"
      + script          = <<-EOT
            #!/bin/bash
            sudo apt update
            sudo apt install -y python3-pip
            pip3 install -r requirements.txt
            python3 src/train.py 
        EOT
      + spot            = 0
      + ssh_private_key = (sensitive value)
      + ssh_public_key  = (sensitive value)
      + status          = (known after apply)
      + timeout         = 86400

      + storage {
          + output  = "shared"
          + workdir = "."
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

2022-04-26T16:35:24.512-0700 [INFO]  provider: configuring client automatic mTLS
2022-04-26T16:35:24.520-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9]
2022-04-26T16:35:24.520-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55612
2022-04-26T16:35:24.520-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9
2022-04-26T16:35:24.546-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:24.546-0700
2022-04-26T16:35:24.554-0700 [DEBUG] provider: using plugin: version=5
2022-04-26T16:35:24.554-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin1681772421 network=unix timestamp=2022-04-26T16:35:24.553-0700
2022-04-26T16:35:24.567-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:24.570-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:35:24.570-0700
iterative_task.tpi-examples-basic: Creating...
2022-04-26T16:35:24.572-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "events" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700
2022-04-26T16:35:24.572-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "logs" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700
2022-04-26T16:35:24.572-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "addresses" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700
2022-04-26T16:35:24.572-0700 [INFO]  provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "status" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700
TPI [INFO] Creation may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait.
2022-04-26T16:35:24.574-0700 [TRACE] provider.stdio: received data: channel=STDERR len=110
2022-04-26T16:35:24.574-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:24.857-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:24.857-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:24.936-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:24.936-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:25.481-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:25.481-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:25.988-0700 [TRACE] provider.stdio: received data: channel=STDERR len=220
2022-04-26T16:35:25.988-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:26.070-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:26.070-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:26.153-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:26.153-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:26.422-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:26.422-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:26.458-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:26.458-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:26.823-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:26.823-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:26.887-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:26.887-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:27.185-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:27.186-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:27.265-0700 [TRACE] provider.stdio: received data: channel=STDERR len=110
2022-04-26T16:35:27.265-0700 [TRACE] provider.stdio: waiting for stdio data
2022-04-26T16:35:27.305-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55
2022-04-26T16:35:27.305-0700 [TRACE] provider.stdio: waiting for stdio data
╷
│ Error: operation error S3: CreateBucket, https response error StatusCode: 409, RequestID: AHVV3JBFBP0KJNW2, HostID: aQNQT3SprqbnqOnO6BqftOJazOqhXLb2srjIDM7eZeiOR9zMQl8hBvws4yzZEhJ8WfOv9n3rtyw=, api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.
│ 
│   with iterative_task.tpi-examples-basic,
│   on main.tf line 5, in resource "iterative_task" "tpi-examples-basic":
│    5: resource "iterative_task" "tpi-examples-basic" {
│ 
╵
╷
│ Error: failed to create
│ 
│   with iterative_task.tpi-examples-basic,
│   on main.tf line 5, in resource "iterative_task" "tpi-examples-basic":
│    5: resource "iterative_task" "tpi-examples-basic" {
│ 
╵
2022-04-26T16:35:27.309-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-04-26T16:35:27.312-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55612
2022-04-26T16:35:27.312-0700 [DEBUG] provider: plugin exited

[0x2b3bfa0]

name - (Optional) Discouraged and may be removed in future - change the resource name instead, i.e. resource "iterative_task" "some_other_example_name". Deterministic task name (e.g. name="Hello, World!" always produces id="tpi-hello-world-5kz6ldls-57wo7rsp").

[0x2b3bfa0]

That's the “expected” behavior and there is nothing we can do to mitigate it. You can avoid it by not specifying the name field, so a new name is generated for every run.

[dacbd]

hmmm, but why the name a conflict for the create bucket operation?

AWS event:

  "eventTime": "2022-04-27T00:21:03Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "CreateBucket",
    "awsRegion": "us-west-1",
    "sourceIPAddress": "oops",
    "userAgent": "[aws-sdk-go-v2/1.11.0 os/linux lang/go/1.17.8 md/GOOS/linux md/GOARCH/amd64 api/s3/1.18.0]",
    "errorCode": "OperationAborted",
    "errorMessage": "A conflicting conditional operation is currently in progress against this resource. Please try again.",
    "requestParameters": {
        "CreateBucketConfiguration": {
            "LocationConstraint": "us-west-1",
            "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/"
        },
        "bucketName": "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry",
        "Host": "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry.s3.us-west-1.amazonaws.com"
    },
    "responseElements": null,
    "additionalEventData": {
        "SignatureVersion": "SigV4",
        "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "bytesTransferredIn": 153,
        "AuthenticationMethod": "AuthHeader",
        "x-amz-id-2": "d5ltZLOpYM0MCnDXf0cj/188ucBmIJW/IwMgeSUOyGlwHpt45y7hlTVEgWLIWlz1HIoD292Ztwc=",
        "bytesTransferredOut": 335
    },
    "requestID": "89DCAM5G4V7HMDB9",
    "eventID": "e62864f9-84dd-4df8-afaa-4cd7e92f19f2",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "548501963152",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry.s3.us-west-1.amazonaws.com"
    }
}

[0x2b3bfa0]

why the name a conflict for the create bucket operation?

Because the old & new buckets had exactly the same name.

[0x2b3bfa0]

You were trying to create a new bucket while the old one was still queued for deletion, and both had the same name.

[dacbd]

I see, and that takes time on the AWS for them to fully complete, and thus the error.