dial tcp: lookup context canceled

[rismoney]

Terraform Version

0.11.1

vSphere Provider Version

1.1.1

Affected Resource(s)

simple terraform run

Terraform Configuration Files

any TF code

Panic Output

11:21:45 * module.mybox.data.vsphere_network.network: 
11:21:45 data.vsphere_network.network: error fetching network: Post 
11:21:45 https://vc.mydom.com/sdk: dial tcp: lookup 
11:21:45 vc.mydom.com: context canceled
11:21:45 
11:21:45 * module.mybox.data.vsphere_virtual_machine.template: 1 error(s) 
11:21:45 occurred:
11:21:45 
11:21:45 
11:21:45 
11:21:45 * module.mybox.data.vsphere_virtual_machine.template: 
11:21:45 data.vsphere_virtual_machine.template: error fetching virtual machine: Post 
11:21:45 https://vc.mydom.com/sdk: dial tcp: lookup 
11:21:45 vc.mydom.com: context canceled

Expected Behavior

TF should have run successfully and connected to vsphere.

Actual Behavior

TF apply or plan failed. Subsequent runs may succeed usually after 1 additional run, sometimes 2.
Almost always fails first time.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply or terraform plan

[vancluever]

@rismoney are you on a VPN with split DNS or full DNS over the VPN for that matter?

This is more than likely a systemic issue with the Go DNS resolver and we are tracking it in hashicorp/terraform#3536.

Unfortunately I don't have much in the way of mitigation - this happens to me too during acceptance tests which is particularly frustrating. What I usually do is turn up my DNS resolver options to the maximum retry settings (I usually try options timeout:8 attempts:5 in my /etc/resolv.conf). This still doesn't seem to have an effect at the worst of times though, and I'm almost just debating adding static entries in my /etc/hosts file instead, which should sidestep the resolver problems altogether.

Sorry this is affecting you 😕

[rismoney]

no VPN in most of my use cases. (Have even tested in local segment adjacent to vsphere).
Do see it definitely on site to site VPN with EC2 servers. I'm not having DNS resolving issues that I can see outside of TF.

TF is running on windows in all these scenarios.

[rismoney]

so i switched the name of the vsphere server to ip address, and it seems to be working flawless. i think this confirms a DNS resolving issue, but I'm pretty certain it is within TF, and not the system.

[vancluever]

@rismoney that's what we are saying as well, more specifically it has do to with the Go resolver - and since you are on Windows it's even more complicated, kind of the same deal on OS X as well, since since both OSes have a very specific DNS resolver implementation that Go does not use (on Linux, the Go resolver at least pulls options from the right place - resolv.conf, but this is not used on Windows obviously, and not on OS X any more as well).

Unfortunately, I don't have a good solution for this (aside from using static IPs and hosts files directly) and I don't know when we will be fixing it in TF core. When those fixes happen, they will also be made to all providers as each provider runs in its own subprocess and will need to be told to use the different resolvers too (we have discussed this a bit internally). I can't provide an ETA though, as such I'm going to close this in favour of hashicorp/terraform#3536. Keep an eye out there if you want to follow the issue.

Sorry again!

[dhekimian]

@vancluever Have you tried GODEBUG=netdns=cgo as in GODEBUG=netdns=cgo terraform apply?

Instead of using the default go-lang dns resolver (netdns=go), it uses the linux built-in dns resolver that should behave as expected.

There is also some additional debugging by doing GODEBUG=netdns=cgo+2

Reference: https://golang.org/pkg/net/

[jason-azze]

@dhekimian I was just battling this problem, so tried. Unfortunately I just had another failure. My platform is Linux.

GODEBUG=netdns=cgo terraform apply "lt02-01"

...

Error: Error applying plan:

2 error(s) occurred:

* module.linux-test.vsphere_virtual_machine.linux[1]: 1 error(s) occurred:

* vsphere_virtual_machine.linux.1: Post https://myvsphereserver.com/sdk: dial tcp: lookup myvsphereserver.com on 10.1.3.11:53: dial udp 10.1.3.11:53: operation was canceled
* module.linux-test.vsphere_virtual_machine.linux[2]: 1 error(s) occurred:

* vsphere_virtual_machine.linux.2: Post https://myvsphereserver.com/sdk: dial tcp: lookup myvsphereserver.com on 10.1.3.11:53: dial udp 10.1.3.11:53: operation was canceled

[dhekimian]

All it takes is one DNS lookup failure to cause havoc on our builds. I was surprised at how many DNS lookups were required since GO doesn't have a dns cache.

After trying a handful of different GODEBUG values, I ended up adding my vcenter to /etc/hosts to bypass all the DNS lookups. I think its pretty safe to say that my vcenter IP isn't going to change that often.

Hopefully, this is fixed upstream in GO at some point... golang/go#22956

[akutz]

FWIW, I just this this as well:

module.lb.aws_lb_listener.xapi: Creation complete after 1s (ID: arn:aws:elasticloadbalancing:us-west-2:...0002/0b52eeae3f43b556/818a6713d95994a3)

Error: Error applying plan:

1 error(s) occurred:

* module.worker.vsphere_virtual_machine.worker: cannot locate virtual machine or template with UUID "42304aa9-699d-c9ab-02d9-55edffd89e07": Post https://vcenter.sddc-54-70-161-229.vmc.vmware.com/sdk: dial tcp: lookup vcenter.sddc-54-70-161-229.vmc.vmware.com on 8.8.8.8:53: no such host

It occurred after I'd been making changes to the TF files, so at first I thought I introduced a bug, but the Go error regarding the failed host lookup made me realize the provider simply failed to complete an API call due to error.

I too have switched to using the IP of the vCenter instead of the host name. Still, since IPs can change, it would be nice if this DNS issue was non-existent.