net/http: go1.24.1 macOS without netgo (native) uses IPv6 instead of Dialcontext provided ipv4

[ldemailly]

Go version

go version go1.24.1 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/dl/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/dl/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/fq/gng4z4915mb73r9js_422h4c0000gn/T/go-build842198577=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/dl/github/fortio.org/multicurl/go.mod'
GOMODCACHE='/Users/dl/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/dl/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/dl/sdk/go1.24.1'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/Users/dl/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/dl/sdk/go1.24.1/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I have a debug service that is hosted on 3 different VMs

debug.fortio.org has address 192.9.227.83
debug.fortio.org has address 192.9.142.5
debug.fortio.org has address 18.222.136.83
debug.fortio.org has IPv6 address 2600:1f16:9c6:b400:282c:a766:6cab:4e82
debug.fortio.org has IPv6 address 2603:c024:c00a:d144:7cd0:4951:7106:96b8
debug.fortio.org has IPv6 address 2603:c024:c00a:d144:6663:5896:7efb:fbf3

And a tool that exercise each address - with go1.24.1 on macos 15.3.1 (24D70) with 10/10 https://test-ipv6.com
Despite using ipv4 from go, it connects using the same shared IPv6 address (thus connects to same server) and only for http (not https)

go1.24.1 run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org # bad

vs

go1.23.7 run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org # good

What did you see happen?

[...]
12:00:40.151 [VRB] 3: DialContext tcp debug.fortio.org:80 -> 18.222.136.83:80
12:00:40.162 [VRB] 3: DialContext 192.168.110.152:58297
12:00:40.174 [INF] 3: Status 200 "200 OK" from 18.222.136.83
⣻  364 b, 0s elapsed, 378.558 Mb/s
Φορτίο version 1.68.0 h1:2M9RuitiN+MgW6QlTJCHjW6PJLAPj/YCXegxSCLQ3rw= go1.23.6 arm64 linux (in fortio.org/proxy 1.21.0)
Debug server on oa1 up for 410h1m50.3s
Request from [2600:1700:1151:b24f:1d44:d45:bb10:ec89]:58297

Ie it connects to the wrong server using ipv6 instead of the specified ipv4

To reduce the output:

$ go1.24.1 run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on"
Debug server on ol1 up for 409h51m40.3s
Debug server on ol1 up for 409h51m40.4s
Debug server on ol1 up for 409h51m40.5s

3 times the same server

What did you expect to see?

Same as go1.23 (or -tags netgo)

$ go1.23.7 run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on"
Debug server on ol1 up for 409h52m36.8s
Debug server on oa1 up for 410h5m18.7s
Debug server on a1 up for 409h58m47.4s

3 different servers

[ldemailly]

Code used to select explicit IP is
https://github.com/fortio/multicurl/blob/v1.16.1/mc/mc.go#L326-L333
and works on linux, windows all go versions as well as macos with netgo or go before 1.24
(or macs without ipv6 connectivity)

I'm struggling to reduce further the repro

[ldemailly]

Same issue with as early as go1.24rc1 as well as gotip bc5f4a5
of note sometimes you get 2 different cached ip out of 3, but a second run is back to single cached ipv6

12:13:36 debug_go_1_24 multicurl$ gotip run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on"
Debug server on ol1 up for 410h2m29.8s
Debug server on oa1 up for 410h15m11.7s
Debug server on oa1 up for 410h15m11.7s
12:14:01 debug_go_1_24 multicurl$ gotip run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on"
Debug server on oa1 up for 410h15m17.2s
Debug server on oa1 up for 410h15m17.2s
Debug server on oa1 up for 410h15m17.2s

[gabyhelp]

Related Issues

• net: ipv4 chosen first when ipv6 is available #68795
• net: Dial does not respond to quickly-broken IPv6 connections by falling back to IPv4 #68237
• net/http/httptest: Server uses ipv4 instead of using both Ipv4 and Ipv6 #71178 (closed)
• x/net/quic: Endpoint.Dial prefers IPv4 to IPv6 #70223
• net: Error when running concurrent DNS queries using cgo on MacOS #30521 (closed)
• net: Setting LocalAddr to v4 address in net.Dialer not effective if v6 available #14672 (closed)
• x/net/http/httpproxy: no_proxy no longer matches an ip suffix when used as hostname #72123
• net.http: http client with dailContext send a https request different from client without dailContext #61136 (closed)
• net: "127.0.0.1:" should be an invalid tcp4 address for ResolveTCPAddr #56144 (closed)
• net: Panics in TCP/UDP with Go 1.23 due to IPv6 link-local zone mishandling #69397 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

You have a race with connection pooling.

https://pkg.go.dev/net/http#Transport

// DialContext runs concurrently with calls to RoundTrip.
// A RoundTrip call that initiates a dial may end up using
// a connection dialed previously when the earlier connection
// becomes idle before the later DialContext completes.

netgo only affects DNS resolution, not connection.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

[Snawoot]

@seankhliao Please kindly take a look at this issue once again. The problem is that net library somehow connects to IPv6 endpoint even in case when numeric IPv4 address specified to actual dialer. The remaining details of this issue suggest that it has something to do with native Mac OS resolving and such "connection coalescing" happens only if resolver was "asked" about domain having such IP addresses in question.

[ldemailly]

spent some time bisect'ing

0bb2183 works

13:06:08  src$ git checkout 0bb2183d45a1acec07470d9f041c55377217c8e2
Previous HEAD position was 6a4feb5644 cmd/link: on Mach-O, generate LC_UUID by default
HEAD is now at 0bb2183d45 cmd/internal/buildid: skip over Mach-O UUID from buildid computation
13:06:29  src$ time ./make.bash ; /Users/dl/dev/gosrc/go/bin/go run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on" ; /Users/dl/dev/gosrc/go/bin/go run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on"
Building Go cmd/dist using /opt/homebrew/Cellar/go/1.24.1/libexec. (go1.24.1 darwin/arm64)
Building Go toolchain1 using /opt/homebrew/Cellar/go/1.24.1/libexec.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/arm64.
---
Installed Go for darwin/arm64 in /Users/dl/dev/gosrc/go
Installed commands in /Users/dl/dev/gosrc/go/bin
*** You need to add /Users/dl/dev/gosrc/go/bin to your PATH.

real	0m36.199s
user	2m14.717s
sys	0m25.240s
Debug server on oa1 up for 411h8m23.6s
Debug server on ol1 up for 410h55m42s
Debug server on a1 up for 411h1m52.6s
Debug server on oa1 up for 411h8m25s
Debug server on ol1 up for 410h55m43.3s
Debug server on a1 up for 411h1m53.8s

6a4feb5 fails

though that commit, while related to darwin doesn't quite explain how it would cause this (@cherrymui any ideas?)

13:07:15  src$ git checkout 6a4feb56448aea1a0fe3485122d48fe7111958b1
Previous HEAD position was 0bb2183d45 cmd/internal/buildid: skip over Mach-O UUID from buildid computation
HEAD is now at 6a4feb5644 cmd/link: on Mach-O, generate LC_UUID by default
13:08:31  src$ time ./make.bash ; /Users/dl/dev/gosrc/go/bin/go run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on" ; /Users/dl/dev/gosrc/go/bin/go run fortio.org/multicurl@v1.16.1 -4 -loglevel debug http://debug.fortio.org 2> /dev/null | grep "server on"
Building Go cmd/dist using /opt/homebrew/Cellar/go/1.24.1/libexec. (go1.24.1 darwin/arm64)
Building Go toolchain1 using /opt/homebrew/Cellar/go/1.24.1/libexec.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/arm64.
---
Installed Go for darwin/arm64 in /Users/dl/dev/gosrc/go
Installed commands in /Users/dl/dev/gosrc/go/bin
*** You need to add /Users/dl/dev/gosrc/go/bin to your PATH.

real	0m35.564s
user	2m14.373s
sys	0m24.477s
Debug server on ol1 up for 410h57m40.2s
Debug server on ol1 up for 410h57m40.2s
Debug server on ol1 up for 410h57m40.2s
Debug server on ol1 up for 410h57m40.7s
Debug server on ol1 up for 410h57m40.8s
Debug server on ol1 up for 410h57m40.8s

[ldemailly]

@seankhliao

You have a race with connection pooling.

No connection to ipv6 is made yet it uses ipv6 even on the first ipv4 connection - also the code is single goroutine

Even making a single connection does unexpected ipv6:

$ go1.24.1 run . -n 1 -4 -loglevel verbose http://debug.fortio.org/
13:17:53.144 [INF] Log level is now 1 Verbose (was 2 Info)
13:17:53.144 [INF] Fortio multicurl dev  go1.24.1 arm64 darwin, using resolver ip4, GET http://debug.fortio.org/
13:17:53.144 [VRB] URLSchemeCheck "http://debug.fortio.org/"
13:17:53.144 [VRB] No port specified, using http
13:17:53.145 [VRB] Resolving ip4 host debug.fortio.org (port http -> 80)
13:17:53.147 [INF] Resolved ip4 debug.fortio.org:http to port 80 and 3 addresses [192.9.142.5 192.9.227.83 18.222.136.83] - keeping first 1
13:17:53.147 [VRB] 1: Using 192.9.142.5
13:17:53.147 [VRB] 1: DialContext tcp debug.fortio.org:80 -> 192.9.142.5:80
13:17:53.162 [VRB] 1: DialContext 192.168.110.152:58579
13:17:53.175 [INF] 1: Status 200 "200 OK" from 192.9.142.5
⣾  361 b, 0s elapsed, 1.007 Gb/s
Φορτίο version 1.68.0 h1:2M9RuitiN+MgW6QlTJCHjW6PJLAPj/YCXegxSCLQ3rw= go1.23.6 arm64 linux (in fortio.org/proxy 1.21.0)
Debug server on oa1 up for 411h19m3.4s
Request from [2600:1700:...my-client-ipv6...bb10:ec89]:58579
[...]

key parts:

13:17:53.147 [VRB] 1: DialContext tcp debug.fortio.org:80 -> 192.9.142.5:80
13:17:53.162 [VRB] 1: DialContext 192.168.110.152:58579
Request from [2600:1700:...my-client-ipv6...bb10:ec89]:58579

(wireshark also shows that switch and GODEBUG=multipathtcp=0 also doesn't help)

[ldemailly]

GODEBUG=netdns=go
also lets g1.24.1 work like 1.23(.7)

[seankhliao]

confirmed in slack that the program was using ipv4. the rewrite to ipv6 happens outside of go.

[ldemailly]

yes indeed yet it only happens with that linker change - somehow - curious how that transparent proxy like thing shows up up with that linker change

[ldemailly]

Mystery solved!

After finding that if I killed networkserviceproxy it would work normally again briefly until that auto restarted, and looking everywhere in my mac settings (hint; it’s not in network/proxy, I had none of that on) - I finally found I had a partially on iCloud+ feature, turning it off solved the problem - check this:

It's interesting that with that go linker change, it enables that feature to affect go binaries