cmd/go: running `go1.23.0 mod vendor` on go1.22.6 source unexpectedly vendors `golang.org/x/crypto/sha3`

[dagood]

Go version

go version go1.23.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dagood/.cache/go-build'
GOENV='/home/dagood/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dagood/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dagood/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/dagood/sdk/go1.23.0'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/home/dagood/sdk/go1.23.0/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/dagood/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/dagood/git/golang_go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build23445954=/tmp/go-build -gno-record-gcc-switches'

What did you do?

After checking out go1.22.6, in src:

$ go1.22.6 mod vendor
$ git diff
$ go1.23.0 mod vendor
$ git diff

(Used https://pkg.go.dev/golang.org/dl to download both Go versions.)

What did you see happen?

The first git diff has no output. This means go1.22.6's vendor directory is reproducible when I run Go 1.22.6.

The second git diff shows:

diff --git a/src/vendor/modules.txt b/src/vendor/modules.txt
index 9a234e59b1..ce8a7b8441 100644
--- a/src/vendor/modules.txt
+++ b/src/vendor/modules.txt
@@ -7,6 +7,7 @@ golang.org/x/crypto/cryptobyte/asn1
 golang.org/x/crypto/hkdf
 golang.org/x/crypto/internal/alias
 golang.org/x/crypto/internal/poly1305
+golang.org/x/crypto/sha3
 # golang.org/x/net v0.19.1-0.20240412193750-db050b07227e
 ## explicit; go 1.18
 golang.org/x/net/dns/dnsmessage

(The golang.org/x/crypto/sha3 source is also added into vendor/.)

What did you expect to see?

I expected reproducible results both times.

I filed this based on https://groups.google.com/g/golang-nuts/c/BEsmO16g37I, where I mentioned why this is what I expected:

I've been working under the assumption that backward compatibility promises would extend to "go mod vendor". This way, developers using 1.23.x and 1.22.x won't have thrashing changes when working on the same module.

[gabyhelp]

Related Issues and Documentation

• cmd/go: bad error message when using `vendor` directory while specifying too old Go version #67587 (closed)
• Vendor directory indirectly included in module ZIP due to including `vendor/modules.txt` #64440 (closed)
• Module Info (runtime.modinfo) prevents binary comparison #49573 (closed)
• go version -m vs go mod vendor #47935 (closed)
• all.bash build from source failed to import new vendors #56076 (closed)
• cmd/go: 'inconsistent vendoring' error when a user module is located within GOROOT/src #34657 (closed)
• cmd/go: record -mod=vendor in buildinfo #57782
• `go mod verify` should consider the vendor directory instead of only the modules cache #48420 (closed)
• cmd/go: build with vendor fails when go.mod doesn't record dependencies #47823 (closed)
• cmd/go: 'go build' in module mode rebuilds vendored dependencies in GOROOT #27285 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

CC @matloob @samthanawalla

[dagood]

GOROOT='/home/dagood/sdk/go1.23.0'

I tried changing GOROOT to point at the repository directory, and that seems to make go1.23.0 mod vendor reproducible. The golang.org/x/crypto/sha3 seems to be bleeding through from 1.23.0's GOROOT.

I suppose this might be reasonable: go mod treats the imports in the std module as imports of the go1.23.0 standard library because that's what GOROOT is pointing at, so go1.23.0 mod vendor finds those dependencies in that source and vendors them. If this is true, then maybe this is special with std and doesn't affect ordinary Go development.

I'm not sure how this is intended to work, but maybe it's fine if documented in README.vendor?

[FiloSottile]

I don't think a cmd/go built from a different tree and/or pointed at a different GOROOT is expected to work correctly on std or cmd.

[gopherbot]

Change https://go.dev/cl/616815 mentions this issue: README.vendor: add note about GOROOT, recommend fresh go

[dagood]

Submitted a CL to add a note, although its claims are weak because I don't know what's intended to be guaranteed here. 😄

[samthanawalla]

This behavior is expected. Different Go versions can sometimes resolve dependencies in slightly different ways (because of optimizations/tweaks) which can lead to variations in the packages that end up in your vendor directory.

The scope of the backwards compatibility promise is that code written for older versions of Go will continue to work on newer versions. Changes to vendoring should not break this promise. While striving for consistency is important, minor variations can still happen.

Thanks for the CL. Closing this out!