cmd/go: 'go mod download' pulls in test dependencies of dependencies

[prashantv]

What version of Go are you using (go version)?

$ go version
go version go1.15.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/prashant/.cache/go-build"
GOENV="/home/prashant/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/prashant/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/prashant/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/prashant/.gimme/versions/go1.15.2.linux.amd64"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/prashant/.gimme/versions/go1.15.2.linux.amd64/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/prashant/go/src/github.com/prashantv/go-mod-download-repro/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build327180583=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run go mod download for a module (called go-mod-download-repro in the example) that takes on a dependency on another module (go.uber.org/atomic in the example), which only has external dependencies in tests.

go.mod:

module github.com/prashantv/go-mod-download-repro

go 1.14

require go.uber.org/atomic v1.7.0

main.go:

package main

import (
        "fmt"

        "go.uber.org/atomic"
)

func main() {
        var i atomic.Int32
        i.Inc()
        fmt.Println(i.Load())
}

go.uber.org/atomic has test dependencies on testify.

Download to a local directory to see what is downloaded: $ GOMODCACHE=$(pwd)/cache go mod download

What did you expect to see?

Expected to see all dependencies of go-mod-download-repro to be downloaded, but not the test dependencies of go.uber.org/atomic.

What did you see instead?

Only dependenceis required to build/test go-mod-download-repro.

If this behaviour is intentional, would it be possible to add an option to limit the download to just packages needed to build, or to packages needed to test the current module?

[jayconrod]

cc @bcmills @matloob

This will be resolved by lazy module loading (#36460), but only when the main module and all of its dependencies have go 1.16 or later in their go.mod files.

It's expected behavior, but it does overfetch, and maybe we can improve it. Perhaps, in addition to #36460, we could narrow the scope of go mod download to just modules that provide packages in all (the new all in 1.16, i.e., packages needed to build packages in the main module, and tests of those packages).

[bcmills]

go mod download intentionally does not load packages: it examines only the module graph (both as implemented today, and with lazy module loading).

If you want to download only those modules that provide packages in all, you can run go list all with either -json or a suitable -f format string, which will list the corresponding modules. (go list all will download the relevant modules as a side-effect.)

[bcmills]

(And note that if your go.mod file specifies go 1.16 or higher, go list all already omits the dependencies of tests outside your module when using a go command built from head, such as golang.org/dl/gotip.)

[bcmills]

Finally, today you should be able to use go list -test -deps ./... to download only the modules transitively imported by the packages in your module.

[bcmills]

I believe this was resolved with #44435.

[bcmills]

Duplicate of #44435