cmd/go: -test.gocoverdir cannot be used for non-curdir packages

[cyphar]

Go version

go version go1.24.3 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/cyphar/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/cyphar/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build213853727=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/cyphar/.local/src/github.com/opencontainers/umoci/go.mod'
GOMODCACHE='/home/cyphar/.local/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/cyphar/.local'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib64/go/1.24'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/cyphar/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib64/go/1.24/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.3'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I'm trying to migrate an existing project that uses a bunch of hacks to get coverage from integration and unit tests to using go build -cover/GOCOVERDIR=foo and go test -cover -test.gocoverdir=foo instead. However, it turns out that go test ./... behaviour changes with -test.gocoverdir=foo.

You can verify this in any project by just observing the output difference between go test -cover ./... and go test -cover -test.gocoverdir=foo ./..., but here's an example:

% mkdir -p gotest/{a,b,c,d,e,f,g}
% cd gotest
% go mod init gotest
go: creating new go.mod: module gotest
go: to add module requirements and sums:
        go mod tidy
% for dir in */; do echo "package ${dir%/}" >"$dir/doc.go"; done
% echo "package main" > main.go
% go test -cover ./...
?       gotest  [no test files]
?       gotest/a        [no test files]
?       gotest/b        [no test files]
?       gotest/c        [no test files]
?       gotest/d        [no test files]
?       gotest/e        [no test files]
?       gotest/f        [no test files]
?       gotest/g        [no test files]
% go test -cover -test.gocoverdir=foo ./...
?       gotest  [no test files]

As far as I can tell, -test.gocoverdir has had this behaviour since at least Go 1.14 (though I believe -test.gocoverdir was an implementation detail that wasn't widely used until GOCOVERDIR and go build -cover support were added in Go 1.20).

EDIT: Upon some further testing, it turns out that even naming subpackages doesn't work:

% go test -cover ./a ./b ./c ./d ./e ./f ./g
?       gotest/a        [no test files]
?       gotest/b        [no test files]
?       gotest/c        [no test files]
?       gotest/d        [no test files]
?       gotest/e        [no test files]
?       gotest/f        [no test files]
?       gotest/g        [no test files]
% go test -cover -test.gocoverdir=foo ./a ./b ./c ./d ./e ./f ./g
?       gotest  [no test files]

Even if you are only running a single test package:

% go test -cover ./a
?       gotest/a        [no test files]
% go test -cover -test.gocoverdir=foo ./a
?       gotest  [no test files]

What did you see happen?

Only the current directory had its tests run:

% go test -cover -test.gocoverdir=foo ./...
?       gotest  [no test files]

-coverpkg=./... doesn't affect the output.

What did you expect to see?

You would expect ./... to recursively run the tests like regular go test does:

% go test -cover -test.gocoverdir=foo ./...
?       gotest  [no test files]
?       gotest/a        [no test files]
?       gotest/b        [no test files]
?       gotest/c        [no test files]
?       gotest/d        [no test files]
?       gotest/e        [no test files]
?       gotest/f        [no test files]
?       gotest/g        [no test files]

... or at least return an error if the behaviour is intentional. And named test packages should absolutely have their tests run.

Silently not running tests is quite concerning behaviour, and I only noticed this because I have a minimum coverage percentage for my project that was broken by switching to -test.gocoverdir=.... I wouldn't have noticed otherwise that my unit tests weren't running. I wonder how many people have switched and haven't noticed that they are running fewer tests than they did before.

The obvious workaround is to manually list the package names with go list ./....

EDIT: go test -cover -test.gocoverdir=foo $(go list ./...) doesn't work. You need to run the test for each subpackage in a subshell in that directory.

[gabyhelp]

Related Issues

• cmd/go: go 1.22.0: go test throws errors when processing folders not listed in coverpkg argument #65653 (closed)
• cmd/go: inconsistent coverage when using -coverprofile with packages without tests #70244 (closed)
• runtime/coverage: confusing go test -cover + GOCOVERDIR behavior #66225
• Unintuitive behavior when using -coverpkg #23786 (closed)
• cmd/go: coverpkg doesn't ignore directories starting with '.' #66038
• runtime/coverage: coverage from other binaries is not collected in tests #60182
• cmd/go: test with coverage unexpect fail #43242 (closed)
• cmd/cover: cross-package coverage logic has changed under gotoolchain 1.24 #71716 (closed)
• cmd/covdata: percent does not support -coverpkg #66464
• cmd/cover: regression in 1.20, cover tool no longer includes packages without tests #58770 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cyphar]

Updated since it turns out that even explicitly naming the subpackages or doing go test -cover -test.gocoverdir=foo $(go list ./...) doesn't do what you expect. Here is my planned workaround:

readarray -t pkgs < <(go list -f "{{ .Dir }}" ./...)
[[ "$GOCOVERDIR" =~ ^/ ]] || GOCOVERDIR="$PWD/$GOCOVERDIR"
for dir in "${pkgs[@]}"
do
	pushd "$dir"
	go test -cover -test.gocoverdir="$GOCOVERDIR" .
	popd
done

[seankhliao]

as I understand, -test.coverdir is an argument understood only by test binaries, rather than go test itself. Once go test sees a flag that it doesn't recognize, any remaining flags and arguments are passed directly to the test binary, so your package list goes to the test binary as arguments, rather than to go test as packages to test.

Since this is using undocumented internal details, I don't think we're going to make any change, especially now that the new behavior has been around for a few years.

[cyphar]

Wait, is this really undocumented? There is no other way of using go test with the new GOCOVERDIR support added in Go 1.20 and it seems that it was intended to be supported in the original implementation but the patch was never merged. What is the "supported" way?

FWIW I found another workaround in that thread though, go test -cover ./... -args -test.gocoverdir=mydir. I guess it works for the same reason you mentioned (the unknown flag causes argument parsing to be skipped).

[cyphar]

I guess there should be a feature request opened to support this then, given that the utility of GOCOVERDIR is quite significantly reduced without this feature. (The "old-school way" at least gave you coverage for everything in a supported way.)