cmd/go: update to '@latest' when promoting an indirect dependency to a direct one?

[carnott-snap]

What version of Go are you using (go version)?

$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="*.sc-corp.net"
GONOSUMDB="*.sc-corp.net"
GOOS="linux"
GOPATH="/home/user/go"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/user/.local/share/umake/go/go-lang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/user/.local/share/umake/go/go-lang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/trash/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build372133803=/tmp/go-build -gno-record-gcc-switches"

What did you do?

// go.mod
module mod

go 1.13

require sub v0.0.0-00010101000000-000000000000

// replace allows the reproducer to live entirely on the filesystem.
// This will work with external/real modules too.
replace sub => ./sub

// main.go
package main

import (
	_ "sub"

	_ "cloud.google.com/go/compute/metadata"
)

func main() {}

// sub/go.mod
module sub

go 1.13

require cloud.google.com/go v0.40.0

// sub/sub.go
package sub

import _ "cloud.google.com/go/compute/metadata"

[user@localhost ~]$ go mod tidy

What did you expect to see?

// go.mod 
module mod

go 1.13

require (
        cloud.google.com/go v0.46.3
        sub v0.0.0-00010101000000-000000000000
)

replace sub => ./sub

What did you see instead?

// go.mod 
module mod

go 1.13

require (
        cloud.google.com/go v0.40.0
        sub v0.0.0-00010101000000-000000000000
)

replace sub => ./sub

[seankhliao]

afaik go mod tidy only adds new dependencies and cleans up unneeded ones, it doesn't upgrade them. For that, use go get -u or go get -u=patch

[carnott-snap]

It concerns me that go mod tidy prefers my dependencies version, to @latest. In my examples above nothing breaks, but that is not necessarily the case: (happy to update things above)

// go.mod
module mod

go 1.13

require sub v0.0.0-00010101000000-000000000000

replace sub => ./sub

// main.go
package main

import (
	_ "sub"

	"cloud.google.com/go/compute/metadata"
)

var _ = metadata.Email

func main() {}

[user@localhost ~]$ go build
# mod
./main.go:9:9: undefined: metadata.Email

// go.mod 
module mod

go 1.13

require (
        cloud.google.com/go v0.40.0
        sub v0.0.0-00010101000000-000000000000
)

replace sub => ./sub

Whereas the following works fine:

// go.mod
module mod

go 1.13

// main.go
package main

import "cloud.google.com/go/compute/metadata"

var _ = metadata.Email

func main() {}

[user@localhost ~]$ go build

// go.mod 
module mod

go 1.13

require cloud.google.com/go v0.46.3

[carnott-snap]

Found a few more cases that I thought I should note:
all use the same main.go file:

// main.go
package main

import (
	"cloud.google.com/go/compute/metadata"
	_ "golang.org/x/oauth2"
)

var _ = metadata.Email

func main() {}

one

[user@localhost ~]$ go mod init mod
[user@localhost ~]$ go get golang.org/x/oauth2
[user@localhost ~]$ go build
# mod
./main.go:8:9: undefined: metadata.Email

two

[user@localhost ~]$ go mod init mod
[user@localhost ~]$ go get golang.org/x/oauth2
[user@localhost ~]$ go get cloud.google.com/go
[user@localhost ~]$ go build

three

[user@localhost ~]$ go mod init mod
[user@localhost ~]$ go build

[bcmills]

CC @jayconrod

[bcmills]

This is an interesting idea, but it would require a substantial amount of refactoring in the module loader in order to notice when we change an indirect dependency to a direct one. (Today that transition only happens when we reduce the complete build list to a minimal one, which occurs after we have finished resolving packages to modules.)

[seankhliao]

This also seems potentially dangerous: suppose you had an indirect dependency on D @ v0.1.0, by making it a direct dependency (say going from short to long variable declaration), it triggers an upgrade to 0.2.0 with a breaking api change.