cmd/go: building requires otherwise-unused .mod files to be present

[stapelberg]

What version of Go are you using (go version)?

$ go version
go version go1.12beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michael/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/gp-repro"
GOPROXY=""
GORACE=""
GOROOT="/home/michael/sdk/go1.12beta1"
GOTMPDIR=""
GOTOOLDIR="/home/michael/sdk/go1.12beta1/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/tmp/repro/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build500484946=/tmp/go-build -gno-record-gcc-switches"

What did you do?

% export GOPATH=/tmp/gp-repro
% mkdir /tmp/repro
% cat > repro.go <<'EOT'
package main

import (
	"fmt"

	"golang.org/x/exp/mmap"
	"gonum.org/v1/gonum/unit"
)

func main() {
	fmt.Printf("%v, %v", mmap.ReaderAt{}, unit.Meter)
}
EOT
% go mod init repro
% go build
go: finding gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e
go: finding golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
go: finding golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b
go: downloading gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e
go: downloading golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: extracting golang.org/x/exp v0.0.0-20181221233300-b68661188fbf
go: extracting gonum.org/v1/gonum v0.0.0-20181223131727-b545e3e77e9e

As you can see, two versions of golang.org/x/exp are involved here: the repro module uses the latest (v0.0.0-20181221233300-b68661188fbf), whereas the gonum.org/v1/gonum module specifies an older version (v0.0.0-20180321215751-8460e604b9de).

In the build itself, the latest version is picked:

% go list -m -json all
{
	"Path": "repro",
	"Main": true,
	"Dir": "/tmp/repro",
	"GoMod": "/tmp/repro/go.mod",
	"GoVersion": "1.12"
}
{
	"Path": "golang.org/x/exp",
	"Version": "v0.0.0-20181221233300-b68661188fbf",
	"Time": "2018-12-21T23:33:00Z",
	"Dir": "/tmp/gp-repro/pkg/mod/golang.org/x/exp@v0.0.0-20181221233300-b68661188fbf",
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20181221233300-b68661188fbf.mod"
}
{
	"Path": "golang.org/x/tools",
	"Version": "v0.0.0-20180525024113-a5b4c53f6e8b",
	"Time": "2018-05-25T02:41:13Z",
	"Indirect": true,
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20180525024113-a5b4c53f6e8b.mod"
}
{
	"Path": "gonum.org/v1/gonum",
	"Version": "v0.0.0-20181223131727-b545e3e77e9e",
	"Time": "2018-12-23T13:17:27Z",
	"Dir": "/tmp/gp-repro/pkg/mod/gonum.org/v1/gonum@v0.0.0-20181223131727-b545e3e77e9e",
	"GoMod": "/tmp/gp-repro/pkg/mod/cache/download/gonum.org/v1/gonum/@v/v0.0.0-20181223131727-b545e3e77e9e.mod"
}

However, when preventing internet access by pointing GOPROXY to an unused port, I can see that the old version of golang.org/x/exp needs to be present for the build to succeed:

% rm repro && GOPROXY=http://localhost:1111/ go build
# works
% rm /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.*
% rm repro && GOPROXY=http://localhost:1111/ go build
go: finding golang.org/x/exp v0.0.0-20180321215751-8460e604b9de
go: golang.org/x/exp@v0.0.0-20180321215751-8460e604b9de: Get http://localhost:1111/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.info: dial tcp [::1]:1111: connect: connection refused
go: error loading module requirements
% echo 'module golang.org/x/exp' > /tmp/gp-repro/pkg/mod/cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.mod
% GOPROXY=http://localhost:1111/ go build                                                                                         
# works

This came as a surprise to me, so I wanted to check in and see if this was intentional or an oversight. Given that the .mod file only contains “module golang.org/x/exp”, it seems to me that the file could easily be synthesized?

I discovered this issue because I was experimenting with packaging Go modules in Linux distributions (e.g. Debian). It would be a significant downside if we needed to provide (and therefore maintain!) all versions that a build ever references, instead of just the versions which are selected in the end.

Thanks,

[mvdan]

It might take a while for this to get a proper reply, as most engineers will be on holiday this week :)

/cc @bcmills @myitcv @thepudds

[thepudds]

@stapelberg two quick comments:

1. you might want to set GOPROXY=off (rather than using a non-existent port). That means go commands in module mode are disallowed from using the network for dependencies. I don't know if that explains your symptoms (and probably it does not), but you might get more standard behavior in general out of GOPROXY=off.

2. you could try with the go 1.12 beta. I think there was at least one fix in 1.12 for avoiding hitting the network unnecessarily (though it might have had to do with replace, which might not help here).

edit: sorry, I see you are using the 1.12 beta.

[thepudds]

Setting GOPROXY=off and attempting to reproduce gives a slightly different error, but still fails:

$ go1.12beta1 build
go: golang.org/x/exp@v0.0.0-20180321215751-8460e604b9de: module lookup disabled by GOPROXY=off
go: error loading module requirements

Someone else might need to comment if this is a bug, vs. necessary behavior, vs. perhaps a not-yet-implemented feature could avoid this.

@stapelberg you mentioned:

It would be a significant downside if we needed to provide (and therefore maintain!) all versions that a build ever references, instead of just the versions which are selected in the end.

I am just looking at this quickly, but is it just meta data about the old version 20180321215751-8460e604b9de that currently seems to be required? E.g., after a clean build (with GOPROXY unset):

$ ls $GOPATH/pkg/mod/cache/download/golang.org/x/exp/@v                      
list
v0.0.0-20180321215751-8460e604b9de.info
v0.0.0-20180321215751-8460e604b9de.mod
v0.0.0-20181221233300-b68661188fbf.info
v0.0.0-20181221233300-b68661188fbf.mod
v0.0.0-20181221233300-b68661188fbf.zip
v0.0.0-20181221233300-b68661188fbf.ziphash

and:

$ ls $GOPATH/pkg/mod/golang.org/x
exp@v0.0.0-20181221233300-b68661188fbf

or:

$ cd $GOPATH/pkg/mod
$ find -name "*8460e604b9de*"
./cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.info
./cache/download/golang.org/x/exp/@v/v0.0.0-20180321215751-8460e604b9de.mod

[stapelberg]

I am just looking at this quickly, but is it just meta data about the old version 20180321215751-8460e604b9de that currently seems to be required? E.g., after a clean build (with GOPROXY unset):

Yes, it’s just the .mod file. Hence my suspicion that this is unintended.

[bcmills]

This is working as designed. Every go.mod file in the transitive requirement graph — even for versions that we know will not be selected — impacts the constraints on which versions we can ultimately choose, so we have to load those files in order to compute the final versions.

The good news is, go.mod files are small and cacheable, so the impact on build times should usually be negligible.

See the cross test-cases for the MVS package for examples:

go/src/cmd/go/internal/mvs/mvs_test.go

Lines 42 to 116 in 006a5e7

	# Cross-dependency between D and E.
	# No matter how it arises, should get result of merging all build lists via max,
	# which leads to including both D2 and E2.
	name: cross1
	A: B C
	B: D1
	C: D2
	D1: E2
	D2: E1
	build A: A B C D2 E2
	name: cross1V
	A: B2 C D2 E1
	B1:
	B2: D1
	C: D2
	D1: E2
	D2: E1
	build A: A B2 C D2 E2
	name: cross1U
	A: B1 C
	B1:
	B2: D1
	C: D2
	D1: E2
	D2: E1
	build A: A B1 C D2 E1
	upgrade A B2: A B2 C D2 E2
	name: cross1R
	A: B C
	B: D2
	C: D1
	D1: E2
	D2: E1
	build A: A B C D2 E2
	name: cross1X
	A: B C
	B: D1 E2
	C: D2
	D1: E2
	D2: E1
	build A: A B C D2 E2
	name: cross2
	A: B D2
	B: D1
	D1: E2
	D2: E1
	build A: A B D2 E2
	name: cross2X
	A: B D2
	B: D1 E2
	C: D2
	D1: E2
	D2: E1
	build A: A B D2 E2
	name: cross3
	A: B D2 E1
	B: D1
	D1: E2
	D2: E1
	build A: A B D2 E2
	name: cross3X
	A: B D2 E1
	B: D1 E2
	D1: E2
	D2: E1
	build A: A B D2 E2

[stapelberg]

This is working as designed. Every go.mod file in the transitive requirement graph — even for versions that we know will not be selected — impacts the constraints on which versions we can ultimately choose, so we have to load those files in order to compute the final versions.

Can you elaborate on the ways in which these files can impact the version selection? Specifically, are you saying it is not sufficient to synthesize these files?

The good news is, go.mod files are small and cacheable, so the impact on build times should usually be negligible.

That’s not my concern, though. In Linux distributions such as Debian, we strive to have precisely one version of a specific module packaged. We can’t keep old packages around just for their .mod files.

If this is really working as intended with no wiggle room, we’ll need to work around that and either modify .mod files at package-time (seems more labor-intensive, but haven’t thought it through entirely) or synthesizing .mod files at build-time (ugh).

[bcmills]

In Linux distributions such as Debian, we strive to have precisely one version of a specific module packaged. We can’t keep old packages around just for their .mod files.

Sorry, but Go's version selection algorithm intentionally prioritizes reproducibility over recency: when you build a module-enabled binary, you use the specific versions of the modules it was tested against, not whatever newer version happens to be installed on your system.

Now that #27859 is fixed, you can probably make something work using wildcard replace directives, but the resulting builds will not be as reproducible as ordinary module-enabled builds.

[thepudds]

@stapelberg Is this in the context of what I think in the past might have been described as "more or less one giant GOPATH" for Debian packaging of Go libraries, or is this more in the context of trying to use Go modules to break out of that (or perhaps nothing to do with any of that)?

I am not sure how much flexibility you have at package time vs. build time, how that changes for a package vs. a binary, the level of control around setting env variables, or your ability to use an auxiliary directory that contains auxiliary data like go.mod files...

...but depending on your constraints, you might be able to do some variety of using GOPROXY=file:/// and/or temporarily setting GOPATH (which controls the location of GOPATH/pkg/mod cache location). That might not help, but here is a concrete example (from @myitcv):

https://github.com/go-modules-by-example/index/tree/master/012_modvendor

That particular example is showing an alternative to traditional vendoring, and I suspect that is not exactly what you'd want, but it serves as a concrete example of the types of things that are enabled via the flexibility of GOPROXY=file:///.

In any event, good chance GOPROXY=file:/// does not help based whatever your constraints are, but wanted to at least mention it in case it gives you an idea, etc.

[stapelberg]

resulting builds will not be as reproducible as ordinary module-enabled builds.

Debian has its own reproducibility efforts, see https://reproducible-builds.org/. The binaries we build are reproducible, and have been before Go modules :)

@stapelberg Is this in the context of what I think in the past might have been described as "more or less one giant GOPATH" for Debian packaging of Go libraries

Yes. This is the current model and we have no plans to change that.

I am not sure how much flexibility you have at package time vs. build time,

Changes at packaging-time take a long time to trickle down to all our volunteers. Build-time changes are preferable for this particular situation.

how that changes for a package vs. a binary,

No change.

the level of control around setting env variables

We can set arbitrary env variables.

or your ability to use an auxiliary directory that contains auxiliary data like go.mod files...

We can do that, but every step we need to set up introduces additional complexity, which makes builds slower and harder to understand/debug.

...but depending on your constraints, you might be able to do some variety of using GOPROXY=file:/// and/or temporarily setting GOPATH (which controls the location of GOPATH/pkg/mod cache location). That might not help, but here is a concrete example (from @myitcv):

Yeah, we currently already synthesize a GOPATH. From this discussion it seems like we need to synthesize all go.mod files ever referenced by the transitive closure as well. I have not yet looked into how easy or hard this would be to do.

[thepudds]

@stapelberg I am curious what your latest thinking here is, at least briefly? Part of the reason for me bouncing this is curiosity, but also some similar questions are coming up elsewhere.

Also, one specific area I do not understand is why you mention synthesizing a large number of go.mod files. Was it specifically limited to synthesizing older go.mod files that are not going to be selected as the final versions? If so, are you thinking they would be empty aside from the module foo line?

Expanding on an earlier point slightly, rather than synthesizing, you might be able use a filesystem-based module cache containing go.mod files. I suspect this is well known to you at this point, but:

• The module download cache location is controlled by GOPATH. In particular, go mod download, go build, etc. populate the module cache in GOPATH/pkg/mod.
• In addition, when you want to use a particular module cache, you can tell the go command to use a local module cache by setting GOPROXY=file:///file/path.
• You can put those two things together:

   # Populate a module download cache in /tmp/gopath-for-cache
   $ GOPATH=/tmp/gopath-for-cache  go mod download

   # Build using the contents of the module download cache in /tmp/gopath-for-cache
   $ GOPROXY=file:///tmp/gopath-for-cache/pkg/mod/cache/download  go build

In other words, that might be an approach to get the actual go.mod files of interest, rather than synthesizing a large number of older go.mod files.

Note that even though you are setting the GOPROXY environment variable in the steps above, there is no actual proxy process involved, and everything is just being read directly from the local filesystem. An even more detailed example is in the link I had mentioned above.

All that said, I don't know if that set of capabilities really helps you or not, and perhaps synthesizing is easier based on your use case.

[stapelberg]

Thanks for your reply.

Also, one specific area I do not understand is why you mention synthesizing a large number of go.mod files. Was it specifically limited to synthesizing older go.mod files that are not going to be selected as the final versions? If so, are you thinking they would be empty aside from the module foo line?

Yes to both questions.

In other words, that might be an approach to get the actual go.mod files of interest

I don’t follow. Where would they come from? Note that Linux distributions generally prohibit internet access during builds, so all files we want to use need to be shipped in other packages (build dependencies) or created by the build infrastructure.