x/vgo: possible non-determinism in updating go.sum

[daaku]

What version of Go are you using (go version)?

go version go1.10.3 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

I'm using the current latest released go and golang/vgo@cdd82a7.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/naitik/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/naitik/usr/go"
GOPROXY=""
GORACE=""
GOROOT="/Users/naitik/usr/go1.10.3"
GOTMPDIR=""
GOTOOLDIR="/Users/naitik/usr/go1.10.3/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fn/8xcrnptx0f75fryl8n30kbyr0000gn/T/go-build924712898=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""

What did you do?

I knew there was an update to one of my dependencies (crawshaw.io/sqlite) and ran vgo get -u to pull in that and any other updates.

What did you expect to see?

I expected to see changed revisions as well as updated cryptographic checksums in go.sum.

What did you see instead?

go.sum shows me new entries for the updated revisions, but the checksums are the same as the old ones. Here's the git diff after running vgo get -u:

diff --git a/go.sum b/go.sum
index e36613c..63d7df0 100644
--- a/go.sum
+++ b/go.sum
@@ -1,4 +1,5 @@
 crawshaw.io/sqlite v0.0.0-20180510162726-4e3e3e8c528e/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
+crawshaw.io/sqlite v0.0.0-20180710105110-95479735d36f/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
 github.com/NYTimes/gziphandler v0.0.0-20180227064818-5032c8878b9d/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
 github.com/blang/semver v0.0.0-20170727064818-2ee87856327b/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/daaku/ensure v0.0.0-20180215095550-9bd7e49e586d/go.mod h1:Ow4+0ZvWSGA0FkpjHd3H5R0LiKuSXFBf7oBPzJpCmSk=
@@ -12,9 +13,12 @@ github.com/gorilla/csrf v1.5.1/go.mod h1:HTDW7xFOO1aHddQUmghe9/2zTvg7AYCnRCs7MxT
 github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/rakyll/statik v0.1.1/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
+github.com/rakyll/statik v0.1.2/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
 github.com/valyala/bytebufferpool v0.0.0-20160817181652-e746df99fe4a/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/quicktemplate v0.0.0-20180430205211-a91e0946457b/go.mod h1:uX8yrAzr955BRr+rAerWX4V1NHb8bFfsePffuN5lFso=
 github.com/vincent-petithory/dataurl v0.0.0-20160330182126-9a301d65acbb/go.mod h1:FHafX5vmDzyP+1CQATJn7WFKc9CvnvxyvZy6I1MrG/U=
 golang.org/x/image v0.0.0-20171214225156-12117c17ca67/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
+golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
 golang.org/x/net v0.0.0-20180524181706-dfa909b99c79/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180710023853-292b43bbf7cb/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 xojoc.pw/useragent v0.0.0-20170215185434-52903803fc66/go.mod h1:71om/Qz9HbIEjbUrkrzmJiF26FSh6tcwqSFdBBkLtJQ=

Next I ran vgo install ./... inside my module, and I saw go.sum was modified again, this time new checksums showed up for some of the entries (notice golang.org/x/net was not changed):

diff --git a/go.sum b/go.sum
index e36613c..847db9e 100644
--- a/go.sum
+++ b/go.sum
@@ -1,4 +1,6 @@
 crawshaw.io/sqlite v0.0.0-20180510162726-4e3e3e8c528e/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
+crawshaw.io/sqlite v0.0.0-20180710105110-95479735d36f h1:vn2WsFRW6O68ppjB8cvQAMVtzUytvh+RNx9ZM+Lx4iE=
+crawshaw.io/sqlite v0.0.0-20180710105110-95479735d36f/go.mod h1:BZaitnE9BVpocOuCdi/y5XReJMUelG53e/rDSLwSFzY=
 github.com/NYTimes/gziphandler v0.0.0-20180227064818-5032c8878b9d/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ=
 github.com/blang/semver v0.0.0-20170727064818-2ee87856327b/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/daaku/ensure v0.0.0-20180215095550-9bd7e49e586d/go.mod h1:Ow4+0ZvWSGA0FkpjHd3H5R0LiKuSXFBf7oBPzJpCmSk=
@@ -12,9 +14,14 @@ github.com/gorilla/csrf v1.5.1/go.mod h1:HTDW7xFOO1aHddQUmghe9/2zTvg7AYCnRCs7MxT
 github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/rakyll/statik v0.1.1/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
+github.com/rakyll/statik v0.1.2 h1:ckeqyWXP9hT/2u8Xfi4L02B2RFlqr6fdedIPgohOAHM=
+github.com/rakyll/statik v0.1.2/go.mod h1:OEi9wJV/fMUAGx1eNjq75DKDsJVuEv1U0oYdX6GX8Zs=
 github.com/valyala/bytebufferpool v0.0.0-20160817181652-e746df99fe4a/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/quicktemplate v0.0.0-20180430205211-a91e0946457b/go.mod h1:uX8yrAzr955BRr+rAerWX4V1NHb8bFfsePffuN5lFso=
 github.com/vincent-petithory/dataurl v0.0.0-20160330182126-9a301d65acbb/go.mod h1:FHafX5vmDzyP+1CQATJn7WFKc9CvnvxyvZy6I1MrG/U=
 golang.org/x/image v0.0.0-20171214225156-12117c17ca67/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
+golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81 h1:00VmoueYNlNz/aHIilyyQz/MHSqGoWJzpFv/HW8xpzI=
+golang.org/x/image v0.0.0-20180708004352-c73c2afc3b81/go.mod h1:ux5Hcp/YLpHSI86hEcLt0YII63i6oz57MZXIpbrjZUs=
 golang.org/x/net v0.0.0-20180524181706-dfa909b99c79/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180710023853-292b43bbf7cb/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 xojoc.pw/useragent v0.0.0-20170215185434-52903803fc66/go.mod h1:71om/Qz9HbIEjbUrkrzmJiF26FSh6tcwqSFdBBkLtJQ=

Something seems to be off?

[daaku]

To make things even more odd, I removed $GOPATH/{pkg,src/mod} and go.sum and ran vgo install ./... and found go.sum had new entries indicating that the contents of the pkg or src/mod directories were affecting go.sum in some surprising way.

[daaku]

Just to provide some more context, I'm seeing some unpredictable (possibly racy?) behavior between the two lines in go.sum. The first line without the /go.mod bit seems to show up when building the same piece of code without any other apparent changes. This doesn't happen with all dependencies, just some.

+github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=

[oiooj]

vgo updated recently，and add the /go.mod suffix on the version:

The hash is a checksum over the one-file tree containing only the go.mod for the module now. It is normal that the checksum not changed, but the version changed.

[rsc]

The details reported so far seems to be working correctly to me. What you're probably missing is that vgo goes out of its way to avoid downloading information until that information is necessary. This is an important optimization.

vgo get -u updates go.mod to reflect the new modules (and their dependencies) but does not run a build of everything in the module. To do the update of go.mod it only needs other go.mod files. It does not need the full source trees for the new dependencies, and so it does not look at them or even necessarily download them, and so it does not put hashes in go.sum. The /go.mod entries in the go.sum are the hashes of just the go.sum files, not the full trees.

'vgo install ./...' after 'vgo get -u' does need the source code for the new dependencies, because now it's actually reading the source code. That's why it adds new entries for some directory trees to go.sum.

If a go.mod says it requires foo.com/bar and no import found during the build points into foo.com/bar, then vgo will not load that module's source code at all. That's why not all /go.mod entries in go.sum have corresponding non-/go.mod entries, even after vgo install ./....

After removing $GOPATH/{pkg,src/mod} and go.sum and re-running vgo install ./..., I don't know exactly what you mean by "new entries". Yes, go.sum will be repopulated. It should not have any entries that were not present in the removed copy, though, since the vgo install ./... should be looking at the same inputs as before. If by "new entries" you mean something that wasn't in the go.sum you removed, I'd very much like to hear more about the details.

I don't understand the comment about go-spew. What command are you running that adds that line to go.sum? Does it happen every time you run the command? Just half the time? If the behavior is non-deterministic, can you say what setup is required to reproduce the non-determinism?

Thanks.

[daaku]

Gotcha, I was expecting the go.sum file to be affected by vgo get -u in a way that now I understand is not what it's designed to do.

I was still experiencing some kind of non-determinism as best as I could tell. I will see if I can come up with a repro case tomorrow.

[daaku]

Closing this since it was due to my misunderstanding of how/when go.sum gets populated.

Specifically my build process involves a vgo run of some otherwise ignored files as well as vgo test both of which triggered pulling in dependencies which the initial vgo install did not. This is what was causing go.sum to get updated.