proxy.golang.org: challenges in simultaneously patching downstream dependencies in go1.13

[zmanian]

What version of Go are you using (go version)?

go 1.13.1

What did you do?

On Tuesday, at 11pm PST, Tendermint discolsed a high severity vulnerability in our software which acts as a state machine replication layer in several billion dollars of value transfer activity in the cryptocurrency ecosystem.

Our goal was to do patch releases of Tendermint and then do releases of all our downstream dependencies simultaneously,

We released tendermint v0.31.9 and then releasesed the cosmos-sdk v0.34.8 with an updated go.mod to depend on Tendermint v0.31.9. The go.sum was no updated in cosmos-sdk v0.34.8 in the initial release.

Users on go1.12.1 were able git clone and make install in the cosmossdk release.

Users on go1.13.1 got the following error message

verifying github.com/tendermint/[email protected]/go.mod: github.com/tendermint/[email protected]/go.mod: reading https://sum.golang.org/lookup/github.com/tendermint/[email protected]: 410 Gone

This lasted about 25 min before https//sum.golang.org updated.

GONOSUMDB=github.com/tendermint/tendermint go mod verify helped for some users but not others.

What did you expect to see?

I would have expected to see new releases recorded in https://sum.golang.org/ relatively quickly and not to have to wait so long. Also with GONOSUMDB=github.com/tendermint/tendermint, it was unclear exactly how this interacts with GOPROXY=direct

CC/@FiloSottile @katiehockman

[heschi]

This is probably about the same case as #34370 (comment). This is more evidence in favor of at least synchronizing the various caches. I don't have any other good ideas off the top of my head. (Shortening the negative cache timeout is an obvious one, but has tradeoffs.)

Setting GOPRIVATE=github.com/tendermint to disable use of the proxy and sumdb should have worked. If disabling just the sumdb helped, that's probably due to differences in cache lifetimes as I described above, and the regional caches would explain the differences between users.

[zmanian]

I don't know if GOPRIVATE=github.com/tendermint would have worked, I didn't understand the advantages of that option until the comment above.

[zmanian]

It seems like GOPRIVATE is the breakglass variable for disabling all the infrastructure between go get and github?

[FiloSottile]

Yes, GOPRIVATE means "you won't find this in the proxy or sumdb, don't try". GONOSUMDB is less useful in isolation, it's meant for running your own proxy for things that are not in the sumdb.

BTW, adding the line manually to the dependency's go.sum would have saved the sumdb lookup, but without GOPRIVATE (or GONOPROXY or GOPROXY=direct) it would still have had to wait for the contents to be available in the proxy.

[hyangah]

In this specific case, the module mirror attempted to fetch the version multiple times but failed because the version was not published yet. Actually the last failed attempt before the successful fetch was just a couple of minutes before the tag became available upstream and placed it into the negative cache.

Given this limitation (unless the mirror can shorten the negative cache ttl dramatically or take completely different approaches in accepting new module versions), module owners have to be cautious during the release process i.e., tag first, fetch through the module mirror, and then announce the version. 😞

[katiehockman]

Thanks for filing the issue. We will clarify our FAQ entry to describe our negative caching a bit more in hopes of clarifying future issues. We also have an action item on our side to synchronize the internal and external caches, which will at least lower the wait time for versions that were negative cached.

However, as others have said, this is mostly WAI (even though it isn't perfect). Since the go command performs package resolution against the proxy by generating requests that are expected to fail, lowering our negative cache TTL isn't something we're ready to do right now.

The suggested approach in the future would be what @hyangah suggested: tag first, verify that the fetch works second, and announce third.

[zmanian]

Thanks @katiehockman