cmd/go: stamp the pseudo-version in builds generated by `go build`

[4ad]

NOTE: The accepted proposal is #50603 (comment).

---

cmd/go embeds dependency version information in binaries, which is very useful. From Go 1.18 onwards, cmd/go also embeds VCS information in binaries, which makes it even more useful than it was before.

As #37475 mentions, people place version information in binaries using -ldflags='-X foo=bar', which requires an additional build wrapper. The new VCS stamping feature of cmd/go should alleviate the need for external wrapper, but I am afraid it comes short.

The version information, in the sense of Go's pseudo version is not recorded for the main module when doing go build:

: emerald:ver; go build
: emerald:ver; go version -m hello | grep 'mod.*hello'
	mod	mgk.ro/hello	(devel)	
: emerald:ver; 

The version is recorded as expected when doing go install:

: emerald:ver; go install robpike.io/ivy@latest
go: downloading robpike.io/ivy v0.1.124
: emerald:ver; go version -m `which ivy` | grep 'mod.*ivy'
	mod	robpike.io/ivy	v0.1.12	h1:qI7dnEiXhorB+za07W6qX3sG+IvBK4EUl38vUHAf53Q=
: emerald:ver; 
: emerald:ver; 

I am afraid this limitation of cmd/go will continue to force people to use external build wrappers that set -ldflags, which is rather unfortunate.

I am not the first to want main module version information in binaries, this has been already asked for in various issues, for example in #29814, which was closed as a duplicate of #37475, but it really wasn't a duplicate, as #37475 is about VCS information, and #29814 is about semantic versioning. Other examples of people asking for this feature are mvdan/sh#519 and #29228 (comment) where various workarounds were proposed.

Speaking of workarounds, the only workaround that I know that currently works would be to create a local module proxy and pass GOPROXY to go install, but that is an extremely high-overhead workaround, and go install is not a replacement for go build anyway, since go install comes with some rather severe limitations regarding how vendoring works and what you can put in go.mod, and go install doesn't support controlling GOBIN when cross-compiling.

I realize that Git tags are a local concept, and by doing the "wrong" git operations one could come up with a different pseudo-version for the same source code. I am afraid I don't have any solution or suggestion regarding this git misfeature, except to note that even in this case the hash information is recorded correctly, and in every case by the virtue of having access to the local source code the programmer can always do some local operation that has the potential to cause a version mislabeling. Git is just more prome to do this by accident, but the ability is there, always.

I don't have any stats to back this up, but from my experience most corporate source code is built by go build, not go install, and it would be great if somehow Go's notion of versioning would be stamped by go build.

CC @bcmills @mvdan @rsc

[mvdan]

At least speaking personally, for cases like mvdan/sh#519, my intent is to show something like devel ${GIT_SHA} when someone does a local Go build out of a git checkout. If someone is manually cloning and building, as opposed to the advertised and easier go install url@latest, I imagine they know what a git hash is. So what 1.18 is currently shipping with is enough for my needs.

It's true that something like a proper module version might be more useful; a git commit hash doesn't give any hint as to how old a version is, whereas a semver version prefix or a timestamp can give a starting point. So, in principle, I agree with you: 1.18 is a big step forward, but it's still unfortunate that the main module version remains as (devel) for local builds.

However, in practice, I still agree with Jay's comment in #29228 (comment); we shouldn't make such a "locally inferred version" look like a normal version, because it's reasonably likely to be wrong or cause confusion with users.

in every case by the virtue of having access to the local source code the programmer can always do some local operation that has the potential to cause a version mislabeling.

Could you give some examples? I can only think of very unlikely scenarios, such as manually corrupting the module download cache after downloading some dependencies. That cache is read-only by default, and go mod verify exists to double-check the contents too.

With the main module in a git checkout, I can think of multiple scenarios which seem more likely:

• What if I've made a commit or tag but not pushed it?
• What if I've edited some files and not committed them?
• What if two people on two computers make the same tag with different code - would they end up with the same exact module version for different software? If one of them pushes their tag to the internet, would the other's computer be affected by the mismatch?

I think that, if we are to implement something like this, the versions must be somehow different from the canonical and unique versions that get computed from fully published commits and tags. This would make it very clear that the versions are inferred from local state, and not guaranteed to be correct. As a simplistic example, imagine that tagging v1.2.3 locally results in a build whose main module version is devel v1.2.3, but when pushed and go installed, gets the version v1.2.3.

[mvdan]

we shouldn't make such a "locally inferred version" look like a normal version, because it's reasonably likely to be wrong or cause confusion with users.

To add a more concrete example: if we made the change proposed here, and locally inferred versions looked like fully published versions, I would have a harder time trusting the output of shfmt -version when my users report bugs. I would have to update the issue template to also ask: did you build from a modified git checkout?

[4ad]

Could you give some examples? I can only think of very unlikely scenarios, such as manually corrupting the module download cache after downloading some dependencies. That cache is read-only by default, and go mod verify exists to double-check the contents too.

I was thinking of the case where since Go itself doesn't expose its own concept of a version to the program, the users themselves are forced to create their own concepts of a version, either through things like VERSION files, or through some build wrappers. By definition, any such concept is under user's control, and the user can and will make mistakes. In fact, from experience, users try to naively use git tags for this which then fail for precisely the reasons you just explained.

Let me rephrase my point. Go can't enforce any useful properties for the user's notion of a version because it doesn't know about it, and as such if we make userVersion==moduleVersion, the fact that Go can't enforce any properties is neither better nor worse for the user. The user is on the hook for doing the right thing in both cases. In one case the user must properly maintain their VERSION, and in the other case the user must properly maintain their git checkouts.

The user does gain something in the latter case though. They don't have to create build wrappers.

With the main module in a git checkout, I can think of multiple scenarios [which might fail ... ] I think that, if we are to implement something like this, the versions must be somehow different from the canonical and unique versions that get computed from fully published commits and tags. [...] As a simplistic example, imagine that tagging v1.2.3 locally results in a build whose main module version is devel v1.2.3, but when pushed and go installed, gets the version v1.2.3.

I very much agree with this, with one caveat. If the locally checked-out version is identical to a published release, I would expect the version to match the release. If the locally checked-out version can not be guaranteed to match any release, then yes, it should be published with something like devel v1.2.3 (which matches what Go does, but why not v1.2.3-devel or v1.2.3-unknown, which is semver-compatible?).

Unfortunately, I can't imagine how this would work without internet access, and quite often a prerequisite of automated systems running go build is to not go to the Internet.

[4ad]

Hold on, another thought. If we always add the commit hash, and some other metadata to the main module version for local builds, essentially always making them a fully qualified Go pseudo-version, then they will always be different from the published version, so there's no potential for confusion there.

Even better, in semver terms these builds will sort before the published version, which is probably what people want.

For this, what I said earlier about

If the locally checked-out version is identical to a published release, I would expect the version to match the release.

can no longer be true, but perhaps that is ok as long as we come up with a documented and stable convention that describes versioning for local builds (as opposed to just dumping a "devel" in the metadata field).

[bcmills]

The main caveat here, I think, is unpublished tags. If I create a local, unpublished tag for, say, v1.1000.0, then my pseudo-versions will be v1.1000.0-0.2022…, but everyone else's pseudo-versions may be on an arbitrarily lower version (say, v0.8.3-0.2022….

That may or may not be a significant issue, though: if we always use a pseudo-version, we'll at least have the commit hash as a common point of reference even if the base versions differ.

[mvdan]

@4ad right, a local build can't always know what is or isn't published, as requiring a network roundtrip takes us back to square one.

Your idea of trying to stick to semver, and always using some form of pseudo-version which includes a hash, sounds good. With one caveat, though: the commit hash isn't enough to make the version unambiguous, because I can have infinite kinds of uncommitted changes that do not change the HEAD commit hash.

@bcmills good point about tags still messing with pseudo-versions, but at least if we always include a timestamp and some form of unique hash, then I think we're good. With the caveat above about uncommitted changes :)

[mvdan]

We do have another hash available to us, though, which changes whenever any input Go code changes: the build IDs used for the build cache. I seem to recall that one such ID is embedded into binaries, too.

Not ideal, as such a hash also includes build parameters like GOOS or -tags, which don't normally affect versions. But at least it fixed the problem with uncommitted files in VCS.

[4ad]

Yes, uncommitted changes should be explicit in the pseudo-version, but I think we can suffix +, just as we do with Go itself, no?

[seankhliao]

the new buildinfo already records whether the workspace is clean with vcs.modified=true|false

[seankhliao]

we could use one of +local (for clean builds) or +dirty (uncommitted changes, implies local) as the semver build id, attached to a pseudoversion which should make the situation clear enough?

So main will always have a version like

vX.Y.Z-timestamp-commit+local
vX.Y.Z-timestamp-commit+dirty

[hyangah]

What is the main motivation of encoding the local version in pseudo-version style rather than keeping those extra info (timestamp?) as extra metadata fields - if it's not guaranteed that they are always available in the origin or proxies?

It seems like the vcs.time is already there in the build info metadata.

---

BTW, I feel like the main module's version isn't sufficient to describe a tool's behavior in certain cases - go version used to go build, third-party tools dependencies, and go build's behavior change (go.work left over somewhere accidentally?) can affect a tool's behavior. So when triaging issues, I hope we develop best practice using go version -m or richer build info dump rather than relying on the main module version string.

[4ad]

What is the main motivation of encoding the local version in pseudo-version style

The main motivation is that go install does it, and many people expect to have a notion of a program's version available and want it, and because they don't have it with go build, they rely on build wrapers or other workarounds, which are undesirable in the broader Go ecosystem.

encoding the local version in pseudo-version style rather than keeping those extra info (timestamp?) as extra metadata fields

Emphasis mine.

It's not rather than, It doesn't replace the existing metadata fields. If you want to read the metadata, you should read it from those fields instead of parsing the pseudo-version. However, that metadata is useful in disambiguating builds produced by go build from published releases. Presumably we could come with some other kind of metadata for the same purpose, but since pseudo-versions are a de-facto standard in the Go ecosystem, why not reuse it?

I feel like the main module's version isn't sufficient to describe a tool's behavior in certain cases - go version used to go build, third-party tools dependencies, and go build's behavior change (go.work left over somewhere accidentally?) can affect a tool's behavior.

This sounds like an argument to always use the build ID as the version suffix instead of the VCS hash.

So when triaging issues, I hope we develop best practice using go version -m or richer build info dump rather than relying on the main module version string.

I hope so too, but again, I think that discussion is out of scope for this thread, which is more about bringing go build in line with go install and providing a solution for users that avoids build wrappers.