cmd/go: stamp git/vcs current HEAD hash/commit hash/dirty bit in binaries

[bradfitz]

(Related but different than #35667)

cmd/go currently embeds all the module dep information in binaries and it's readable with e.g. https://godoc.org/rsc.io/goversion/version but it does not include any information about the top-level module's version.

I propose that cmd/go look at {git,svn,etc} state and include in the binary:

• HEAD commit time
• HEAD hash
• dirty bit (if there are uncommitted changes)

Currently many projects do this by hand with a build-program.sh and stamping it manually with --ldflags=-X foo=bar, but that means programs built the normal Go way lack that information, and people end up with non-portable (shell, often) build scripts.

I've hit this enough times with my own projects that it's actively frustrating me. It's worse when programs are clients that want to report their version number to a server (which might want to do analytics, build horizon enforcement, protocol version negotiation, etc) and then can't. There are alternative ways to do all that, but they're tedious.

Mostly I'm concerned that people have bespoke, often non-portable build scripts.

Note, July 15, 2021: This proposal was accepted but has not yet been implemented. Per discussion in #37693, when it does get implemented, we need to include a flag to turn this behavior off. - rsc

[bcmills]

This may be a duplicate of #29814.

[rsc]

We should figure out exactly what we want to record.
The other modules are all versions or pseudo-versions.
Should this one be too?
Can we find that quickly enough to be reasonable to run during every 'go build'?
(We can skip it during 'go test' like we skip dwarf.)
The go command already has code to turn a commit hash into a version; we should probably just use that same code and add a +modified if the working directory is modified.

It would be helpful to time how much overhead this would be in 'go build'.

@bcmills, do you have any numbers about how much time this would add?

[rsc]

BTW I agree it's a duplicate of #29814 but I'll keep using this one because it is marked as a proposal and already appeared in the minutes.

[bcmills]

It would be helpful to time how much overhead this would be in 'go build'.

git status --porcelain=v2 in the go repository is around 50ms for me, and git log -n 1 is around 25ms.

go install cmd/go for me is 1.6s dirty, and 140ms clean. So assuming that we can run the VCS commands in parallel with builds for non-main packages, the latency hit should be negligible.

CC @jayconrod @matloob

[bcmills]

Hmm, I realized that I didn't account for checking tags in the above calculations. Still, I expect those costs will be order-of-magnitude similar to any other git command.

[dottedmag]

I'd like to point out a (maybe small) problem with this approach: changing the version of source code, but not the code itself will cause the binary to change.

Let me explain the use-case I have that will be broken by this change:

• A number of binaries are built from a monorepo, each binary is packaged in Docker, and a Helm chart is generated for every binary. This chart includes Docker image ID, not the tag.
• These Helm charts are installed to Kubernetes

If the Helm chart contents and the binaries don't change, then no upgrades are performed by Kubernetes.

If the version of the checkout is stamped into every Go binary, then this scheme crumbles and:

• either a version needs to be purged from every binary before packaging it into a Docker image,
• or some crazy scheme has to be invented: remove the version from the binary, take a checksum, check that there is no Docker image with this version as a label published in the registry, publish the image and label it, use this label in Helm charts.

[bradfitz]

@dottedmag, what if we made it conditional on importing a new package, say runtime/version, containing the accessor funcs to get at the info? Then if you don't use it, no change in behavior.

Would that work for your use case?

[mark-rushakoff]

I'm in a practically identical case to @dottedmag.

Inevitably, somewhere in the monorepo we will (perhaps unintentionally) bring in a dependency that depends on a magic package that breaks deterministic builds.

I think for most common cases, having this proposal enabled by default would be preferable. For my use case, I would be satisfied if there was a documented way to opt out of it. We already are using -ldflags=-buildid= to force a consistent build ID as part of deterministic binaries, so another esoteric flag to opt out of recording VCS state would be completely acceptable.

[dottedmag]

@dottedmag, what if we made it conditional on importing a new package, say runtime/version, containing the accessor funcs to get at the info? Then if you don't use it, no change in behavior.

I agree with @mark-rushakoff: relying on imports will be brittle unless this import is considered only for main packages.

It's not an author of some recursively included library, but a builder of a final binary who in a position to decide whether to put versioning information into the binary or not.

[bcmills]

@dottedmag, note that many functionally-equivalent builds will already produce slightly different binaries due to the version-stamping for runtime/debug.ReadBuildInfo. The version stamps will change with each change to the corresponding module versions, even if the contents of the specific imported packages are the same.

This proposal would case more of the same sort of version churn, but it is fundamentally the same churn.

That suggests that we may want to provide an option to disable version stamping in general. IMO, that should be a separate proposal.