cmd/go: go build -json

[aclements]

(Edited 2024-01-18: Changed PackageID to ImportPath.)

I propose we add structured JSON output to go build, similar to and compatible with the structured JSON output of go test -json. This JSON output will be enabled by a new go build -json flag, and also implicitly enabled for all builds done on behalf of go test -json. This proposal aims to address #23037 and #35169, as well as improve structured all.bash output (#37486).

The main motivation for this proposal is to enable build output that is consistent and compatible with go test -json output. Currently, if a test or imported package fails to build during go test -json, the build error text will be interleaved with the JSON output of tests. Furthermore, there’s currently no way to reliably associate a build error with the test package or packages it affected. This creates unnecessary friction and complexity in tools that consume the go test -json output.

For reference, the go test -json format is as follow:

type TestEvent struct {
	Time    time.Time // encodes as an RFC3339-format string
	Action  string
	Package string
	Test    string
	Elapsed float64 // seconds
	Output  string
}

A detailed description can be found in go doc test2json.

Proposal

There are two related parts to this proposal.

1. I propose we add a -json flag to go build that suppresses the text build errors written to stderr and instead writes a JSON-encoded stream to stdout described by the following type:

type BuildEvent struct {
	ImportPath string
	Action     string
	Output     string `json:",omitempty"` // Non-empty if Action == “build-output”
}

The ImportPath field gives the package ID of the package being built. This matches the somewhat misnamed Package.ImportPath field of go list -json and what golang.org/x/tools/go/packages calls the “ID”. This differs from the TestEvent.Package field, which is a plain import path, but using the full package ID is important for disambiguating errors from different builds and for consistency with the rest of the build process.

The Action field is one of the following:

• build-output - The toolchain printed output
• build-fail - The build failed

The Output field is set for Action == "build-output" and is defined exactly the same way as the TestEvent.Output field. A single event may contain one or more lines of output and there may be more than one output event for a given package ID.

This struct is designed so that parsers can distinguish interleaved TestEvents and BuildEvents by inspecting the Action field. Furthermore, as with TestEvent, parsers can simply concatenate the Output fields of all events to reconstruct the text format output, as it would have appeared from go build without the -json flag. (As a consequence, this means the output includes all post-processing done by the go tool, including path rewriting and adding the “# package” header before each package.)

Environmental errors (e.g., bad GOOS values, file system errors) and module-level errors (e.g., syntax errors in go.mod) will still be printed in plain text to stderr by the go command. These errors cause an immediate exit and happen very early, so they should never appear with JSON output.

2. I propose we add an optional FailedBuild field to TestEvent to connect tests that fail because of build errors back to the build error. When a test fails due to a build error, the go command would emit the usual "start" event for the test package, then an "output" event giving the "FAIL package name [build failed]" or equivalent message, followed by a "fail" event with the new FailedBuild field set to the package ID of the package that failed to build.

This same approach can be used for setup failures as well, which print a "FAIL package name [setup failed]" message. Setup failures reflect errors that prevent the go command from even invoking the compiler, such as errors in imports, but they are still logically build failures.

This approach to reporting build failures that affect tests tries to balance several considerations:

1. A test may fail because of a build failure in an imported package. Thus, we can’t assume that the build failure is in the test package itself.
2. A view that narrows down a large test log to "why did this particular test fail?" (a common feature of CI systems) should be able to report the text of the build failure, even if it wasn't in the test package itself. Thus, we need some way to connect each failing test package back to the root build failure.
3. Multiple test packages may fail because of a build failure in one common imported package. Preferably, the details of the build failure would appear only once in the log, regardless of how many tests fail because of it.
4. Consumers must be able to concatenate the Output fields to recover the original text output. Hence, the current "FAIL" line has to appear in some output event. This also forces us to report the details of each build failure only once, since that’s how it appears in the text output.
5. CI systems should be able to correlate the same test package across test logs from different commits, even if the test package runs in some logs and doesn’t run because of build failures in others. Since TestEvents use package paths and BuildEvents use package IDs, that means the CI system needs some way to get the package path of the failing test. Preferably this can be done without parsing the package ID.
6. Sequential test order should still be recoverable from the log with build failures. Hence, the "start" TestEvents should still appear and be in order.

Example

Given a test package with a simple build error, currently go test -json emits the following:

# github.com/aclements/gotest/testdata/builderror [github.com/aclements/gotest/testdata/builderror.test]
./main_test.go:6:2: undefined: x
FAIL	github.com/aclements/gotest/testdata/builderror [build failed]

This is, in fact, identical to the output without the -json flag.

With the proposal, go test -json would instead print:

{"ImportPath":"github.com/aclements/gotest/testdata/builderror [github.com/aclements/gotest/testdata/builderror.test]","Action":"build-output","Output":"./main_test.go:6:2: undefined: x\n"}
{"ImportPath":"github.com/aclements/gotest/testdata/builderror [github.com/aclements/gotest/testdata/builderror.test]","Action":"build-fail"}
{"Time":"...","Action":"start","Package":"github.com/aclements/gotest/testdata/builderror"}
{"Time":"...","Action":"output","Package":"github.com/aclements/gotest/testdata/builderror","Output":"FAIL\tgithub.com/aclements/gotest/testdata/builderror [build failed]\n"}
{"Time":"...","Action":"fail","Package":"github.com/aclements/gotest/testdata/builderror","FailedBuild":"github.com/aclements/gotest/testdata/builderror [github.com/aclements/gotest/testdata/builderror.test]"}

Open issues and questions

Should we emit BuildEvents for successful package builds, like we emit TestEvents for passing tests? Those could include useful information like timing. If we think of this as akin to verbose output, do we emit start events, too? I did not include these in the proposal because they aren’t necessary to solve the immediate problems of processing go test -json output, but we could add these events in the future.

We could emit environmental and module-level errors in JSON. Above I proposed that these continue to be printed in text to stderr because they are fatal and happen before any build errors that would be printed in JSON, making them easy for tooling to process as text. From an implementation standpoint, there are also simply a huge number of possible causes for early, fatal errors, not all of which are even under the control of the go command, making it difficult to capture all of them. If we wanted to capture more errors, we would have to complicate BuildEvent to specify their sources, probably by making the Package field optional and adding an optional Module field. Module-level errors would populate the Module field and environmental errors would omit both fields. This all seems like unnecessary complexity.

BuildEvent.ImportPath and TestEvent.Package are defined slightly differently. There are good reasons for this (given above), and I chose different field names to help make this clear, but it may be confusing for consumers.

Finally, we could make the JSON build output much more structured, for example by encoding path and line information of errors as JSON fields. We chose to simply wrap the text output of the compiler because of the degree of variation in compiler output, including sub-errors and optional debug output.

[dmitshur]

Thanks for the detailed proposal.

I noticed currently there's no discussion of exit codes here. It's a small detail, but still fairly important in that each future consumer of the JSON will need to process the exit code before they can begin to process JSON events. Perhaps the proposal is to inherit the same exit codes as go build currently emits when there's no -json flag, or for a custom exit code mapping when -json is on: possibly always 0 when valid JSON is successfully emitted, or non-zero in specific cases even when JSON is written successfully, or something else.

To use the existing go mod download -json behavior as an example: its CLI requires the caller to parse JSON both if exit code is 0 and 1. go mod download -json returns exit code 0 if there were no download errors in any modules, and exit code 1 if at least one module had a non-zero error. In both cases, it writes JSON to stdout. (See #35380 in general and #35380 (comment) in particular.)

(This is a comparatively minor detail, I hope not to distract too much from the more important aspects of the proposal.)

[aclements]

I noticed currently there's no discussion of exit codes here.

I wasn't planning on differing from the go command's current exit codes. And there could be JSON output even if it exits with 0. Certainly if there are build failures, there will be JSON output and a non-zero exit code. But, for example, if you pass certain debug flags to the compiler, it will emit output that should appear in JSON even on a successful build. Also, if we do decide to emit events on "successful" builds, as I mentioned in the "Open issues" section, that would also potentially emit JSON and then exit with a 0 status, so I certainly wouldn't want to lock us in to some rule about the relationship between the exit status and emitting JSON.

[gopherbot]

Change https://go.dev/cl/529120 mentions this issue: cmd/go: avoid writing non-JSON "build failed" errors from 'go test -json'

[gopherbot]

Change https://go.dev/cl/529220 mentions this issue: cmd/go: delete showOutput, formatOutput, and processOutput.

[gopherbot]

Change https://go.dev/cl/529218 mentions this issue: cmd/go: consolidate showOutput, formatOutput, and processOutput into reportCmd

[gopherbot]

Change https://go.dev/cl/529219 mentions this issue: cmd/go: replace formatOutput/showOutput with reportCmd

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group