cmd/compile: some x/sys versions no longer build due to "go:linkname must refer to declared function or variable"

[mvdan]

Using an x/sys version that is too old for a Go version may result in build errors, and that has clearly confused at least hundreds of Go users in the past year or so:

• https://stackoverflow.com/questions/71507321/go-1-18-build-error-on-mac-unix-syscall-darwin-1-13-go253-golinkname-mus
• go:linkname must refer to declared function or variable #51706

Given that x/sys uses //go:linkname directives into unexported Go APIs, it is fairly tightly bound to a set of Go versions. If the Go version is too old or too new, it's entirely possible that the build may fail like the example above on GOOS=darwin.

If the x/sys version is too old, the solution is pretty easy: go get golang.org/x/sys to update the dependency. That is not obvious given the error message, though. And I bet that a good number of those end users are only indirectly depending on x/sys, so they might not even know what the library does or why it's being built.

Similarly, if one tries to use a recent version of x/sys with an ancient version of Go, I imagine the build may also fail in weird and confusing ways, though this side of the coin doesn't seem to be as common of a problem.

I think we should try to make these errors more intuitive. As a strawman idea, I suggest that we force the current master version of x/sys to fail to build on Go 1.17 and older, as well as Go 1.21 and newer, via build tags such as:

$ cat go_older.go
//go:build !go1.18
// +build !go1.18

"this version of golang.org/x/sys supports Go 1.18 and Go 1.19; upgrade your Go version or, if you cannot, downgrade x/sys"
$ cat go_newer.go
//go:build go1.21

"this version of golang.org/x/sys supports Go 1.18 and Go 1.19; upgrade x/sys via: go get golang.org/x/sys"

The string literals would cause a parse error which would show the string to the user, as a way of customizing the error message:

$ go1.16.15 build go_older.go 
go_older.go:4:1: expected 'package', found "this version of golang.org/x/sys supports Go 1.18 and Go 1.19; upgrade your Go version or, if you cannot, downgrade x/sys"

We would not error on Go 1.20, as that is the current tip/master version, and we are continuously supporting and testing it. Erroring on Go 1.21 today is perhaps unnecessary, but it's the only way to prepare the current master versions for a future when Go 1.21 is widely used and may not work with today's versions of x/sys. When Go 1.20 is released and we start working towards 1.21, the build tags in x/sys master would be advanced by one version.

A potential alternative would be a toolchain line in go.mod, per #55092. However, that might only allow us to set a minimum Go toolchain version, and presumably not a maximum. It's also not yet implemented, nor is it understood by older Go versions. We definitely want our solution to make Go 1.16 fail to build the latest x/sys master version, for example.

cc @golang/runtime @ianlancetaylor @bradfitz @tklauser per https://dev.golang.org/owners

[mvdan]

We can always make this a proposal if needed, but hopefully this is not a controversial idea. If we agree that this is a valid problem, but we're not happy with the solution, I am also happy to re-purpose this issue to simply be a UX bug report.

[bcmills]

Given that x/sys uses //go:linkname directives into unexported Go APIs

Why does it need to do that? Perhaps we should file a proposal for whatever new standard-library APIs would be needed to avoid it. (Looks like it's mostly just variations on syscall.syscall.* wrappers?)

If the Go version is too old or too new, it's entirely possible that the build may fail like the example above on GOOS=darwin.

“Too old” should be covered by the go directive: if x/sys is changed to require an API that is only available in a more recent Go release (and doesn't include fallback code to maintain compatibility with older releases), its go.mod file should name that release or higher.

“Too new” seems like a violation of the compatibility policy. Since x/sys is part of the Go project and the compatibility policy is essentially “packages that used to work should continue to work”, it seems to me that — barring some mechanism to automatically upgrade x/sys (like the one described in the second half of #30241) — we should treat any internal APIs that x/sys requires as de facto exported and subject to compatibility requirements.

[mvdan]

Why does it need to do that?

I'm not sure; I've never developed that module. Removing the coupling would indeed be best.

“Too old” should be covered by the go directive: if x/sys is changed to require an API that is only available in a more recent Go release (and doesn't include fallback code to maintain compatibility with older releases), its go.mod file should name that release or higher.

I guess you're right. I think we can largely put this concern aside, because most people run into "Go version too new" instead.

“Too new” seems like a violation of the compatibility policy.

I think that would make https://go-review.googlesource.com/c/go/+/333109/ the breaking change that we could reconsider. With it, newer Go versions fail to build older x/sys/unix packages which still used the now-banned mechanism. Perhaps the limit should only be applied when -lang is go1.18 or later, for example.

[bcmills]

I think that would make https://go-review.googlesource.com/c/go/+/333109/ the breaking change that we could reconsider.

Yep, looks like it. Ideally that change in error behavior should have been conditional on the go version declared in the go.mod file. 😩

[mvdan]

cc @josharian @mdempsky

[mvdan]

I have repurposed this issue to be about cmd/compile's apparent regression.

[mvdan]

Thinking outloud, if we do resolve this regression somehow, we should probably backport the fix to Go 1.18.x and 1.19.x - otherwise the affected users are stuck on 1.17.x until 1.20 is released. They can always update x/sys, and I imagine most can do that, but that might be tricky in some cases like go install foo.com/bar@latest commands to install third party software.

[cherrymui]

I'd argue the opposite. The way x/sys accesses the internal details of the runtime and standard library using unsafe construct and assembly code is not covered by the compatibility guarantee. This makes it very difficult to keep old versions of x/sys work while Go evolves. Ideally x/sys shouldn't do this, but perhaps that is not possible? Maybe the API boundary between x/sys and the Go standard library + toolchain is not ideal. Unfortunately it is hard to co-evolve x/sys and the Go toolchain+std (unlike, say, the compiler and runtime). This is what the original post is trying to resolve, as well as #55092 and #55090.

Another example is that in Go 1.17 when we switched to the register ABI, even older versions of x/sys also broke on some platforms, due to the use of the assembly code. Even with our best effort to not break assembly code in our design (e.g. automatic wrapper generation), it is still very hard (if not impossible) to not break, as the assembly code is too sensitive to details like stack layout. (I guess one way not to break is in the compiler to hard-code golang.org/x/sys import path to use old ABI, but that doesn't sound right.) Do we want to back off register ABI, then?

Again, as the original post says, I think the problem is the difficulty of co-evolving x/sys and Go toolchain+std, or updating them at ~same time. And the solution discussed in the original post, and/or #55092, #55090, is probably a good direction. Thanks.