cmd/link: internal link issue with outline atomics feature on arm64

[shawndx]

What version of Go are you using (go version)?

$ go version
tip

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/xiaji01/.cache/go-build"
GOENV="/home/xiaji01/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/xiaji01/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/xiaji01/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/xiaji01/util/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/xiaji01/util/go/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build044102122=/tmp/go-build -gno-record-gcc-switches"

What did you do?

GCC-9.4 introduced the outline atomics support for arm64 and the feature was made default since GCC-10.1. Go's internal linking mode doesn't work well with the feature, here is what have been spotted so far and a sample case, further analysis is ongoing.

With GCC-10.1 and default CGO flags, or GCC-9.4+with an additional "-moutline-atomics" flag, the sample case runs into a link issue with '-linkmode=internal', internal linking tries to resolve undefined external references in libgcc.a, while the problematic variable __aarch64_have_lse_atomic and its init function don't get resolved.

$ go build -ldflags='-linkmode=internal'
libgcc(.text): unknown symbol __aarch64_have_lse_atomics

lse.c

int test_cas_atomic_int (int *ptr, int *expected, int desired)
{
  return __atomic_compare_exchange_n (ptr, expected, desired, 0, 0, 0);
}

mult.c

#include <stdlib.h>
#include <stdio.h>

extern int test_cas_atomic_int (int *ptr, int *expected, int desired);

int test_cgo(int a, int b) {
  int old = a;
  int new = b;

  printf("before: old = %d, new = %d\n", old, new);
  test_cas_atomic_int(&old, &old, new);
  printf("after: old = %d, new = %d\n", old, new);
  return new * old + old;
}

mult.go

package main

/*
extern int test_cgo(int a, int b);
*/
import "C"
import (
 "fmt"
)

func main() {
 a := 3
 b := 5
 c := C.test_cgo(C.int(a), C.int(b))
 fmt.Println("test_cgo:", int(c))
}

What did you expect to see?

Trying to apply GCC-10 and its new LSE features to CGO part, expect they work smoothly.

What did you see instead?

Internal linking doesn't work.

[shawndx]

Wondering if intentionally loading the lse-init.o from libgcc.a, if it's present and other criteria satisfies, and letting cgo call the init routine at start-up a right way.
@ianlancetaylor

[shawndx]

more details regarding reproducing the issue:

1. GCC-9.4+
   A. go build -ldflags='-linkmode=internal', no issue
   B. CGO_CFLAGS="-g -O2 -moutline-atomics" go build -ldflags='-linkmode=internal', error out

2. GCC-10.1
   A. go build -ldflags='-linkmode=internal', error out
   B. CGO_CFLAGS="-g -O2 -mno-outline-atomics" go build -ldflags='-linkmode=internal', no issue

[ianlancetaylor]

It's important for internal linking to work when using the standard library. We don't consider it important for internal linking to work when using cgo code that is not in the standard library. There are always going to be cases where it fails. This is probably just another of those cases.

Why do you want to use internal linking?

[shawndx]

@ianlancetaylor It was exposed by ./all.bash, and could be reproduced by running "go test -ldflags='-linkmode=internal' under misc/cgo/test, then we created the sample case to analyze,here is the screen shot of ./all.bash, we didn't set any additional flags/variables.

By the way, given the following error message, is there any way to help identify which test causes the issue, thought it should be issue5242 and failed to reproduce the issue by extracting relevant code into a small case.

Thanks.

===============================================================

# misc/cgo/test
./test.go: In function 'issue5242':
./test.go:376:5: note: parameter passing for argument of type 'bar' changed in GCC 9.1
376 | int issue5242(foo f, bar b) {
       |     ^~~~~~~~~
# misc/cgo/test.test
libgcc(.text): unknown symbol __aarch64_have_lse_atomics
FAIL	misc/cgo/test [build failed]

[ianlancetaylor]

Thanks. I misunderstood. Sounds like arm64 internal linking no longer works with GCC 10, at least if the C code calls any atomic instructions. The simple fix might be to add "arm64" to the cases for which (*tester).internalLink returns false in cmd/dist/test.go.

[shawndx]

Thanks @ianlancetaylor , have a few more questions.

1. Did you see any value and feasibility to have internal linking work with the new outline atomics?
2. Besides the fix to dist test, I wonder if a meaningful message is expected rather than the "unknown symbol" text for user's cgo program, assume we decide not supporting internal linking with the outline atomics being enabled.

[ianlancetaylor]

Looking at the implementation of -moutline-atomics, I now wonder why it doesn't work. The Go linker in internal linking mode will normally link in libgcc, which is whether the required symbols are defined. So it is probably worth somebody looking into exactly what is going wrong.

[cherrymui]

I wonder if it has something to do with the symbol __aarch64_have_lse_atomics being marked as hidden in libgcc. It is not surprising to me if there is some bugs in the linker with hidden symbols. In the linker there are some scary TODOs with VisibilityHidden here and there, and, admittedly I don't really understand them. I'll take a careful look... (In the case of internal linking with libgcc.a, I don't know why VisibilityHidden even matters...)

[cherrymui]

Out of curiosity, why GCC 10 defaults to outline atomics? (Seems a weird decision to me. I must have missed something...)

[shawndx]

Looking at the implementation of -moutline-atomics, I now wonder why it doesn't work. The Go linker in internal linking mode will normally link in libgcc, which is whether the required symbols are defined. So it is probably worth somebody looking into exactly what is going wrong.

Thanks, I'd like to work on it, could you please help assign?

[shawndx]

I wonder if it has something to do with the symbol __aarch64_have_lse_atomics being marked as hidden in libgcc. It is not surprising to me if there is some bugs in the linker with hidden symbols. In the linker there are some scary TODOs with VisibilityHidden here and there, and, admittedly I don't really understand them. I'll take a careful look... (In the case of internal linking with libgcc.a, I don't know why VisibilityHidden even matters...)

That symbol is of R_AARCH64_ADR_PREL_PG_HI21 and R_AARCH64_ADD_ABS_LO12_NC types, adding support to hidden visibility of these two types in linker could resolve the symbol, but looks like more changes are needed (tried a simple fix and build worked but execution crashed), among of them might be a call to init_have_lse_atomics at proper place.
Wondering if force loading the object file where the symbol resides (lse-init.o) and calling init_have_lse_atomics in cgo start-up code is reasonable.