cmd/link: SIGSEGV running 'openshift-install version' for release-4.8 using external linking on PPC64LE

[hamzy]

What version of Go are you using (go version)?

$ go version
go version go1.16.3 linux/ppc64le

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

[root@mihawklp111 installer]# go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/go/src/github.com/openshift/installer/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build456481733=/tmp/go-build -gno-record-gcc-switches"

[root@mihawklp111 installer]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)
[root@mihawklp111 installer]# uname -a
Linux mihawklp111 4.18.0-240.el8.ppc64le #1 SMP Wed Sep 23 05:08:15 EDT 2020 ppc64le ppc64le ppc64le GNU/Linux

What did you do?

[root@mihawklp111 ~]# cd ~/go/src/github.com/openshift/
[root@mihawklp111 openshift]# git clone https://github.com/openshift/installer.git && cd installer
[root@mihawklp111 installer]# git checkout origin/release-4.8
...
HEAD is now at 52fef9e1d Merge pull request #4875 from multi-arch/ppc64le-link
[root@mihawklp111 installer]# hack/build.sh
[root@mihawklp111 installer]# bin/openshift-install version
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
goroutine 1 [running]:
github.com/aws/aws-sdk-go/service/devicefarm.init()
        /root/go/src/github.com/openshift/installer/vendor/github.com/aws/aws-sdk-go/service/devicefarm/errors.go:89 +0x24

What did you expect to see?

The version of the openshift installer.

What did you see instead?

A SIGSEGV

[cherrymui]

Does it work with a different version of Go, 1.16 or 1.16.1, or tip of the master branch? Issue #45564 is recently fixed (and not triggered in some earlier versions of Go). It is possible that you don't need to force external linking.

[cherrymui]

Or different versions of openshift?

[hamzy]

FYI, I've tried 1.16 and 1.16.1 and they both fail. Hopefully David Benoit can comment. He also reproduced the problem and investigated a little more thoroughly.

And the previous version of openshift installer had the trampoline problem.

[cherrymui]

It should not have trampoline problem if you use Go 1.16, 1.16.1, or tip, per #45564 . Could you try that? Thanks.

[dbenoit17]

@cherrymui I've been able to reproduce this issue, and can also confirm that combining
https://go-review.googlesource.com/c/go/+/314449/ with internal linking does not produce the error. This SIGSEGV seems to occur with external linking only.

[hamzy]

[root@mihawklp111 installer]# /bin/rm -rf /usr/local/go/; tar -C /usr/local -xzf /root/go1.16.1.linux-ppc64le.tar.gz
[root@mihawklp111 installer]# go version
go version go1.16.1 linux/ppc64le
[root@mihawklp111 installer]# git checkout 911eb7f69a1316c48e883d08644cda0e7304a373
[root@mihawklp111 installer]# hack/build.sh
...
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4518-g911eb7f69a1316c48e883d08644cda0e7304a373 -X github.com/openshift/installer/pkg/version.Commit=911eb7f69a1316c48e883d08644cda0e7304a373 -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
# github.com/openshift/installer/cmd/openshift-install
github.com/ovirt/go-ovirt.(*IconServiceGetRequest).Send: unexpected trampoline for shared or dynamic linking
...
/usr/local/go/pkg/tool/linux_ppc64le/link: too many errors
[root@mihawklp111 installer]# git checkout origin/release-4.8
Previous HEAD position was 911eb7f69 Merge pull request #4868 from wking/stable-4.8
HEAD is now at 52fef9e1d Merge pull request #4875 from multi-arch/ppc64le-link
[root@mihawklp111 installer]# hack/build.sh
...
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4520-g52fef9e1d7b765470e493670177eefd73bc27c6f -X github.com/openshift/installer/pkg/version.Commit=52fef9e1d7b765470e493670177eefd73bc27c6f -s -w -linkmode external' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
[root@mihawklp111 installer]# bin/openshift-install version
unexpected fault address 0x2fffffff4170
fatal error: fault     
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2fffffff4170 pc=0x1a69e95c]
                                                     
goroutine 1 [running, locked to thread]:                                                                                                                                                       
runtime.throw(0x217086d3, 0x5)                          
        /usr/local/go/src/runtime/panic.go:1117 +0x5c fp=0xc00149f4f8 sp=0xc00149f4b8 pc=0x152d1d6c                                                                                            
runtime.sigpanic()           
        /usr/local/go/src/runtime/signal_unix.go:741 +0x274 fp=0xc00149f540 sp=0xc00149f4f8 pc=0x152e9984
github.com/aws/aws-sdk-go/service/efs.init()
        /root/go/src/github.com/openshift/installer/vendor/github.com/aws/aws-sdk-go/service/efs/errors.go:194 +0x24 fp=0xc00149f5b0 sp=0xc00149f560 pc=0x18aa1764
runtime.doInit(0x29778280)
        /usr/local/go/src/runtime/proc.go:6265 +0xf0 fp=0xc00149f710 sp=0xc00149f5b0 pc=0x152e29a0
runtime.doInit(0x297c9440)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc00149f870 sp=0xc00149f710 pc=0x152e2914
runtime.doInit(0x29796cc0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc00149f9d0 sp=0xc00149f870 pc=0x152e2914
runtime.doInit(0x2978a3a0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc00149fb30 sp=0xc00149f9d0 pc=0x152e2914
runtime.doInit(0x297a3ae0)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc00149fc90 sp=0xc00149fb30 pc=0x152e2914
runtime.doInit(0x29780700)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc00149fdf0 sp=0xc00149fc90 pc=0x152e2914
runtime.doInit(0x297a6900)
        /usr/local/go/src/runtime/proc.go:6242 +0x64 fp=0xc00149ff50 sp=0xc00149fdf0 pc=0x152e2914
runtime.main()
        /usr/local/go/src/runtime/proc.go:208 +0x214 fp=0xc00149ffc0 sp=0xc00149ff50 pc=0x152d43d4
runtime.goexit()
        /usr/local/go/src/runtime/asm_ppc64x.s:869 +0x4 fp=0xc00149ffc0 sp=0xc00149ffc0 pc=0x15307b74
...

[cherrymui]

Thanks. Would it be possible to try Go 1.15.x ? Thanks.

cc @pmur @laboger

(Maybe openshift can stop forcing external linking after next Go 1.16.x and Go 1.17 release.)

[hamzy]

Either a trap or a trampoline, you take your pick :)

[root@mihawklp111 installer]# /bin/rm -rf /usr/local/go/; tar -C /usr/local -xzf /root/go1.15.11.linux-ppc64le.tar.gz
[root@mihawklp111 installer]# go version
go version go1.15.11 linux/ppc64le
[root@mihawklp111 installer]# git status                                                                                                                                                        HEAD detached at origin/release-4.8                                                                                                                                                            
nothing to commit, working tree clean
[root@mihawklp111 installer]# hack/build.sh
...
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4520-g52fef9e1d7b765470e493670177eefd73bc27c6f -X github.com/openshift/installer/pkg/version.Commit=52fef9e1d7b765470e493670177eefd73bc27c6f -s -w -linkmode external' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
[root@mihawklp111 installer]# bin/openshift-install version
unexpected fault address 0xffffffffffff3a60
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0xffffffffffff3a60 pc=0x1bbf3664]
...

[root@mihawklp111 installer]# git checkout 911eb7f69a1316c48e883d08644cda0e7304a373
Previous HEAD position was 52fef9e1d Merge pull request #4875 from multi-arch/ppc64le-link
HEAD is now at 911eb7f69 Merge pull request #4868 from wking/stable-4.8
[root@mihawklp111 installer]# hack/build.sh
...
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-4518-g911eb7f69a1316c48e883d08644cda0e7304a373 -X github.com/openshift/installer/pkg/version.Commit=911eb7f69a1316c48e883d08644cda0e7304a373 -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
# github.com/openshift/installer/cmd/openshift-install
github.com/ovirt/go-ovirt.(*ClusterServiceRemoveRequest).Send: unexpected trampoline for shared or dynamic linking
...

[pmur]

I am investigating. It looks like the external linker is inserted a TOC relative trampoline to runtime.makemap_small<1> from github.com/aws/aws-sdk-go/service/devicefarm.init I see a few other ld generated trampolines too. We aren't maintaining the TOC pointer in this build configuration.

[laboger]

This is the same problem we saw a while ago with some descriptions in #20497 and also a related discussion in #20492. We don't want ld to be inserting trampolines like this because of this issue -- by default Go doesn't maintain r2. The odd thing is that when I look at the openshift-installer binary I don't see any tramp symbols and the text section is not split unless I'm missing something. I'm not sure why we haven't hit this before either.

[cherrymui]

@pmur yes, this is also what I'm seeing. I wonder if it works with -buildmode=pie, which will save/restore R2.

@laboger it is not the Go linker inserting trampolines, but the external linker. I do see external linker inserted trampolines.