misc/cgo/testsanitizers: TestTSAN/tsan9 fails on arm64

[zhangfannie]

What version of Go are you using (go version)?

$ go version
go version devel +f4e32aeed1 Wed Oct 30 08:17:29 2019 +0000 linux/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/fanzha02/.cache/go-build"
GOENV="/home/fanzha02/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/fanzha02/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/fanzha02/work/go_project/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/fanzha02/work/go_project/golang/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/fanzha02/work/go_project/golang/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build943556601=/tmp/go-build -gno-record-gcc-switches"

What did you do?

run go tool dist test -run testsanitizers

What did you expect to see?

pass

What did you see instead?

../misc/cgo/testsanitizers

--- FAIL: TestTSAN (8.88s)
--- FAIL: TestTSAN/tsan9 (3.29s)
tsan_test.go:53: /tmp/TestTSAN118028569/tsan9 exited with signal: segmentation fault
FAIL
2019/10/30 10:55:54 Failed: exit status 1

[zhangfannie]

1. go build misc/cgo/testsanitizers/testdata/tsan9.go
2. used gdb tool and found src/runtime/sys_linux_arm64.s:300 MOVD g, (R22) cause segment fault.
3. The code is added by this CL https://go-review.googlesource.com/c/go/+/202759.

[ianlancetaylor]

CC @cherrymui

[cherrymui]

Thanks for the report, and sorry for the breakage.
@zhangfannie could you get a stack trace from gdb? This will be much helpful. The builder doesn't have gdb installed and debugging this kind of failure has been a hard time for me.

[shawndx]

By modifying the code in walltime1 and nanotime1 to bypass the backtrace breaking caused by RSP adjustment, we can get a little more stack trace info, please check if it makes any sense.

runtime.nanotime1 () at /home/xiaji01/local/go_tsan/src/runtime/sys_linux_arm64.s:297
297 MOVD g, (R22)
(gdb) bt
#0 runtime.nanotime1 () at /home/xiaji01/local/go_tsan/src/runtime/sys_linux_arm64.s:297
#1 0x00000000004b5034 in runtime.nanotime () at /home/xiaji01/local/go_tsan/src/runtime/time_nofake.go:19
#2 runtime.(cpuProfile).add (p=0x151bd00 <runtime.cpuprof>, gp=0xffffb8274900, stk=...)
at /home/xiaji01/local/go_tsan/src/runtime/cpuprof.go:106
#3 0x00000000004e44c4 in runtime.sigprof (pc=5141596, sp=281473682498352, lr=5113104, gp=0xffffb8274900,
mp=0xffffb8272380) at /home/xiaji01/local/go_tsan/src/runtime/proc.go:3892
#4 0x00000000004ed7d4 in runtime.sighandler (sig=27, info=0xffffbe663208, ctxt=0xffffbe663288, gp=0xffffb8274900)
at /home/xiaji01/local/go_tsan/src/runtime/signal_linux_arm64.go:51
#5 0x00000000004ecde8 in runtime.sigtrampgo (sig=27, info=0xffffbe663208, ctx=0xffffbe663288)
at /home/xiaji01/local/go_tsan/src/runtime/signal_unix.go:359
#6 0x0000000000506624 in runtime.sigtramp () at /home/xiaji01/local/go_tsan/src/runtime/sys_linux_arm64.s:440
#7 0x0000000000424260 in __tsan::CallUserSignalHandler(__tsan::ThreadState, bool, bool, bool, int, my_siginfo_t*, void*)
()
#8 0x0000000000426314 in __tsan::ProcessPendingSignals(__tsan::ThreadState*) ()
#9 0x0000000000426400 in __tsan::ScopedInterceptor::~ScopedInterceptor() [clone .part.28] ()
#10 0x000000000045dc74 in mmap ()
#11 0x00000000005541e8 in x_cgo_mmap ()

(gdb) info reg
x0 0x1 1
x1 0xffffb3dbd9b0 281473699273136
x2 0x15364f8 22242552
x3 0x0 0
x4 0x0 0
x5 0xffffb3dbdaa8 281473699273384
x6 0x1 1
x7 0x0 0
x8 0x0 0
x9 0x0 0
x10 0x5e38c8 6174920
x11 0x6764e0 6776032
x12 0x0 0
x13 0x0 0
x14 0x0 0
x15 0x0 0
x16 0x6 6
x17 0xffffbf6a49f8 281473893157368
x18 0x9f 159
x19 0x664000 6701056
x20 0xffffb3dbd9b0 281473699273136
x21 0xffffb8288e00 281473771408896
x22 0xffffb35be900 281473690888448
x23 0x0 0
x24 0x1 1
x25 0x0 0
x26 0x0 0
x27 0x0 0
x28 0xffffb828a900 281473771415808
x29 0xffffb3dbd9a8 281473699273128
x30 0x4b5034 4935732
sp 0xffffb3dbd9b0 0xffffb3dbd9b0
pc 0x506430 0x506430 <runtime.nanotime1+40>
cpsr 0x60000000 [ EL=0 C Z ]
fpsr 0x10 16
fpcr 0x0 0

(gdb) px $x21
$2 = 0xffffb8288e00

[cherrymui]

Thanks. This is really helpful. I'll look into it.

The fault is accessing (R22), which is 0xffffb35be900. Is this a valid address? Could you also print the memory mapping?

[cherrymui]

Could you test CL http://golang.org/cl/204441 ? Thanks!

[gopherbot]

Change https://golang.org/cl/204441 mentions this issue: runtime: setg after sigFetchG

[cherrymui]

As a side note, I ran the sanitizer tests on the Linux/ARM64 builder, this is what I got:

root@57c3a7b17a85:/workdir/go/misc/cgo/testsanitizers# go test -v
=== RUN   TestShared
=== PAUSE TestShared
=== RUN   TestMSAN
=== PAUSE TestMSAN
=== RUN   TestTSAN
=== PAUSE TestTSAN
=== CONT  TestShared
=== CONT  TestTSAN
=== CONT  TestMSAN
=== RUN   TestShared/msan_shared
=== PAUSE TestShared/msan_shared
=== RUN   TestShared/tsan_shared
=== PAUSE TestShared/tsan_shared
=== CONT  TestShared/msan_shared
=== CONT  TestShared/tsan_shared
    TestMSAN: msan_test.go:16: gcc: error: unrecognized argument to -fsanitize= option: 'memory'
        gcc: error: unrecognized argument to -fsanitize= option: 'memory'
        
--- SKIP: TestMSAN (0.18s)
    TestShared/msan_shared: cshared_test.go:47: gcc: error: unrecognized argument to -fsanitize= option: 'memory'
        gcc: error: unrecognized argument to -fsanitize= option: 'memory'
        
    TestTSAN: tsan_test.go:16: `gcc -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build570263858=/tmp/go-build -gno-record-gcc-switches -fsanitize=thread -fPIC -fsanitize=thread -fPIC -static-libtsan -o /tmp/thread291603918/return0 /tmp/thread291603918/return0.c` generated broken executable: exit status 66
        FATAL: ThreadSanitizer CHECK failed: ../../../../src/libsanitizer/tsan/tsan_platform_linux.cc:264 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff)
--- SKIP: TestTSAN (0.54s)
    TestShared/tsan_shared: cshared_test.go:47: `gcc -fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build570263858=/tmp/go-build -gno-record-gcc-switches -fsanitize=thread -fPIC -fsanitize=thread -fPIC -static-libtsan -o /tmp/thread291603918/return0 /tmp/thread291603918/return0.c` generated broken executable: exit status 66
        FATAL: ThreadSanitizer CHECK failed: ../../../../src/libsanitizer/tsan/tsan_platform_linux.cc:264 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff)
--- PASS: TestShared (0.18s)
    --- SKIP: TestShared/msan_shared (0.01s)
    --- SKIP: TestShared/tsan_shared (0.36s)
PASS
ok  	misc/cgo/testsanitizers	0.547s

Would it be possible to install a proper C toolchain that supports the sanitizer, so the tests won't be skipped?
cc @bradfitz

[zhangfannie]

The tsan9 case still fails with the fix CL https://golang.org/cl/204441 .

[bradfitz]

Would it be possible to install a proper C toolchain that supports the sanitizer, so the tests won't be skipped?

@cherrymui, see https://github.com/golang/build/blob/master/env/linux-arm64/Dockerfile ... what's not proper there? It has gcc 8.3.0 (https://packages.debian.org/buster/gcc)

[shawndx]

tsan9.pmap.txt

Please check the enclosed pmap output (without your recent CL, going to try it later). Thanks.

[shawndx]

The case still crashes with a similar stack trace, now 'g' is a valid pointer and usually indicates a 'gsignal' (by setg(g.m.gsignal) in sigtrampgo), but its stack bottom is invalid address, post the details in CL #204441. Thanks.

[cherrymui]

Thanks. How about CL http://golang.org/cl/204519 ?