misc/cgo/test: morestack on g0 on Solaris under "unlimit stacksize"

[fazalmajid]

At least on my SmartOS box running joyent_20150514T133314Z:

##### ../misc/cgo/test
SIGTRAP: trace trap (not reset when caught)
PC=0x4a8236 m=0
signal arrived during cgo execution

goroutine 25 [running, locked to thread]:
runtime.morestack()
        /usr/local/go/src/runtime/asm_amd64.s:302 +0x26 fp=0xfffffd7fffdeed88 sp=0xfffffd7fffdeed80
created by testing.RunTests
        /usr/local/go/src/testing/testing.go:561 +0x86d

goroutine 1 [chan receive]:
runtime.gopark(0x69fe68, 0xc82009e418, 0x6535a0, 0xc, 0x17, 0x3)
        /usr/local/go/src/runtime/proc.go:185 +0x179 fp=0xc820059b28 sp=0xc820059b00
runtime.goparkunlock(0xc82009e418, 0x6535a0, 0xc, 0x17, 0x3)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc820059b60 sp=0xc820059b28
runtime.chanrecv(0x5bb9a0, 0xc82009e3c0, 0xc820059d78, 0xc820059c01, 0x4c0000)
        /usr/local/go/src/runtime/chan.go:448 +0x47b fp=0xc820059c20 sp=0xc820059b60
runtime.chanrecv1(0x5bb9a0, 0xc82009e3c0, 0xc820059d78)
        /usr/local/go/src/runtime/chan.go:349 +0x2b fp=0xc820059c50 sp=0xc820059c20
testing.RunTests(0x69fb08, 0x76ec20, 0x38, 0x38, 0x481401)
        /usr/local/go/src/testing/testing.go:562 +0x8ad fp=0xc820059dd0 sp=0xc820059c50
testing.(*M).Run(0xc820059ef8, 0xc820014140)
        /usr/local/go/src/testing/testing.go:494 +0x70 fp=0xc820059e58 sp=0xc820059dd0
main.main()
        _/home/majid/build/go-1.5/misc/cgo/test/_test/_testmain.go:166 +0x116 fp=0xc820059f50 sp=0xc820059e58
runtime.main()
        /usr/local/go/src/runtime/proc.go:111 +0x2cb fp=0xc820059fa0 sp=0xc820059f50
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820059fa8 sp=0xc820059fa0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820048fb8 sp=0xc820048fb0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x69fe68, 0x7807f0, 0x653e40, 0xf, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x179 fp=0xc820034758 sp=0xc820034730
runtime.goparkunlock(0x7807f0, 0x653e40, 0xf, 0xc820000114, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc820034790 sp=0xc820034758
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:152 +0xc1 fp=0xc8200347c0 sp=0xc820034790
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200347c8 sp=0xc8200347c0
created by runtime.init.4
        /usr/local/go/src/runtime/proc.go:141 +0x2b

goroutine 3 [GC sweep wait]:
runtime.gopark(0x69fe68, 0x780980, 0x64c4d0, 0xd, 0x46ac14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x179 fp=0xc820034f48 sp=0xc820034f20
runtime.goparkunlock(0x780980, 0x64c4d0, 0xd, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc820034f80 sp=0xc820034f48
runtime.bgsweep(0xc82001e070)
        /usr/local/go/src/runtime/mgcsweep.go:51 +0xba fp=0xc820034fb8 sp=0xc820034f80
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820034fc0 sp=0xc820034fb8
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:205 +0x53

goroutine 4 [finalizer wait]:
runtime.gopark(0x69fe68, 0x7a4470, 0x653ab0, 0xe, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x179 fp=0xc820035718 sp=0xc8200356f0
runtime.goparkunlock(0x7a4470, 0x653ab0, 0xe, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc820035750 sp=0xc820035718
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:154 +0xb3 fp=0xc8200357c0 sp=0xc820035750
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200357c8 sp=0xc8200357c0
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:135 +0x60

goroutine 5 [syscall]:
runtime.notetsleepg(0x7a4660, 0xffffffffffffffff, 0x1)
        /usr/local/go/src/runtime/lock_sema.go:264 +0x8f fp=0xc820035f40 sp=0xc820035f00
runtime.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:111 +0x132 fp=0xc820035f78 sp=0xc820035f40
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18 fp=0xc820035fc0 sp=0xc820035f78
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820035fc8 sp=0xc820035fc0
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [syscall, locked to thread]:
runtime.cgocall(0x44d620, 0xc8200367b0, 0x0)
        /usr/local/go/src/runtime/cgocall.go:120 +0x11d fp=0xc820036778 sp=0xc820036748
_/home/majid/build/go-1.5/misc/cgo/test._Cfunc_usleep(0xc800002710, 0xc800000000)
        ??:0 +0x35 fp=0xc8200367b0 sp=0xc820036778
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200367b8 sp=0xc8200367b0
created by _/home/majid/build/go-1.5/misc/cgo/test.lockOSThreadCallback
        /home/majid/build/go-1.5/misc/cgo/test/issue3775.go:35 +0x100

rax    0xc8200bc000
rbx    0x7812e0
rcx    0xffffdd7fffaa2a40
rdx    0xc820030588
rdi    0x49e4c0
rsi    0x780d80
rbp    0xc8200305d0
rsp    0xfffffd7fffdeed80
r8     0xc8200bc000
r9     0xc820030668
r10    0x1
r11    0xfffffffffbc05648
r12    0x10
r13    0x69ab51
r14    0x1e
r15    0x8
rip    0x4a8236
rflags 0x246
cs     0x53
fs     0x0
gs     0x0
exit status 2
FAIL    _/home/majid/build/go-1.5/misc/cgo/test 0.009s
2015/08/19 15:07:27 Failed: exit status 1

[fazalmajid]

The hardware is a HP Z230, 3.5GHz Intel Xeon E3-1270v3 Quad Core (8 with Hyperthreading), 8GB of RAM allocated to the zone.

[binarycrusader]

For what it's worth, this works on Oracle Solaris, so there's likely something subtle here.

[binarycrusader]

@fazalmajid do you have gcc installed? If so, what version?

[minux]

FYI, the solaris builder is running smartos joyent_20141212T011737Z and the test passes.

[fazalmajid]

Yes, GCC 5.2.0 built from source, defaulting to amd64. I build the bootstrap Go-1.4.2 using:
gcc -O2 -std=gnu11 -D__EXTENDED__=1 -D_XPG6=1
(the XPG and EXTENDED flags being required as GCC 5.2 defaults to C11 and some Solaris headers don't like that without stipulating XPG6 compliance.

[fazalmajid]

I can try and bisect the tests to find out which one is triggering the SIGTRAP, if you'd like.

[rsc]

Bisecting would be great. Thanks.

[binarycrusader]

@fazalmajid is this failure in comparison to the release version of Go 1.4 or did this test pass previously at some point with Go 1.5 while it was in development?

I ask, because Go 1.5 is the first version of Go where cgo support (and thus the cgo test that's failing) was enabled for Solaris:

commit 2b90c3e8edf36c0a76545fa13c195bca68a29420
Author: Aram Hăvărneanu < <aram@mgk.ro>
Date:   Mon Mar 30 23:15:51 2015 +0200

    go/build: enable cgo by default on solaris/amd64

    Change-Id: I0110b01fe4c64851ac2cfb5a92c31ce156831bc8
    Reviewed-on: https://go-review.googlesource.com/8265
    Reviewed-by: Minux Ma <minux@golang.org>

Also, for the record, why are you building Go with -std=c11? It isn't required, and it's unlikely that C11 or C++11 will work as expected on SmartOS. You should have been able to build Go 1.4 with gcc using -std=c89 or -std=gnu89 and without the other special flags you added.

[fazalmajid]

Really strange: running the tests manually works:
env GOPATH=pwd go tool dist test -no-rebuild
it's when run under all.bash that it fails.

@binarycrusader: I am passing those flags to the bootstrap Go 1.4.2. Without them, I get the error below (this is on an OpenIndiana oi151a1 machine, but I get the same results on SmartOS). If you use an older version of GCC than 5.2, you can probably do without. -std=gnu11 is the default for GCC 5.x.
Go 1.5 itself I build using just env CC=gcc, but if I understand correctly, Go 1.5 does not use gcc at all but its own toolchain (written in Go and compiled using the bootstrap.

(cd go-1.4.2/src; env GOROOT=`pwd`/go-1.4.2 CC="gcc" ./all.bash)
# Building C bootstrap tool.
cmd/dist

# Building compilers and Go bootstrap tool for host, solaris/amd64.
lib9
In file included from /usr/include/inttypes.h:41:0,
                 from /home/majid/build/go-1.4.2/include/u.h:60,
                 from /home/majid/build/go-1.4.2/src/lib9/await.c:31:
/usr/local/lib/gcc/x86_64-pc-solaris2.11/5.2.0/include-fixed/sys/feature_tests.h:362:2: error: #error "Compiler or options invalid for pre-UNIX 03 X/Open applications  and pre-2001 POSIX applications"
 #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
  ^
In file included from /usr/include/inttypes.h:41:0,
                 from /home/majid/build/go-1.4.2/include/u.h:60,
                 from /home/majid/build/go-1.4.2/src/lib9/_p9dir.c:28:
/usr/local/lib/gcc/x86_64-pc-solaris2.11/5.2.0/include-fixed/sys/feature_tests.h:362:2: error: #error "Compiler or options invalid for pre-UNIX 03 X/Open applications  and pre-2001 POSIX applications"
 #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
  ^
In file included from /usr/include/inttypes.h:41:0,
                 from /home/majid/build/go-1.4.2/include/u.h:60,
                 from /home/majid/build/go-1.4.2/src/lib9/_exits.c:28:
/usr/local/lib/gcc/x86_64-pc-solaris2.11/5.2.0/include-fixed/sys/feature_tests.h:362:2: error: #error "Compiler or options invalid for pre-UNIX 03 X/Open applications  and pre-2001 POSIX applications"
 #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
  ^
In file included from /usr/include/inttypes.h:41:0,
                 from /home/majid/build/go-1.4.2/include/u.h:60,
                 from /home/majid/build/go-1.4.2/src/lib9/atoi.c:28:
/usr/local/lib/gcc/x86_64-pc-solaris2.11/5.2.0/include-fixed/sys/feature_tests.h:362:2: error: #error "Compiler or options invalid for pre-UNIX 03 X/Open applications  and pre-2001 POSIX applications"
 #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
  ^
go tool dist: FAILED: gcc -Wall -Wstrict-prototypes -Wextra -Wunused -Wno-sign-compare -Wno-missing-braces -Wno-parentheses -Wno-unknown-pragmas -Wno-switch -Wno-comment -Wno-missing-field-initializers -Werror -fno-common -ggdb -pipe -Wuninitialized -O2 -fmessage-length=0 -c -m64 -I /home/majid/build/go-1.4.2/include -DPLAN9PORT -I /home/majid/build/go-1.4.2/src/lib9 -o $WORK/atoi.o /home/majid/build/go-1.4.2/src/lib9/atoi.c
go tool dist: FAILED: gcc -Wall -Wstrict-prototypes -Wextra -Wunused -Wno-sign-compare -Wno-missing-braces -Wno-parentheses -Wno-unknown-pragmas -Wno-switch -Wno-comment -Wno-missing-field-initializers -Werror -fno-common -ggdb -pipe -Wuninitialized -O2 -fmessage-length=0 -c -m64 -I /home/majid/build/go-1.4.2/include -DPLAN9PORT -I /home/majid/build/go-1.4.2/src/lib9 -o $WORK/_exits.o /home/majid/build/go-1.4.2/src/lib9/_exits.c
go tool dist: FAILED: gcc -Wall -Wstrict-prototypes -Wextra -Wunused -Wno-sign-compare -Wno-missing-braces -Wno-parentheses -Wno-unknown-pragmas -Wno-switch -Wno-comment -Wno-missing-field-initializers -Werror -fno-common -ggdb -pipe -Wuninitialized -O2 -fmessage-length=0 -c -m64 -I /home/majid/build/go-1.4.2/include -DPLAN9PORT -I /home/majid/build/go-1.4.2/src/lib9 -o $WORK/await.o /home/majid/build/go-1.4.2/src/lib9/await.c
go tool dist: FAILED: gcc -Wall -Wstrict-prototypes -Wextra -Wunused -Wno-sign-compare -Wno-missing-braces -Wno-parentheses -Wno-unknown-pragmas -Wno-switch -Wno-comment -Wno-missing-field-initializers -Werror -fno-common -ggdb -pipe -Wuninitialized -O2 -fmessage-length=0 -c -m64 -I /home/majid/build/go-1.4.2/include -DPLAN9PORT -I /home/majid/build/go-1.4.2/src/lib9 -o $WORK/_p9dir.o /home/majid/build/go-1.4.2/src/lib9/_p9dir.c

[binarycrusader]

@fazalmajid right, I was suggesting that instead of building Go 1.4 with -std=gnu11, you build it with -std=gnu89. I know that gnu11/c11 is the default for GCC 5.2, I was suggesting you specify the older standard explicitly to see what the result was.

Also, as I asked before, what version of Go did you successfully run all tests with before this failure? Was 1.4 the last version you tested?

[fazalmajid]

Yes, 1.4.2 was the last version I tested. Go 1.4.2 builds successfully with gcc -std=gnu99 but misc/cgo/test has the same failure when building Go 1.5 using the gnu89 Go-1.4.2.

As for running the test manually, I found out what is going wrong (or right): it's testing $GOROOT_FINAL/misc/cgo/test/cgo_test.go (that I patched to comment out all cgo tests so I could install), not the unpatched one in the build directory which fails. When I copy the unpatched one back to $GOROOT_FINAL, I can reproduce the error and will get back to bisecting the cgo tests.

[fazalmajid]

@rsc any of these tests in misc/cgo/cgo_test.go will cause the failure on OI-151a1/GCC-5.2:

func TestCallback(t *testing.T)              { testCallback(t) }
func TestCallbackGC(t *testing.T)            { testCallbackGC(t) }
func TestCallbackPanic(t *testing.T)         { testCallbackPanic(t) }
func TestCallbackPanicLoop(t *testing.T)     { testCallbackPanicLoop(t) }
func TestCallbackPanicLocked(t *testing.T)   { testCallbackPanicLocked(t) }
func TestBlocking(t *testing.T)              { testBlocking(t) }

[ianlancetaylor]

The trend there is pretty obvious: calls from Go -> C -> Go fail.