runtime: CL 481061 "runtime/cgo: store M for C-created thread in pthread key" causes C TSAN failures

[prattmic]

When built with C TSAN, x_cgo_getstackbound triggers race detection on g->stacklo because the synchronization is in Go, which isn't instrumented.

Hopefully simple repro coming soon.

cc @cherrymui @doujiang24

[gopherbot]

Change https://go.dev/cl/485316 mentions this issue: Revert "Revert "Revert "runtime: consolidate function descriptor definitions on PPC64"""

[gopherbot]

Change https://go.dev/cl/485317 mentions this issue: Revert "runtime: correct GoCheckBindM's C declaration in EnsureBindM test"

[gopherbot]

Change https://go.dev/cl/485275 mentions this issue: Revert "runtime/cgo: store M for C-created thread in pthread key"

[gopherbot]

Change https://go.dev/cl/485315 mentions this issue: Revert "runtime/cgo: use pthread_attr_get_np on Illumos"

[prattmic]

This is reproducible with this program:

// Copyright 2023 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

package main

/*
#cgo CFLAGS: -fsanitize=thread
#cgo LDFLAGS: -fsanitize=thread

#include <pthread.h>

void go_callback();

static void *thr(void *arg) {
    go_callback();
    return 0;
}

static void foo() {
    pthread_t th;
    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_setstacksize(&attr, 256 << 10);
    pthread_create(&th, &attr, thr, 0);
    pthread_join(th, 0);
}
*/
import "C"

import (
        "time"
)

//export go_callback
func go_callback() {
}

func main() {
        for i := 0; i < 2; i++ {
                go func() {
                        for {
                                C.foo()
                        }
                }()
        }

        time.Sleep(1000*time.Millisecond)
}

It only seems to trigger with TSAN v3, which is included with clang but not gcc on my machine:

$ CGO_CFLAGS=-fsanitize=thread CC=clang go build tsan.go
$ TSAN_OPTIONS="halt_on_error=1" ./tsan
==================
WARNING: ThreadSanitizer: data race (pid=423188)
  Write of size 8 at 0x7f368c0064e0 by thread T28:
    #0 x_cgo_getstackbound <null> (tsan14+0x527119) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #1 runtime.asmcgocall.abi0 /usr/local/google/home/mpratt/src/go/src/runtime/asm_amd64.s:902 (tsan14+0x5219e5) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #2 thr tsan14.cgo2.c (tsan14+0x525ec3) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)

  Previous write of size 8 at 0x7f368c0064e0 by thread T29:
    #0 x_cgo_getstackbound <null> (tsan14+0x527119) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #1 runtime.asmcgocall.abi0 /usr/local/google/home/mpratt/src/go/src/runtime/asm_amd64.s:902 (tsan14+0x5219e5) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #2 thr tsan14.cgo2.c (tsan14+0x525ec3) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)

  Thread T28 (tid=423217, running) created by main thread at:
    #0 pthread_create <null> (tsan14+0x45048d) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #1 foo tsan14.cgo2.c (tsan14+0x525e6f) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #2 _cgo_0dfa424d8f6e_Cfunc_foo <null> (tsan14+0x525dfe) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #3 runtime.asmcgocall.abi0 /usr/local/google/home/mpratt/src/go/src/runtime/asm_amd64.s:872 (tsan14+0x5219a7) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)

  Thread T29 (tid=423218, finished) created by thread T2 at:
    #0 pthread_create <null> (tsan14+0x45048d) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #1 foo tsan14.cgo2.c (tsan14+0x525e6f) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #2 _cgo_0dfa424d8f6e_Cfunc_foo <null> (tsan14+0x525dfe) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)
    #3 runtime.asmcgocall.abi0 /usr/local/google/home/mpratt/src/go/src/runtime/asm_amd64.s:872 (tsan14+0x5219a7) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3)

SUMMARY: ThreadSanitizer: data race (/usr/local/google/home/mpratt/src/go/misc/cgo/testsanitizers/testdata/tsan14+0x527119) (BuildId: 80075c8ad0035a8addeb81107abcda0c2aa5bce3) in x_cgo_getstackbound
==================

[prattmic]

The "race" occurs on the write to g->stacklo in https://go-review.git.corp.google.com/c/go/+/481061/2/src/runtime/cgo/gcc_stack_unix.c.

My understanding of what happens is:

1. T1 calls into Go, calls needm.
2. needm gets M1, and calls cgo_getstackbound, writing M1.g0.stack.lo.
3. T1 exits, calls dropm via thread destructor, which puts M1 back on the extra list.
4. T2 calls into Go, calls needm.
5. needm gets M2 since it is back on the extra list, and calls cgo_getstackbound, writing M1.g0.stack.lo.

I believe that TSAN sees this as a racing write because it is unaware of synchronization in the runtime preventing two threads from using the same M (lockextra).

I tried to resolve this by adding a dummy barrier just prior to the crosscall in pthread_thread_destructor, in an attempt to annotate the barrier that dropm provides, but it did not work.

diff --git a/src/runtime/cgo/gcc_libinit.c b/src/runtime/cgo/gcc_libinit.c
index 9676593211..ec95f90a18 100644
--- a/src/runtime/cgo/gcc_libinit.c
+++ b/src/runtime/cgo/gcc_libinit.c
@@ -142,6 +142,8 @@ pthread_key_destructor(void* g) {
                // We restore g by using the stored g, before dropm in runtime.cgocallback,
                // since the g stored in the TLS by Go might be cleared in some platforms,
                // before this destructor invoked.
+                _cgo_tsan_acquire();
+                _cgo_tsan_release();
                x_crosscall2_ptr(NULL, g, 0, 0);
        }
 }

cc @ianlancetaylor @cherrymui do either of you know if I am misunderstanding the semantics of _cgo_tsan_acquire/_cgo_tsan_release? Or perhaps I missed a case and there really is a race here.

[gopherbot]

Change https://go.dev/cl/485500 mentions this issue: runtime/cgo: store M for C-created thread in pthread key