runtime/cgo: "pthread_create failed" during syscall.Exec on Darwin/OpenBSD/DragonFly

[bcmills]

Twice this week I've helped people debug panics in Go programs due to runtime/cgo: pthread_create failed: Resource temporarily unavailable. Both times, it has turned out that they were calling syscall.Exec and it happened to execute concurrently with a pthread_create for a background goroutine, causing the pthread_create to fail with EAGAIN.

There have been a couple of previous threads on go-nuts involving similar issues.

We could consider modifying the runtime to stop the world and/or shutdown threads during calls to syscall.Exec, but that seems like a fair amount of work and the syscall package is frozen / deprecated anyway.

As a simpler step, I think we should have vet warn about any calls to syscall.Exec from outside the standard library.

[ianlancetaylor]

If you can write a test case for the problem, we will simply change runtime/cgo to retry the pthread_create if it fails with EAGAIN. The only reason I didn't do that years ago is that I wasn't able to write a test case.

[bcmills]

I'll try to write one tomorrow based on the failures I've been seeing.

I think the key elements are:

• linking with C (to use pthread_create)
• starting nontrivial background goroutines during package init (to produce a shortage of threads)
• calling Exec very early (so that we haven't reached GOMAXPROCS yet when it executes)

[bcmills]

The following test gives me a flake rate of about 0.1% of attempts:

// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

// Issue 18146: pthread_create failure during syscall.Exec.

package cgotest

import "C"

import (
	"bytes"
	"crypto/md5"
	"os"
	"os/exec"
	"runtime"
	"syscall"
	"testing"
)

func test18146(t *testing.T) {
	const (
		attempts = 1000
		threads  = 4
	)

	if os.Getenv("test18146") == "exec" {
		runtime.GOMAXPROCS(1)
		for n := threads; n > 0; n-- {
			go func() {
				for {
					_ = md5.Sum([]byte("Hello, !"))
				}
			}()
		}
		runtime.GOMAXPROCS(threads)
		if err := syscall.Exec("/bin/true", []string{"/bin/true"}, nil); err != nil {
			t.Fatal(err)
		}
	}

	var cmds []*exec.Cmd
	defer func() {
		for _, cmd := range cmds {
			cmd.Process.Kill()
		}
	}()

	args := append(append([]string(nil), os.Args[1:]...), "-test.run=Test18146")
	for n := attempts; n > 0; n-- {
		cmd := exec.Command(os.Args[0], args...)
		cmd.Env = append(os.Environ(), "test18146=exec")
		buf := bytes.NewBuffer(nil)
		cmd.Stdout = buf
		cmd.Stderr = buf
		if err := cmd.Start(); err != nil {
			t.Error(err)
			return
		}
		cmds = append(cmds, cmd)
	}

	failures := 0
	for _, cmd := range cmds {
		err := cmd.Wait()
		if err == nil {
			continue
		}

		t.Errorf("syscall.Exec failed: %v\n%s", err, cmd.Stdout)
		failures++
	}

	if failures > 0 {
		t.Logf("Failed %v of %v attempts.", failures, len(cmds))
	}
}

[ianlancetaylor]

@bcmills thanks for figuring out the test. Sent https://golang.org/cl/33894.

[gopherbot]

CL https://golang.org/cl/33894 mentions this issue.

[ianlancetaylor]

I believe this is now fixed on systems other than OpenBSD (requires changes to runtime/cgo/gcc_libinit_openbsd.c) and Darwin (test fails for unknown reasons).