Segmentation fault at pkg/runtime/proc.c:2273 #7065

gopherbot · 2014-01-05T17:18:44Z

by guohouzuo:

What steps will reproduce the problem?

Sorry! I have not been able to make a small & reproducible example. See additional
information below.

Which compiler are you using (5g, 6g, 8g, gccgo)?

Golang 1.2, downloaded and compiled by Homebrew

Which operating system are you using?

OSX Mavericks, Darwin 13.0 64-bit

Which version are you using?  (run 'go version')

go version go1.2 darwin/amd64

Please provide any additional information below.

In the middle of debugging a performance issue in my project tiedot, I turned on CPU
profiling (runtime/pprof), and the segmentation fault happens at a very small chance.
The stack trace of fault is:

SIGSEGV: segmentation violation
PC=0x1e382

acquirep(0x0)
    /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/proc.c:2273 +0x52
stopm()
    /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/proc.c:940 +0x157
findrunnable()
    /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/proc.c:1237 +0x42d
schedule()
    /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/proc.c:1320 +0xe3
park0(0x22663d5a0)
    /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/proc.c:1361 +0xd8
runtime.mcall(0x29af2)
    /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/asm_amd64.s:178 +0x4b

Complete stacktrace of all dumped goroutines is attached.

Attachments:

complete_goroutine_trace (25285 bytes)

dvyukov · 2014-01-09T12:03:28Z

Comment 1:

It seems that a thread is woken w/o m->nextp. If it can happen, then it would crash with
the same stack trace. But I have not seen such crashes before. It's possible that it
crashes with a very low probability.
It seems to be an open-source project. Can you provide reproduction instructions?

Labels changed: added release-go1.3maybe, repo-main.

gopherbot · 2014-01-09T12:17:29Z

Comment 2 by guohouzuo:

Thanks Dmitry. I will soon try my best to make a minimized reproducible.

dvyukov · 2014-01-09T12:33:09Z

Comment 3:

To make it clear, I do not care too much about size of the involved code base. Do if can
just provide commands that I need to execute and necessary input data (if any), that
would be enough for starters.

dvyukov · 2014-01-09T12:33:34Z

Comment 4:

Labels changed: added release-go1.2.1, removed release-go1.3maybe.

gopherbot · 2014-01-10T06:53:44Z

Comment 5 by guohouzuo:

Hello Dmitry.
Please clone tiedot "segfault" branch:
    https://github.com/HouzuoGuo/tiedot/tree/segfault
And compile it as per usual:
    go build
To trigger the segfault, run the compiled executable with flag -bench3:
    ./tiedot -bench3
Clean up happens automatically before each run, so no manual cleanup is required.
It almost always completes successfully, but the segfault happens every 40 or 50 runs.
I attach another segfault goroutine trace.
Thanks!

Attachments:

goroutine_trace2 (22370 bytes)

dvyukov · 2014-01-10T09:49:21Z

Comment 6:

I don't know whether I guessed how to do these steps correctly or not. My checkout seems
to be on segfault:
gopath/src/github.com/HouzuoGuo/tiedot$ git status
On branch segfault
Your branch is up-to-date with 'origin/segfault'.
But the binary does not recognize -bench3:
$ ./tiedot -bench3
flag provided but not defined: -bench3
Usage of ./tiedot:

gopherbot · 2014-01-10T09:50:37Z

Comment 7 by guohouzuo:

Oops, my apologize, I was supposed to say:
    ./tiedot -mode=bench3

dvyukov · 2014-01-10T11:41:41Z

Comment 8:

Can run it now. No crashes yet.
You've said that if happens only if you enable CPU profiling. Do I need to do anything
special to enable it? Or it is enabled by default?

dvyukov · 2014-01-10T12:05:27Z

Comment 9:

When I run it, it consumes 50+GB of memory. Is it possible to limit amount of memory it
consumes?

gopherbot · 2014-01-10T13:40:22Z

Comment 10 by guohouzuo:

Hello Dmitry.
CPU profile is already enabled in the code, see line 296 in benchmark.go.
It sounds very unusual to consume 50+GB of memory. On my system, the peak resident
memory usage is around 2GB.
Out of curiosity, are you running tiedot on a OSX system?

dvyukov · 2014-01-12T16:33:37Z

Comment 11:

Owner changed to @dvyukov.

Status changed to Accepted.

dvyukov · 2014-01-12T16:33:45Z

Comment 12:

>CPU profile is already enabled in the code, see line 296 in benchmark.go.
Ah, OK, I see. However note that CPU profiling was disabled on OSX in Go 1.2... So I am
not sure how it affects the issue...
> It sounds very unusual to consume 50+GB of memory. On my system, the peak resident
memory usage is around 2GB.
No, that was linux 64-bit. I wanted to see whether it reproduces there.
I was able to reproduce it on tip on OSX so far.

dvyukov · 2014-01-12T16:34:44Z

Comment 13:

I suspect it may be related to 
https://golang.org/issue/5139

gopherbot · 2014-01-12T16:40:39Z

Comment 14 by guohouzuo:

Thanks Dmitry.
So... the performance output has been inaccurate and useless hahaha. I have tried very
hard to optimise tiedot according to the output!

gopherbot · 2014-01-12T16:48:35Z

Comment 15 by guohouzuo:

By the way, it never crashes with segfault with CPU profiling turned off... at least it
hasn't happened to me.

dvyukov · 2014-01-12T16:54:20Z

Comment 16:

How many times have you seen it? Have you seen any other crashes, like "notewakeup -
double wakeup"?

gopherbot · 2014-01-12T16:55:58Z

Comment 17 by guohouzuo:

Double wakup has never happened to me.
The segfault has happened three times so far - twice before I submitted the bug report,
and once when I cut the branch to reproduce it.

dvyukov · 2014-01-14T16:06:45Z

Comment 18:

I've sent:
https://golang.org/cl/52090045
that fixes a very obscure data race in runtime. It can be potentially the cause of the
crashes you see. Difficult to say.

dvyukov · 2014-01-15T15:41:09Z

Comment 19:

https://golang.org/cl/52090045/
is submitted and it may fix the issue
I've tried to reproduce it several days on several machines, but reproduced only once
and w/o and special debugging output...

gopherbot · 2014-01-15T18:52:49Z

Comment 20 by guohouzuo:

Thanks very much Dmitry. Shall I compile Go with the patch and try again?

dvyukov · 2014-01-16T03:14:59Z

Comment 21:

It would be great if you can confirm whether it's fixed or not.
The patch is already submitted, so you need to checkout tip version of Go and build.

adg · 2014-02-13T22:52:51Z

Comment 22:

Status changed to WaitingForReply.

gopherbot · 2014-02-14T08:22:20Z

Comment 23 by guohouzuo:

my apologize for the late reply. i shall compile go1.2 with the patch and give it a spin
this weekend.

gopherbot · 2014-02-16T09:53:59Z

Comment 24 by guohouzuo:

Hello Dmitry.
Thanks very much for the fix! I compiled Go 1.2 with the patch, and ran the program 100
times in a loop, the bug has disappeared.

dvyukov · 2014-02-16T10:00:50Z

Comment 25:

Thanks!
Let's consider this as Fixed. If you spot this again (with tip or Go1.3) please notify
us.
Removing 1.2.1 label, because issue #5139 is marked as 1.2.1.

Labels changed: added release-go1.3, removed release-go1.2.1.

Status changed to Fixed.

««« CL 52090045 / 302bdb5b08b1 runtime: fix data race in GC Fixes #5139. Update #7065. R=golang-codereviews, bradfitz, minux.ma CC=golang-codereviews https://golang.org/cl/52090045 »»» LGTM=r R=golang-codereviews, r CC=golang-dev https://golang.org/cl/69790043

gopherbot added fixed labels Feb 16, 2014

rsc added this to the Go1.3 milestone Apr 14, 2015

rsc removed the release-go1.3 label Apr 14, 2015

golang locked and limited conversation to collaborators Jun 25, 2016

gopherbot added the FrozenDueToAge label Jun 25, 2016

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault at pkg/runtime/proc.c:2273 #7065

Segmentation fault at pkg/runtime/proc.c:2273 #7065

gopherbot commented Jan 5, 2014

dvyukov commented Jan 9, 2014

gopherbot commented Jan 9, 2014

dvyukov commented Jan 9, 2014

dvyukov commented Jan 9, 2014

gopherbot commented Jan 10, 2014

dvyukov commented Jan 10, 2014

gopherbot commented Jan 10, 2014

dvyukov commented Jan 10, 2014

dvyukov commented Jan 10, 2014

gopherbot commented Jan 10, 2014

dvyukov commented Jan 12, 2014

dvyukov commented Jan 12, 2014

dvyukov commented Jan 12, 2014

gopherbot commented Jan 12, 2014

gopherbot commented Jan 12, 2014

dvyukov commented Jan 12, 2014

gopherbot commented Jan 12, 2014

dvyukov commented Jan 14, 2014

dvyukov commented Jan 15, 2014

gopherbot commented Jan 15, 2014

dvyukov commented Jan 16, 2014

adg commented Feb 13, 2014

gopherbot commented Feb 14, 2014

gopherbot commented Feb 16, 2014

dvyukov commented Feb 16, 2014

Segmentation fault at pkg/runtime/proc.c:2273 #7065

Segmentation fault at pkg/runtime/proc.c:2273 #7065

Comments

gopherbot commented Jan 5, 2014

dvyukov commented Jan 9, 2014

gopherbot commented Jan 9, 2014

dvyukov commented Jan 9, 2014

dvyukov commented Jan 9, 2014

gopherbot commented Jan 10, 2014

dvyukov commented Jan 10, 2014

gopherbot commented Jan 10, 2014

dvyukov commented Jan 10, 2014

dvyukov commented Jan 10, 2014

gopherbot commented Jan 10, 2014

dvyukov commented Jan 12, 2014

dvyukov commented Jan 12, 2014

dvyukov commented Jan 12, 2014

gopherbot commented Jan 12, 2014

gopherbot commented Jan 12, 2014

dvyukov commented Jan 12, 2014

gopherbot commented Jan 12, 2014

dvyukov commented Jan 14, 2014

dvyukov commented Jan 15, 2014

gopherbot commented Jan 15, 2014

dvyukov commented Jan 16, 2014

adg commented Feb 13, 2014

gopherbot commented Feb 14, 2014

gopherbot commented Feb 16, 2014

dvyukov commented Feb 16, 2014