fatal error: all goroutines are asleep - deadlock!

[taekwonv]

Hello,

I would like to report an error encountered recently after update with golang v1.9.1 . The following error frequently happens when it calls. Not always, but sometime happens. Can anybody please look into this and hopefully fix this soon?

Thank you!

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [select]:
net/http.(persistConn).roundTrip(0xc042092fc0, 0xc04206aae0, 0x0, 0x0, 0x0)
C:/Go/src/net/http/transport.go:1970 +0x60f
net/http.(Transport).RoundTrip(0x8008a0, 0xc0420fa000, 0x8008a0, 0x0, 0x0)
C:/Go/src/net/http/transport.go:413 +0x999
net/http.send(0xc0420fa000, 0x7ceca0, 0x8008a0, 0x0, 0x0, 0x0, 0xc042076048, 0x4
1034d, 0xc042073390, 0x1)
C:/Go/src/net/http/client.go:249 +0x1b0
net/http.(Client).send(0x7fc020, 0xc0420fa000, 0x0, 0x0, 0x0, 0xc042076048, 0x0
, 0x1, 0x12200ffffffff)
C:/Go/src/net/http/client.go:173 +0x104
net/http.(Client).Do(0x7fc020, 0xc0420fa000, 0x0, 0xc0420735c8, 0x4437de)
C:/Go/src/net/http/client.go:602 +0x294
github.com/couchbase/go-couchbase.doHTTPRequest(0xc0420fa000, 0x0, 0x0, 0x0)
C:/goproject/src/github.com/couchbase/go-couchbase/pools.go:534 +0x9f
github.com/couchbase/go-couchbase.queryRestAPI(0xc042080100, 0x6a6c3a, 0x6, 0x0,
0x0, 0x638a40, 0xc0420ec1f8, 0x0, 0x0)
C:/goproject/src/github.com/couchbase/go-couchbase/pools.go:632 +0x26a
github.com/couchbase/go-couchbase.(Client).parseURLResponse(0xc0420ec1e0, 0x6a6
c3a, 0x6, 0x638a40, 0xc0420ec1f8, 0x0, 0x410801)
C:/goproject/src/github.com/couchbase/go-couchbase/pools.go:652 +0x6f
github.com/couchbase/go-couchbase.ConnectWithAuth(0xc042074030, 0x24, 0x0, 0x0,
0x346010, 0x0, 0x0, 0x805280, 0x3406a8, 0x0, ...)
C:/goproject/src/github.com/couchbase/go-couchbase/pools.go:773 +0xdf
github.com/couchbase/go-couchbase.Connect(0xc042074030, 0x24, 0x0, 0x0, 0x0, 0x0
, 0x0, 0x0, 0x0, 0x0, ...)
C:/goproject/src/github.com/couchbase/go-couchbase/pools.go:792 +0xcd
main.(repoCouchbase3).open(0xc042073cc8, 0xc042074030, 0x24, 0xc0420560c0, 0x7,
0xc0420d8660, 0x20)
C:/goproject/src/:298 +0xd9
main.bucketsImpl(0xc042074030, 0x24, 0xc0420560c0, 0x7, 0xc0420521f0, 0x1, 0x1,
0x0, 0x0)
C:/goproject/src/:245 +0xcf
laszlo/tree.Main(0x6bab10, 0x6bab20, 0x6baaf8, 0x6bab00, 0x6bab30, 0x6baae8, 0x6
87000)
C:/goproject/src/:194 +0x13c9
main.main()
C:/goproject/src/******:32 +0x222

goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0xb24f70, 0x72, 0x0)
C:/Go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc042094c98, 0x72, 0x7cd200, 0x0, 0x0)
C:/Go/src/internal/poll/fd_poll_runtime.go:85 +0xb5
internal/poll.(*ioSrv).ExecIO(0x802d78, 0xc042094b58, 0x6baa30, 0x0, 0x0, 0x0)
C:/Go/src/internal/poll/fd_windows.go:195 +0x13a
internal/poll.(*FD).Read(0xc042094b40, 0xc04210a000, 0x1000, 0x1000, 0x0, 0x0, 0
x0)
C:/Go/src/internal/poll/fd_windows.go:439 +0x266
net.(*netFD).Read(0xc042094b40, 0xc04210a000, 0x1000, 0x1000, 0x345430, 0x0, 0x0
)
C:/Go/src/net/fd_windows.go:151 +0x59
net.(*conn).Read(0xc042076058, 0xc04210a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
C:/Go/src/net/net.go:176 +0x74
net/http.(*persistConn).Read(0xc042092fc0, 0xc04210a000, 0x1000, 0x1000, 0x20, 0
x677300, 0xc042105b68)
C:/Go/src/net/http/transport.go:1391 +0x147
bufio.(*Reader).fill(0xc0420ec600)
C:/Go/src/bufio/bufio.go:97 +0x121
bufio.(*Reader).Peek(0xc0420ec600, 0x1, 0xc042016060, 0xc042105c80, 0x805280, 0x
3406a8, 0x0)
C:/Go/src/bufio/bufio.go:129 +0x41
net/http.(*persistConn).readLoop(0xc042092fc0)
C:/Go/src/net/http/transport.go:1539 +0x18c
created by net/http.(*Transport).dialConn
C:/Go/src/net/http/transport.go:1186 +0xa35

goroutine 23 [select]:
net/http.(*persistConn).writeLoop(0xc042092fc0)
C:/Go/src/net/http/transport.go:1759 +0x16c
created by net/http.(*Transport).dialConn
C:/Go/src/net/http/transport.go:1187 +0xa5a

[tleyden]

@taekwonv this could also be an error in your app where you are causing a deadlock, and it just happens to be printing out go-couchbase related goroutines. Is it possible to provide a minimal test case to repro and isolate the issue to go-couchbase?

Also is that the full goroutine dump?

[taekwonv]

@tleyden Thanks for your prompt response. Unfortunately, I cannot share a binary to repro but I can share a piece of the code. The dump is full and the error happens consistently once it happens. Same location, same error. It happens when it calls 'couchbase.Connect(repo.serverURL)' shown below.

    c, err := couchbase.Connect(repo.serverURL)
if err != nil {
	return err
}
p, err := c.GetPool("default")
if err != nil {
	return err
}
b, err := p.GetBucket(repo.projectName)
if err != nil {
	return err
}

[tleyden]

So are you saying with enough repeated attempts, it will produce that deadlock? Approximately how often does it happen? 1/10 attempts? 1/1000 attempts?

Also, which OS/version are you on?

[taekwonv]

About 1 or 2 / 10 attempts. It is Windows Server 2008 R2 Standard (Version 6.1.7601). I also did a couple of test on both another W2K8R2 std and Ubuntu 14.04 LT and did not see the problem yet. That could tells this could be an host specific issue? But, this problem started happening right after golang 1.9.1 update, some change might trigger this issue but I am not sure how wide this issue can happen. Let me try golang clean reinstall and will update you. Thanks!

[tleyden]

Can you run with -race flag?

[taekwonv]

I tried to repro with -race flag, but the issue never happened with -race flag. I removed and reinstall Go and go-couchbase package, the issue still happens on the instance.

[tleyden]

Actually, maybe it's possible for goroutines to deadlock without any data races that the race detector would catch.

Can you push your complete test program up to a github gist? That way I can submit it to the Couchbase QE folks to test against on that same platform.

On Sync Gateway we are using GoCB except for the DCP access, where we use the cbdatasource package within go-couchbase, but we don't make any bucket connections with go-couchbase. So this won't get caught by our test suite, but maybe it will in other products within couchbase.

[taekwonv]

Again, this program cannot be shared outside since it is internal one... This could be something that golang developers should look into even though it happens a particular host. Let me update you if any progress is made.
Thanks!

[taekwonv]

Update.
The problem has gone after upgrade Go version from 1.9.1 to 1.9.2.
This looks to be related with this -> golang/go#22394 . After using 1.9.2, no more issue has been happening.

[tleyden]

Thanks for following up on this!