net/http: panic: runtime error: index out of range

[alexbrainman]

Just seen on windows-amd64-race builder http://build.golang.org/log/81d3f339f9c1248dcbd80defc911b338d61c4118

ok      net 30.538s
panic: runtime error: index out of range

goroutine 2528 [running]:
panic(0x8d2140, 0xc0820060e0)
    C:/workdir/go/src/runtime/panic.go:500 +0x1a6
net/http_test.(*rwTestConn).RemoteAddr(0xc0820ee420, 0xb500e0, 0xc0823ca000)
    <autogenerated>:113 +0x9c
net/http.(*conn).serve(0xc0823ce200)
    C:/workdir/go/src/net/http/server.go:1388 +0x74
created by net/http.(*Server).Serve
    C:/workdir/go/src/net/http/server.go:2146 +0x3f9
FAIL    net/http    12.205s
ok      net/http/cgi    6.434s

I am not sure what to do here. But I am also confused about <autogenerated>:113 in stack trace. I wonder what happened here. Am I missing something?

Alex

[alexbrainman]

It happened again http://build.golang.org/log/e29df5d52f88ce13b851539511ec5bd4d05b4cb5

[martisch]

If you just want to see the autogenerated code:
you could build the http_test binary and then objdump to see the asm for autogenerated code.

Not 100% sure but maybe this could be the auto generated pointer receiver version of:
func (noopConn) RemoteAddr() net.Addr { return dummyAddr("remote-addr") }

This shows for me on OSX:

TEXT net/http_test.(*rwTestConn).RemoteAddr(SB) <autogenerated>
        <autogenerated>:113     0x1976c0        65488b0c25a0080000      GS MOVQ GS:0x8a0, CX
        <autogenerated>:113     0x1976c9        483b6110                CMPQ 0x10(CX), SP
        <autogenerated>:113     0x1976cd        7672                    JBE 0x197741
        <autogenerated>:113     0x1976cf        4883ec38                SUBQ $0x38, SP
        <autogenerated>:113     0x1976d3        488b5920                MOVQ 0x20(CX), BX
        <autogenerated>:113     0x1976d7        4885db                  TESTQ BX, BX
        <autogenerated>:113     0x1976da        740d                    JE 0x1976e9
        <autogenerated>:113     0x1976dc        488d7c2440              LEAQ 0x40(SP), DI
        <autogenerated>:113     0x1976e1        48393b                  CMPQ DI, 0(BX)
        <autogenerated>:113     0x1976e4        7503                    JNE 0x1976e9
        <autogenerated>:113     0x1976e6        488923                  MOVQ SP, 0(BX)
        <autogenerated>:113     0x1976e9        488b442440              MOVQ 0x40(SP), AX
        <autogenerated>:113     0x1976ee        8400                    TESTL AL, 0(AX)
        <autogenerated>:113     0x1976f0        488d059bcb2200          LEAQ 0x22cb9b(IP), AX
        <autogenerated>:113     0x1976f7        4889442428              MOVQ AX, 0x28(SP)
        <autogenerated>:113     0x1976fc        48c74424300b000000      MOVQ $0xb, 0x30(SP)
        <autogenerated>:113     0x197705        488d05945f3e00          LEAQ 0x3e5f94(IP), AX
        <autogenerated>:113     0x19770c        48890424                MOVQ AX, 0(SP)
        <autogenerated>:113     0x197710        488d442428              LEAQ 0x28(SP), AX
        <autogenerated>:113     0x197715        4889442408              MOVQ AX, 0x8(SP)
        <autogenerated>:113     0x19771a        48c744241000000000      MOVQ $0x0, 0x10(SP)
        <autogenerated>:113     0x197723        e81862e7ff              CALL runtime.convT2I(SB)
        <autogenerated>:113     0x197728        488b4c2420              MOVQ 0x20(SP), CX
        <autogenerated>:113     0x19772d        488b542418              MOVQ 0x18(SP), DX
        <autogenerated>:113     0x197732        4889542448              MOVQ DX, 0x48(SP)
        <autogenerated>:113     0x197737        48894c2450              MOVQ CX, 0x50(SP)
        <autogenerated>:113     0x19773c        4883c438                ADDQ $0x38, SP
        <autogenerated>:113     0x197740        c3                      RET
        <autogenerated>:113     0x197741        e8ea9aebff              CALL runtime.morestack_noctxt(SB)
        <autogenerated>:113     0x197746        e975ffffff              JMP net/http_test.(*rwTestConn).RemoteAddr(SB)
        <autogenerated>:113     0x19774b        cc                      INT $0x3
...

[bradfitz]

@randall77, I'm going to assume this is a code generation issue for now and not actually a net/http problem?

[randall77]

I don't see anything obviously wrong from the code.

This is the autogenerated code for (*rwTestConn).RemoteAddr. It is autogenerated because rwTestConn contains an embedded type noopConn that provides the RemoteAddr implementation. The stub basically does:

func (x *rwTestConn) RemoteAddr() net.Addr {
    return (&x.noopConn).RemoteAddr()
}

The call out to noopConn.RemoteAddr is inlined so you don't see the actual call.

It's weird that the error is "index out of range". There are no bounds checks in this stub.
One possibility is that with -race we call racereadrange with a size of 0 (for the read of noopConn, which is a struct{}). Maybe race is confused by zero size? Seems unlikely, but @dvyukov just in case.

The only non-race thing it calls is convT2I, which is basically malloc+memmove. Hard to see how a bounds check could fail down there. It might be helpful to get a GOTRACEBACK=system trace for this failure.

I don't know what might be windows-specific about this. I'm going to try to reproduce on linux overnight.

[alexbrainman]

maybe this could be the auto generated pointer receiver version of:
func (noopConn) RemoteAddr() net.Addr { return dummyAddr("remote-addr") }

It is quite possible. But then I expect trace to say:

net/http_test.(*rwTestConn).RemoteAddr(0xc0820ee420, 0xb500e0, 0xc0823ca000)
    C:/workdir/go/src/net/http/serve_test.go:113 +0x9c

and not

net/http_test.(*rwTestConn).RemoteAddr(0xc0820ee420, 0xb500e0, 0xc0823ca000)
    <autogenerated>:113 +0x9c

And here I have built windows-amd64-race test exe on my system:

TEXT net/http_test.(*rwTestConn).RemoteAddr(SB) <autogenerated>
    <autogenerated>:113 0x61ca10    65488b0c2528000000  GS MOVQ GS:0x28, CX
    <autogenerated>:113 0x61ca19    488b8900000000      MOVQ 0(CX), CX
    <autogenerated>:113 0x61ca20    483b6110        CMPQ 0x10(CX), SP
    <autogenerated>:113 0x61ca24    0f86b4000000        JBE 0x61cade
    <autogenerated>:113 0x61ca2a    4883ec48        SUBQ $0x48, SP
    <autogenerated>:113 0x61ca2e    488b5920        MOVQ 0x20(CX), BX
    <autogenerated>:113 0x61ca32    4885db          TESTQ BX, BX
    <autogenerated>:113 0x61ca35    740d            JE 0x61ca44
    <autogenerated>:113 0x61ca37    488d7c2450      LEAQ 0x50(SP), DI
    <autogenerated>:113 0x61ca3c    48393b          CMPQ DI, 0(BX)
    <autogenerated>:113 0x61ca3f    7503            JNE 0x61ca44
    <autogenerated>:113 0x61ca41    488923          MOVQ SP, 0(BX)
    <autogenerated>:113 0x61ca44    488b442448      MOVQ 0x48(SP), AX
    <autogenerated>:113 0x61ca49    48890424        MOVQ AX, 0(SP)
    <autogenerated>:113 0x61ca4d    e81ed7e3ff      CALL runtime.racefuncenter(SB)
    <autogenerated>:113 0x61ca52    488b442450      MOVQ 0x50(SP), AX
    <autogenerated>:113 0x61ca57    4883c020        ADDQ $0x20, AX
    <autogenerated>:113 0x61ca5b    48890424        MOVQ AX, 0(SP)
    <autogenerated>:113 0x61ca5f    48c744240800000000  MOVQ $0x0, 0x8(SP)
    <autogenerated>:113 0x61ca68    e813d6e3ff      CALL runtime.racereadrange(SB)
    <autogenerated>:113 0x61ca6d    488b442450      MOVQ 0x50(SP), AX
    <autogenerated>:113 0x61ca72    8400            TESTL AL, 0(AX)
    <autogenerated>:113 0x61ca74    488d05b22f3200      LEAQ 0x322fb2(IP), AX
    <autogenerated>:113 0x61ca7b    4889442438      MOVQ AX, 0x38(SP)
    <autogenerated>:113 0x61ca80    48c74424400b000000  MOVQ $0xb, 0x40(SP)
    <autogenerated>:113 0x61ca89    488d05902a5300      LEAQ 0x532a90(IP), AX
    <autogenerated>:113 0x61ca90    48890424        MOVQ AX, 0(SP)
    <autogenerated>:113 0x61ca94    488d442438      LEAQ 0x38(SP), AX
    <autogenerated>:113 0x61ca99    4889442408      MOVQ AX, 0x8(SP)
    <autogenerated>:113 0x61ca9e    48c744241000000000  MOVQ $0x0, 0x10(SP)
    <autogenerated>:113 0x61caa7    e8a415dfff      CALL runtime.convT2I(SB)
    <autogenerated>:113 0x61caac    488b4c2420      MOVQ 0x20(SP), CX
    <autogenerated>:113 0x61cab1    48894c2430      MOVQ CX, 0x30(SP)
    <autogenerated>:113 0x61cab6    488b542418      MOVQ 0x18(SP), DX
    <autogenerated>:113 0x61cabb    4889542428      MOVQ DX, 0x28(SP)
    <autogenerated>:113 0x61cac0    e8ebd6e3ff      CALL runtime.racefuncexit(SB)
    <autogenerated>:113 0x61cac5    488b4c2428      MOVQ 0x28(SP), CX
    <autogenerated>:113 0x61caca    48894c2458      MOVQ CX, 0x58(SP)
    <autogenerated>:113 0x61cacf    488b4c2430      MOVQ 0x30(SP), CX
    <autogenerated>:113 0x61cad4    48894c2460      MOVQ CX, 0x60(SP)
    <autogenerated>:113 0x61cad9    4883c448        ADDQ $0x48, SP
    <autogenerated>:113 0x61cadd    c3          RET
    <autogenerated>:113 0x61cade    e82d9ee3ff      CALL runtime.morestack_noctxt(SB)
    <autogenerated>:113 0x61cae3    e928ffffff      JMP net/http_test.(*rwTestConn).RemoteAddr(SB)
    <autogenerated>:113 0x61cae8    cc          INT $0x3
    <autogenerated>:113 0x61cae9    cc          INT $0x3
    <autogenerated>:113 0x61caea    cc          INT $0x3
    <autogenerated>:113 0x61caeb    cc          INT $0x3
    <autogenerated>:113 0x61caec    cc          INT $0x3
    <autogenerated>:113 0x61caed    cc          INT $0x3
    <autogenerated>:113 0x61caee    cc          INT $0x3
    <autogenerated>:113 0x61caef    cc          INT $0x3

Alex

[randall77]

I was unable to reproduce this bug overnight on linux/amd64.
Time to gomote a windows instance.

[randall77]

reproduced with GOTRACEBACK=system enabled:

khr@ultimate:~/sandbox/readonly/src$ gomote run user-khr-windows-amd64-race-0 go/bin/go test -count 100 -timeout 40m -race -short net/http
panic: runtime error: index out of range

goroutine 560140 [running]:
panic(0x8d2520, 0xc08200a040)
    C:/workdir/go/src/runtime/panic.go:500 +0x1a6 fp=0xc082089b88 sp=0xc082089b00
runtime.panicindex()
    C:/workdir/go/src/runtime/panic.go:27 +0x6a fp=0xc082089bb0 sp=0xc082089b88
runtime.nextBarrierPC(0x40c5f6)
    C:/workdir/go/src/runtime/mstkbar.go:320 +0x3e fp=0xc082089bb8 sp=0xc082089bb0
runtime.getcallerpc(0xc082089bf0, 0xc0820c9607)
    C:/workdir/go/src/runtime/asm_amd64.s:803 +0x1b fp=0xc082089bc8 sp=0xc082089bb8
runtime.convT2I(0xb50520, 0xc082089c38, 0x0, 0x0, 0x0)
    C:/workdir/go/src/runtime/iface.go:176 +0x4c fp=0xc082089c00 sp=0xc082089bc8
net/http_test.(*rwTestConn).RemoteAddr(0xc0824c8360, 0xb510e0, 0xc0821e21a0)
    <autogenerated>:113 +0x9c fp=0xc082089c50 sp=0xc082089c00
net/http.(*conn).serve(0xc082456200, 0xb519a0, 0xc0824c83c0)
    C:/workdir/go/src/net/http/server.go:1414 +0x74 fp=0xc082089f88 sp=0xc082089c50
runtime.goexit()
    C:/workdir/go/src/runtime/asm_amd64.s:2013 +0x1 fp=0xc082089f90 sp=0xc082089f88
created by net/http.(*Server).Serve

@aclements looks like something to do with stack barriers.
Found the bug, there's a bad call to getcallerpc. Will send a CL.

[gopherbot]

CL https://golang.org/cl/21931 mentions this issue.

[minux]

I wonder if we should set GOTRACEBACK=2 on all the builders.

[bradfitz]

@minux, probably. File a bug?