encoding/json: unmarshaling into a struct from json fails with "reflect: Field index out of range"

[SvenGasterstaedt]

Go version

go version go1.22.2 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1557993680=/tmp/go-build -gno-record-gcc-switches'

What did you do?

using datadog(https://github.com/DataDog/dd-trace-go) tries to marshal/unmarshal serveral structs as json. Sporadically the marshal/unmarshal of traces and logs fails (even tho the source data look fine).

For example (input data - input data was verified via debugging when the panic occured)

{
    "rate_by_service": {
        "service:,env:": 0.5576923076923077,
        "service:test,env:development": 1,
        "service:test123,env:": 0.5576923076923077,
        "service:test234,env:dev": 0.5576923076923077,
        "service:test456,env:dev": 0.8055555555555556,
        "service:test567,env:dev:": 1
    }
}

which is unmarshalled here:
tracer/sampler.go

	var payload struct {
		Rates map[string]float64 `json:"rate_by_service"`
	}
	if err := json.NewDecoder(rc).Decode(&payload); err != nil {
		return err
	}

What did you see happen?

a panic occured:

panic: reflect: Field index out of range
goroutine 1223 [running]:
reflect.Value.Field({0x1613180, 0xc000400080, 0x199}, 0x3ff2d679f1f26f)
	/usr/local/go/src/reflect/value.go:1288 +0x37c
encoding/json.(*decodeState).object(0xc000286028, {0x1613180, 0xc000400080, 0x199})
	/usr/local/go/src/encoding/json/decode.go:720 +0x3789
encoding/json.(*decodeState).value(0xc000286028, {0x15a5300, 0xc000400080, 0x16})
	/usr/local/go/src/encoding/json/decode.go:374 +0x349
encoding/json.(*decodeState).unmarshal(0xc000286028, {0x15a5300, 0xc000400080})
	/usr/local/go/src/encoding/json/decode.go:181 +0x293
encoding/json.(*Decoder).Decode(0xc000286000, {0x15a5300, 0xc000400080})
	/usr/local/go/src/encoding/json/stream.go:73 +0x42b
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*prioritySampler).readRatesJSON(0xc00020a8a0, {0x18b6608, 0xc000c32160})
	/app/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer/sampler.go:113 +0x16a
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*agentTraceWriter).flush.func1(0xc000c36c60)
	/app/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer/writer.go:115 +0xc76
created by gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*agentTraceWriter).flush in goroutine 12
	/app/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer/writer.go:91 +0x32d

the weird thing here is that the index of the field (whiuch is trying to be accessed) is very very high 0x3ff2d679f1f26f = 17999926515593839 and looks more like a pointer/address to me.

reflect/value.go

What did you expect to see?

the marshal should work find (as the json and the struct look fine to me).

[ianlancetaylor]

Have you tried running your program under the race detector? See https://go.dev/blog/race-detector.

[SvenGasterstaedt]

yes, it revealed one race condition tracked with DataDog/dd-trace-go#2648, but otherwise the code was fine. (i temporary fixed it myself within the vendor code but the panics keep occuring)

[ianlancetaylor]

This looks like a data corruption problem, which could be caused by a race condition. That is the first place to look. If you are convinced that this is a problem in the Go standard library, we will need some way to reproduce the problem ourselves. Is that possible? Thanks.

[SvenGasterstaedt]

thanks for pointing me in the right direction - some long term race detection has revealed some "questionable" unsafe.Pointer-usage in a vendor library.

see
SolaceProducts/pubsubplus-go-client#16