[BUG] dd fails to unmarshal rates after writing traces

[SvenGasterstaedt]

Version of dd-trace-go
gopkg.in/DataDog/dd-trace-go.v1 v1.61.0
gopkg.in/DataDog/dd-trace-go.v1 v1.62.0

Describe what happened:
sometime when dd tries to write traces to the dd agent it fails with a panic when unmarshalling the response from the datadog agent.
(this is not always the case but sometimes (like 1/1000))

panic: reflect: Field index out of range

goroutine 701 [running]:
reflect.Value.Field({0x103e640?, 0xc0008883f8?, 0xc000a0c802?}, 0x196511c2c78171)
        /usr/local/go/src/reflect/value.go:1288 +0x265
encoding/json.(*decodeState).object(0xc000a2b6a8, {0xfd1be0?, 0xc0008883f8?, 0xc000fa9bff?})
        /usr/local/go/src/encoding/json/decode.go:720 +0x1fae
encoding/json.(*decodeState).value(0xc000a2b6a8, {0xfd1be0?, 0xc0008883f8?, 0x1ac?})
        /usr/local/go/src/encoding/json/decode.go:374 +0xaf
encoding/json.(*decodeState).unmarshal(0xc000a2b6a8, {0xfd1be0, 0xc0008883f8})
        /usr/local/go/src/encoding/json/decode.go:181 +0x2d7
encoding/json.(*Decoder).Decode(0xc000a2b680, {0xfd1be0, 0xc0008883f8})
        /usr/local/go/src/encoding/json/stream.go:73 +0x3c5
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*prioritySampler).readRatesJSON(0xc00018ec60, {0x12b49e0, 0xc0005d0a80})
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer/sampler.go:113 +0x172
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*agentTraceWriter).flush.func1(0xc00060ef00)
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer/writer.go:115 +0x63a
created by gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*agentTraceWriter).flush in goroutine 14
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer/writer.go:91 +0x2e5

strangly this kind of errors occur on various places, i suspect that something is wrong with the unix domain socket connection itself (as most calls fail reading from the corresponding stream)

panic: reflect: call of reflect.Value.IsNil on string Value [recovered]
        panic: reflect: call of reflect.Value.IsNil on string Value

goroutine 606 [running]:
encoding/json.(*encodeState).marshal.func1()
        /usr/local/go/src/encoding/json/encode.go:293 +0xb8
panic({0x1017980?, 0xc000012be8?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
reflect.Value.IsNil(...)
        /usr/local/go/src/reflect/value.go:1574
encoding/json.sliceEncoder.encode({0x480b69?}, 0xc00099eec0, {0xfdd040?, 0xc0000bce70?, 0x5825a8?}, {0x37?, 0xe6?})
        /usr/local/go/src/encoding/json/encode.go:801 +0x82d
encoding/json.structEncoder.encode({{{0xc0008ab688, 0x7, 0x8}, 0xc000c7e0f0, 0xc000c7e120}}, 0xc00099eec0, {0x10d34a0?, 0xc0000bce70?, 0x10d34a0?}, {0x0, ...})
        /usr/local/go/src/encoding/json/encode.go:704 +0x2be
encoding/json.arrayEncoder.encode({0xc00003d9e0?}, 0xc00099eec0, {0xfd4820?, 0xc000da74b0?, 0xc000da74b7?}, {0xa0?, 0x74?})
        /usr/local/go/src/encoding/json/encode.go:847 +0x103
encoding/json.sliceEncoder.encode({0x480b69?}, 0xc00099eec0, {0xfd4820?, 0xc000da74b0?, 0x5825a8?}, {0x47?, 0xe5?})
        /usr/local/go/src/encoding/json/encode.go:820 +0x543
encoding/json.structEncoder.encode({{{0xc000c86000, 0x2, 0x2}, 0xc000c7e1b0, 0xc000c7e1e0}}, 0xc00099eec0, {0x106a200?, 0xc000da74a0?, 0xc000010d00?}, {0x0, ...})
        /usr/local/go/src/encoding/json/encode.go:704 +0x2be
encoding/json.ptrEncoder.encode({0x480b31?}, 0xc00099eec0, {0xfc2b20?, 0xc000da74a0?, 0x12c3ea8?}, {0x20?, 0x2b?})
        /usr/local/go/src/encoding/json/encode.go:876 +0x3f2
encoding/json.(*encodeState).reflectValue(0xc00099eec0, {0xfc2b20?, 0xc000da74a0?, 0xc000010d9d?}, {0xa8?, 0x25?})
        /usr/local/go/src/encoding/json/encode.go:321 +0x84
encoding/json.interfaceEncoder(0xc00099eec0, {0x100c280?, 0xc000833648?, 0x6?}, {0x80?, 0xff?})
        /usr/local/go/src/encoding/json/encode.go:658 +0x165
encoding/json.structEncoder.encode({{{0xc00085a008, 0x9, 0x10}, 0xc0002a82d0, 0xc0002a8300}}, 0xc00099eec0, {0x10e09a0?, 0xc000833600?, 0x8?}, {0x0, ...})
        /usr/local/go/src/encoding/json/encode.go:704 +0x2be
encoding/json.ptrEncoder.encode({0x480b31?}, 0xc00099eec0, {0xfc2660?, 0xc000833600?, 0x12c3ea8?}, {0x60?, 0x26?})
        /usr/local/go/src/encoding/json/encode.go:876 +0x3f2
encoding/json.(*encodeState).reflectValue(0xc00099eec0, {0xfc2660?, 0xc000833600?, 0x38?}, {0xa0?, 0x4f?})
        /usr/local/go/src/encoding/json/encode.go:321 +0x84
encoding/json.(*encodeState).marshal(0xc00099eec0, {0xfc2660, 0xc000833600}, {0x69?, 0xb?})
        /usr/local/go/src/encoding/json/encode.go:297 +0xeb
encoding/json.Marshal({0xfc2660, 0xc000833600})
        /usr/local/go/src/encoding/json/encode.go:163 +0x12c
gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry.(*Request).trySubmit(0xc000da7500)
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry/client.go:540 +0x66
gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry.(*Request).submit(0xc000da7500)
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry/client.go:498 +0x3c
gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry.(*client).flush.func1()
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry/client.go:411 +0x72
created by gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry.(*client).flush in goroutine 605
        /app/vendor/gopkg.in/DataDog/dd-trace-go.v1/internal/telemetry/client.go:409 +0x3e5

Describe what you expected:
no error when unmarshalling the response.

Steps to reproduce the issue:
sadly i can't provide a minimal example :/

Additional environment details (Version of Go, Operating System, etc.):
go 1.22.1 & 1.22.2
linux/amd64

communiction via unix domain sockets where multiple services write to a single pipe and the dd-agent consumes this.

data about the datadog agent

curl localhost:8126/info
{
        "version": "7.49.1",
        "git_commit": "1790cab",
        "endpoints": [
                "/v0.3/traces",
                "/v0.3/services",
                "/v0.4/traces",
                "/v0.4/services",
                "/v0.5/traces",
                "/v0.7/traces",
                "/profiling/v1/input",
                "/telemetry/proxy/",
                "/v0.6/stats",
                "/v0.1/pipeline_stats",
                "/evp_proxy/v1/",
                "/evp_proxy/v2/",
                "/evp_proxy/v3/",
                "/debugger/v1/input",
                "/debugger/v1/diagnostics",
                "/symdb/v1/input",
                "/dogstatsd/v1/proxy",
                "/dogstatsd/v2/proxy",
                "/v0.7/config",
                "/config/set"
        ],
        "client_drop_p0s": true,
        "span_meta_structs": true,
        "long_running_spans": true,
        "config": {
                "default_env": "dev",
                "target_tps": 10,
                "max_eps": 200,
                "receiver_port": 8126,
                "receiver_socket": "/var/run/datadog/apm.socket",
                "connection_limit": 0,
                "receiver_timeout": 0,
                "max_request_bytes": 26214400,
                "statsd_port": 8125,
                "max_memory": 0,
                "max_cpu": 0,
                "analyzed_spans_by_service": {},
                "obfuscation": {
                        "elastic_search": true,
                        "mongo": true,
                        "sql_exec_plan": false,
                        "sql_exec_plan_normalize": false,
                        "http": {
                                "remove_query_string": false,
                                "remove_path_digits": false
                        },
                        "remove_stack_traces": false,
                        "redis": {
                                "Enabled": true,
                                "RemoveAllArgs": false
                        },
                        "memcached": {
                                "Enabled": true,
                                "KeepCommand": false
                        }
                }
        }

[SvenGasterstaedt]

could this be related to DataDog/datadog-agent#24391?

[darccio]

@SvenGasterstaedt I tried to reproduce it, but it seems complex. It's not a malformed JSON, it's something else.

[SvenGasterstaedt]

@darccio yes, indeed. i was able to debug the probem/code. It seems very very weird.

in this case the update-request fails to mashal - in the middle of the message

or this other "related" case, where the json marshal/unmarshal went out of bounce:
encoding/json/decode.go

The debugger showed that the reflect.Value has an invalid index (which then leads to the out of bounds later on).

We don't use encoding/json ourself withing this microservice (it only usued within datadog).

the service runs with the offical image:
https://hub.docker.com/layers/library/golang/1.22.2/images/sha256-3db3999ce82f443c0aa6b8da50d8ecea2df945dcf2a78101aff57d2f61f83969?context=explore

[SvenGasterstaedt]

@darccio i have opened an issue at golang as well see golang/go#66762. Maybe they have a clue what is going on here :)

[darccio]

@SvenGasterstaedt I used the payload in golang/go#66762, and it works fine. I think they are right pointing to a possible data corruption.

We'll need to investigate the Unix Domain Socket integration, as the issue seems that data is overrided due to concurrent access.

[SvenGasterstaedt]

i think they are right - some additional race detection has revealed some unsafe.Pointers in a vendor library which might cause data corruption.

SolaceProducts/pubsubplus-go-client#16

i will close this issues (as the dd code looks fine to me 😎)

[darccio]

@SvenGasterstaedt Cool, thanks for the effort getting into this rabbit hole!