Invalid MessageAttributes on SNS

[jnyrup]

Describe the bug
After we upgraded to 2.32.0 our subscription filters on AWS SNS unfortunately broke, so messages were not forwarded to SQS queues.
Looking into AWS CloudWatch metrics we saw NumberOfNotificationsFilteredOut-InvalidAttributes had started to happen.

The NumberOfNotificationsFilteredOut-InvalidAttributes metric shows messages that your filter policy rejected because the message attributes weren't in a valid format.

https://repost.aws/knowledge-center/sns-subscription-filter-policy-issues

After downgrading to 2.31.0 we stopped getting NumberOfNotificationsFilteredOut-InvalidAttributes.

Here's the MessageAttributes of one of our messages, but nothing in there immediately looks suspicious to me.

"MessageAttributes" : {
  "ourKey" : {"Type":"String","Value":"OurData"},
  "_datadog" : {"Type":"String","Value":"{\"traceparent\":\"00-000000000000000023b62f1bd3212848-1fe678cc9b22cbd2-01\",\"tracestate\":\"dd=s:1;t.dm:-1\",\"x-datadog-trace-id\":\"2573296033641605192\",\"x-datadog-parent-id\":\"2298657479990627282\",\"x-datadog-sampling-priority\":\"1\",\"x-datadog-tags\":\"_dd.p.dm=-1\"}"}
}

To Reproduce

Expected behavior
MessageAttributes should contain valid data, parsable by AWS SNS subscription filters.

Runtime environment (please complete the following information):

• Instrumentation mode: automatic
• Tracer version: 2.32.0
• OS: Windows Server 2019 (AWS EB Windows Server)
• CLR: .NET 6.0.18
• AWSSDK.SimpleNotificationService: 3.3.101.11

Additional context

[astuyve]

Hey @jnyrup thanks for reaching out, this is a known issue and we're working on a fix here

[jnyrup]

Looking forward to see a release with the fix then 👍

I hate to be the person asking, but since it was fixed in DataDog/dd-trace-py#3404 15 months ago, is there are way you could have known about this and avoided enabling SNS support for dotnet before including a similar fix?

[astuyve]

Hey @jnyrup that's my fault personally, I didn't properly communicate this possibility when assigning the task, and it seems that there was misunderstanding about AWS actually fixing this issue within SNS, it appears they have not.

I apologize for overlooking this risk.

[jnyrup]

I'm sorry, by "you" I definitely didn't intent to refer directly to you personally or any other individual on the DataDog teams, but merely to general cross-team collaboration.

Feel free to close this issue or keep it as tracking issue for the fixing PR.

[zARODz11z]

If you'd like to continue using Tracer version: 2.32.0 you can disable the sns integration using DD_TRACE_AwsSns_ENABLED=0 I take responsibility for this issue as well, apologies for any inconvenience. The release is coming soon.

[pierotibou]

Fix released in https://github.com/DataDog/dd-trace-dotnet/releases/tag/v2.33.0