botocore distributed tracing breaks SNS filter policies (by exposing an AWS bug)

[mscheifer]

Which version of dd-trace-py are you using?

0.58.2

Which version of pip are you using?

21.2.4

Which version of the libraries are you using?

botocore @ file:///pip-packages/botocore-1.23.7-py3-none-any.whl

How can we reproduce your problem?

Create an SNS topic and subscription (simplest subscription is probably email). Set a filter policy on the subscription. Create a Python service that publishes to this SNS topic. Observe that the messages pass the filter. Install the ddtrace library and patch botocore (use ddtrace-run). Observe that the messages no longer pass the filter.

What is the result that you get?

No message with the injected datadog attribute reaches subscriptions with any kind of filter policy set

What is the result that you expected?

ddtrace patching shouldn't change the behavior of which messages get filtered out.

So this isn't really your guys fault. It seems like a bug in AWS and I've created an issue over there: https://repost.aws/questions/QUyUJ6EvX6So2I1MAS-c-dmg/an-sns-message-with-a-double-quote-in-any-attribute-is-filtered-out-is-this-a-bug but probably something should be done in the meantime here to work around it until AWS addresses it. The problem is that the _datadog attribute you inject uses a JSON blob for the value which includes unescaped quote characters thus triggering the bug.

As a workaround for now we can set ddtrace.config.botocore['distributed_tracing'] or DD_BOTOCORE_DISTRIBUTED_TRACING to false

[astuyve]

Hi @mscheifer - thanks for reaching out. We've confirmed with AWS that this is a bug in SNS, but we've got a workaround which uses the binary type instead of the string type for the _datadog MessageAttribute and thus is not subject to this bug.

We're working on the fix now and will update this ticket when it is released.

[sindrig]

Is there anything more to the workaround than changing the datatype here? Is there anything we can do to help, or do you have an ETA for the fix?

[astuyve]

Unfortunately it appears (and is confirmed by AWS support) that passing a well-formatted JSON string as a String-typed value in Message Attributes seems to break SNS Filters entirely, so adding _datadog to a filter policy on your subscription won't work either.

The ETA is as soon as possible, but there are a couple of moving pieces between this library and the dd-lambda-python library, where we depend on this.

[brettlangdon]

Fixed in #3404.

We will work on getting this fix released.

[zARODz11z]

@mscheifer @sindrig Thank you for reporting the issue, the fix is released in v3.54.0 of DD Lambda Python and Datadog Lambda Python Layer 54, as well as DD Trace Py v0.59.2

[Kyle-Verhoog]

Going to close the issue as the fix has been released. Please re-open if the issue is still present! 🙂