Slow cloud request/response tagging in Docker Lambda function with datadog-lambda

[maxfriedrich]

Hi, not sure if this is the right place or it should rather go to ddtrace or somewhere else:

Reproduction repo with CDK: https://github.com/maxfriedrich/datadog-cloud-payload-tagging-repro

The cloud request/response payload tagging feature causes a big slowdown for us for any DynamoDB query, even when "disabled" with the environment variable value "$.*" (0.xx seconds to > 10 seconds query), with the following setup:

• Docker Lambda based on the Lambda Python 3.11 (or 3.13) base image
• Installing the Datadog Lambda extension into the container
• Installing datadog-lambda and ddtrace with into the container
• Running the function with datadog_lambda.handler

(this function outgrew the zip size limit, this is why it's built like this -- is this a supported setup?)

We can work around it by setting the environment variable values to "", fully disabling the feature, not just redacting all fields.

The problem only occurs with a Docker based Lambda function, the handler from my reproduction is fine in the Python runtime and the Datadog extension and Lambda layers.

Output from my test script:

Testing slow function: arn:aws:lambda:eu-central-1:...:function:DdTaggingReproStack-Function1904AA941-QLjJ2nwy96AV
Execution time: 11.545 seconds
Execution time: 10.865 seconds
Execution time: 11.279 seconds
Execution time: 11.604 seconds
Execution time: 11.659 seconds

Testing fast function: arn:aws:lambda:eu-central-1:...:function:DdTaggingReproStack-Function28EA0674A-36P7xO9x9CbG
Execution time: 0.235 seconds
Execution time: 0.188 seconds
Execution time: 0.168 seconds
Execution time: 0.194 seconds
Execution time: 0.206 seconds

Do you have any ideas what could cause this? Feel free to move the issue if it belongs somewhere else. Thank you very much!

[joeyzhao2018]

Thank you for reporting this issue and sorry for the inconvenience.
Payload tagging feature does introduce some overhead. Usually it's negligible but a large-size payload can potentially cause some delays.
If you REMOVE the ENVs DD_TRACE_CLOUD_REQUEST_PAYLOAD_TAGGING and DD_TRACE_CLOUD_RESPONSE_PAYLOAD_TAGGING, they should be completely turned off. I recommended turn if off by removing the envs for now. I'll take a closer look at the reproduction repo and get back to you.

[maxfriedrich]

Hi, it only happens in this setup with Docker though, a Python runtime Lambda function does not have this 10 sec (!!) overhead for the small request I'm sending with empty response.

The fact that it's 10 seconds + a little bit is suspicious to me, maybe waiting for some internal timeout?