Retry flush metrics from ThreadStats to Datadog over RemoteDisconnected errors.

[tianchu]

What does this PR do?

Retry metric flushing from ThreadStats to Datadog over RemoteDisconnected errors. The root issue https://hg.python.org/cpython/rev/eba80326ba53 has been discussed extensively for years, but from what I can tell no fix was implemented at any level. The main reason to not fix (e.g., retry at the http client level) seems to be concerns with POST requests are often not idempotent and not always safe to retry, which is generally speaking true. But according to my comprehensive tests (one submission per minute for 2 days), all the failures resulted in missing data and therefore were safe to be retried. The testing function has been running for a day and I did not see any duplicate date points.

Based on the lengthy discussion on the internet over this particular issue, I don't think we can really wait for a true/better fix in the underlying libraries. The benefits of moving forward with a retry far outweigh the potential concerns over idempotency.

Specific changes are:

• Unmute the datadog api client to surface the ClientError to datadog-lambda, so we can catch and handle it
• Retry once if the error is RemoteDisconnected, otherwise log the exception for debugging

Motivation

Metric data points are lost sporadically when being submitted synchronously from the Datadog Lambda library to Datadog API (i.e., without using the Forwarder Lambda or Lambda Extension). The specific error reads below:

datadog.api.exceptions.ClientError: Could not request POST https://api.datadoghq.com/api/v1/distribution_points: 
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). 
Please check the network connection or try again later. If the problem persists, please contact [email protected]

Testing Guidelines

Additional Notes

Types of Changes

• Bug fix
• New feature
• Breaking change
• Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

• This PR's description is comprehensive
• This PR contains breaking changes that are documented in the description
• This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
• This PR impacts documentation, and it has been updated (or a ticket has been logged)
• This PR's changes are covered by the automated tests
• This PR collects user input/sensitive content into Datadog
• This PR passes the integration tests (ask a Datadog member to run the tests)

[hghotra]

Suggested change

	# The nature of the root issue https://bugs.python.org/issue41345 is to be complex,
	# The nature of the root issue https://bugs.python.org/issue41345 is complex,

[hghotra]

💯