Add retry mechanism to telemetry requests #617
Conversation
Signed-off-by: Sai Shree Pradhan <[email protected]>
saishreeeee
requested review from
deeksha-db,
samikshya-db,
jprakash-db,
jackyhu-db,
madhav-db,
gopalldb,
jayantsing-db,
vikrantpuppala and
shivam2680
as code owners
saishreeeee
changed the title
saishreeeee
changed the title
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
|
|
||
| from databricks.sql.telemetry.telemetry_client import TelemetryClientFactory | ||
|
|
||
| telemetry_client = TelemetryClientFactory.get_telemetry_client(session_id_hex) |
|
|
||
| # Check that the request was retried (should be 2 calls: initial + 1 retry) | ||
| assert mock_get_conn.return_value.getresponse.call_count == 2 | ||
| assert "Retrying after" in caplog.text |
There was a problem hiding this comment.
can we collapse these tests a bit i.e. test 503, 429, retry-after header, retry count etc in a single test (first response returns 503, second does 429 etc), appreciate the detailed testing but i would like it to be more readable and maintainable in the long term
| self.wait_for_async_request() | ||
| TelemetryClientFactory.close(client._session_id_hex) | ||
|
|
||
| # Based on the logs, 400 IS being retried (this is the actual behavior for CommandType.OTHER) |
There was a problem hiding this comment.
400 is being retried? are you sure?
|
|
||
| # 403 should not be retried based on the retry policy | ||
| mock_get_conn.return_value.getresponse.assert_called_once() | ||
| assert "Telemetry request failed with status code: 403" in caplog.text |
There was a problem hiding this comment.
would like to collapse 4xx error codes into a single test with something like parametrized tests
| @@ -31,6 +33,24 @@ | |||
| logger = logging.getLogger(__name__) | |||
|
|
|||
|
|
|||
| class TelemetryHTTPAdapter(HTTPAdapter): | |||
There was a problem hiding this comment.
let's see if we can skip this adapter
| TELEMETRY_RETRY_STOP_AFTER_ATTEMPTS_COUNT = 3 | ||
| TELEMETRY_RETRY_DELAY_MIN = 0.5 # seconds | ||
| TELEMETRY_RETRY_DELAY_MAX = 5.0 # seconds | ||
| TELEMETRY_RETRY_STOP_AFTER_ATTEMPTS_DURATION = 30.0 |
There was a problem hiding this comment.
let's align these values across drivers or have these consistent across the python driver
| mock_response.isclosed.return_value = False | ||
| return mock_response | ||
|
|
||
| @pytest.mark.usefixtures("caplog") |
There was a problem hiding this comment.
do we really need caplog?
What type of PR is this?
Description
Retry mechanism for telemetry requests
How is this tested?
Related Tickets & Documents
PECOBLR-586