Json Error

[nitishxp]

Lib: google-cloud-bigquery==2.23.1

Getting below errors occasionally when calling client.delete_table(<>, not_found_ok=True)

File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py",
line 1788, in delete_table self._call_api( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/bigquery/client.py",
line 747, in _call_api return call() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py",
line 286, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py",
line 189, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http.py",
line 473, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http.py",
line 337, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/_http.py",
line 375, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/auth/transport/requests.py",
line 478, in request self.credentials.before_request(auth_request, method, url, request_headers) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/auth/credentials.py",
line 133, in before_request self.refresh(request) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/service_account.py",
line 376, in refresh access_token, expiry, _ = _client.jwt_grant( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/_client.py",
line 193, in jwt_grant response_data = _token_endpoint_request(request, token_uri, body) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/_client.py",
line 161, in _token_endpoint_request response_status_ok, response_data = _token_endpoint_request_no_throw( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/oauth2/_client.py",
line 121, in _token_endpoint_request_no_throw response_data = json.loads(response_body) File "/opt/python3.9/lib/python3.9/json/__init__.py",
line 346, in loads return _default_decoder.decode(s) File "/opt/python3.9/lib/python3.9/json/decoder.py",
line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/python3.9/lib/python3.9/json/decoder.py",
line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[tswast]

I believe this happens when an intermediate server fails, so the BigQuery server doesn't get a chance to send back a proper JSON response. I suspect we'll want to detect this and retry.

[jimfulton]

This looks like a bug in https://github.com/googleapis/google-auth-library-python

In logic to retry requests to freshen credentials, it's trying to decode the body before checking the status:

google-auth-library-python/google/oauth2/_client.py

Lines 119 to 121 in 08c987d

	response_data = json.loads(response_body)
	if response.status == http.client.OK:

If it checked the status first, assuming that the status wasn't 200:

1. It would retry a couple of times
2. And if it still failed, would probably fail with something more informative, possibly something we're already retrying at a higher level.

If the status was 200, I wonder what the content was. Was it an empty string? Or did it have bad data?
I don't think code should blindly decode the data without a try/except.

FWIW, I've seen this bug over and over, including in my own code. :)

[jimfulton]

@tswast shall I work up a PR for google-auth-library-python?

[tswast]

Good find! Yes, please do work on that. I'll move this issue to the google-auth repo.

[arithmetic1728]

downgrade to p3