google.auth.exceptions.TransportError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[nitishxp]

Hi,

I am using using Cloud function to call google api. Below is the version of of auth package that i am using

google-auth==1.24.0

Below is the error that i am getting

Error while updating environment ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/opt/python3.8/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/opt/python3.8/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/opt/python3.8/lib/python3.8/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen retries = retries.increment( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment raise six.reraise(type(error), error, _stacktrace) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/layers/google.python.pip/pip/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "/opt/python3.8/lib/python3.8/http/client.py", line 1347, in getresponse response.begin() File "/opt/python3.8/lib/python3.8/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/opt/python3.8/lib/python3.8/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 182, in __call__ response = self.session.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/main.py", line 993, in upgradeEnvironment fn(version) File "/workspace/main.py", line 351, in fn response = authed_session.request('GET', "https://logging.googleapis.com/v2/projects/" + project + "/sinks", headers=headers) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 460, in request self.credentials.before_request(auth_request, method, url, request_headers) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request self.refresh(request) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/service_account.py", line 361, in refresh access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/_client.py", line 153, in jwt_grant response_data = _token_endpoint_request(request, token_uri, body) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/oauth2/_client.py", line 105, in _token_endpoint_request response = request(method="POST", url=token_uri, headers=headers, body=body) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 188, in __call__ six.raise_from(new_exc, caught_exc) File "<string>", line 3, in raise_from google.auth.exceptions.TransportError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Below is my code which i am using

from google.auth.transport.requests import AuthorizedSession
authed_session = AuthorizedSession(credentials)
authed_session.request('GET', "https://logging.googleapis.com/v2/projects/<>/sinks", params=<>)

Does google-auth library suppose to retry or an retry capability should be added to handle all this error?

[busunkim96]

Hi @nitishxp,

Can you provide more detail on when the error appears/how often it happens? There are some known issues with client connections going stale in Cloud Functions depending on the invocation pattern.

It looks like Connection aborted is not currently in the list of retried errors, but you can customize this when you create the AuthorizedSession.

google-auth-library-python/google/auth/transport/requests.py

Lines 325 to 333 in 36e6f0f

	def __init__(
	self,
	credentials,
	refresh_status_codes=transport.DEFAULT_REFRESH_STATUS_CODES,
	max_refresh_attempts=transport.DEFAULT_MAX_REFRESH_ATTEMPTS,
	refresh_timeout=None,
	auth_request=None,
	default_host=None,
	):

google-auth-library-python/google/auth/transport/__init__.py

Lines 32 to 38 in 36e6f0f

	DEFAULT_REFRESH_STATUS_CODES = (http_client.UNAUTHORIZED,)
	"""Sequence[int]: Which HTTP status code indicate that credentials should be
	refreshed and a request should be retried.
	"""
	DEFAULT_MAX_REFRESH_ATTEMPTS = 2
	"""int: How many times to refresh the credentials and retry a request."""

@arithmetic1728 Should we retry Connection aborted by default?

[nitishxp]

Hi @busunkim96
The error was not frequent but we got many of those errors in production system in past 2 days. The invocation of CF is pub sub based.

I would be glad if you can add ConnectionError in list of retry codes since i reach out individually to python-bigquery and python-storage to added this to there list of retry.

[arithmetic1728]

@busunkim96 Yes, I think it makes sense to retry on Connection aborted by default in google-auth, so the client libraries don't need to do so individually.

[nitishxp]

@busunkim96

Will the below connectionError will be fixed in this bug?

"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 561, in create_table api_response = self._call_api( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 574, in _call_api return call() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 472, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 336, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 374, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 464, in request response = super(AuthorizedSession, self).request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[nitishxp]

@busunkim96 Could you also add https://github.com/googleapis/python-bigquery/blob/6a48e80bc7d347f381b181f4cf81fef105d0ad0d/google/cloud/bigquery/retry.py#L26 in list of retry

[busunkim96]

@parthea added the retries to the api-core layer, so they don't need to be retried here.

We ultimately decided to put them in api-core as it is easier for the end user to adjust, and api-core already has retry/timeout logic built in.

[nitishxp]

@busunkim96 Can we add this https://docs.python.org/3/library/exceptions.html#ConnectionError

[fbraza]

Hello @busunkim96

I am also facing a similar issue:

ERROR 2023-01-05T13:00:22.935075Z Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/opt/app/sunalgo_worker/app.py", line 51, in run_scoring processed_night: ProcessedNight = worker.custom_process( File "/opt/app/sunalgo_worker/core/services/worker.py", line 52, in custom_process save_features( File "/opt/app/sunalgo_worker/core/services/worker.py", line 113, in save_features gcs_files_data_access.save_file_with_content( File "/opt/app/sunalgo/data_access/gcs_files_data_access.py", line 98, in save_file_with_content blob.upload_from_string(content, timeout=300) File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2814, in upload_from_string self.upload_from_file( File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2539, in upload_from_file created_json = self._do_upload( File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2370, in _do_upload response = self._do_resumable_upload( File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2197, in _do_resumable_upload upload, transport = self._initiate_resumable_upload( File "/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py", line 2078, in _initiate_resumable_upload upload.initiate( File "/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 420, in initiate return _request_helpers.wait_and_retry( File "/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 171, in wait_and_retry raise error File "/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py", line 148, in wait_and_retry response = func() File "/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/upload.py", line 412, in retriable_request result = transport.request( File "/usr/local/lib/python3.10/site-packages/google/auth/transport/requests.py", line 549, in request response = super(AuthorizedSession, self).request( File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 547, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  {
    "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.10/site-packages/flask/app.py\", line 2525, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/usr/local/lib/python3.10/site-packages/flask/app.py\", line 1822, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/usr/local/lib/python3.10/site-packages/flask/app.py\", line 1820, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/local/lib/python3.10/site-packages/flask/app.py\", line 1796, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n  File \"/opt/app/sunalgo_worker/app.py\", line 51, in run_scoring\n    processed_night: ProcessedNight = worker.custom_process(\n  File \"/opt/app/sunalgo_worker/core/services/worker.py\", line 52, in custom_process\n    save_features(\n  File \"/opt/app/sunalgo_worker/core/services/worker.py\", line 113, in save_features\n    gcs_files_data_access.save_file_with_content(\n  File \"/opt/app/sunalgo/data_access/gcs_files_data_access.py\", line 98, in save_file_with_content\n    blob.upload_from_string(content, timeout=300)\n  File \"/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py\", line 2814, in upload_from_string\n    self.upload_from_file(\n  File \"/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py\", line 2539, in upload_from_file\n    created_json = self._do_upload(\n  File \"/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py\", line 2370, in _do_upload\n    response = self._do_resumable_upload(\n  File \"/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py\", line 2197, in _do_resumable_upload\n    upload, transport = self._initiate_resumable_upload(\n  File \"/usr/local/lib/python3.10/site-packages/google/cloud/storage/blob.py\", line 2078, in _initiate_resumable_upload\n    upload.initiate(\n  File \"/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/upload.py\", line 420, in initiate\n    return _request_helpers.wait_and_retry(\n  File \"/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py\", line 171, in wait_and_retry\n    raise error\n  File \"/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/_request_helpers.py\", line 148, in wait_and_retry\n    response = func()\n  File \"/usr/local/lib/python3.10/site-packages/google/resumable_media/requests/upload.py\", line 412, in retriable_request\n    result = transport.request(\n  File \"/usr/local/lib/python3.10/site-packages/google/auth/transport/requests.py\", line 549, in request\n    response = super(AuthorizedSession, self).request(\n  File \"/usr/local/lib/python3.10/site-packages/requests/sessions.py\", line 587, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/local/lib/python3.10/site-packages/requests/sessions.py\", line 701, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/local/lib/python3.10/site-packages/requests/adapters.py\", line 547, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))",
    "insertId": "63b6c9e6000e44a3d8fb6f75",
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "project_id": "anonymous",
        "location": "europe-west1",
        "configuration_name": "worker-dev",
        "service_name": "worker-dev",
        "revision_name": "worker-dev-00037-xer"
      }
    },
    "timestamp": "2023-01-05T13:00:22.935075Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "0071bb4815d471f2e39abf2b1f53e35c295edfb3918d3b0d17288b6d86e57ec6b1140cb28e3bad93351496854fa8dc826322092e3cc27444e3b92f060fe9d30454"
    },
    "logName": "projects/anonymous/logs/run.googleapis.com%2Fstderr",
    "receiveTimestamp": "2023-01-05T13:00:23.268775833Z"
  }

You mentioned in your previous message that the retires were added to the api-core layer. I am not sure what this means.
Do the user still need to kind of catch and retry the request himself? Or will a fix be merged soon in future version of the library?

As @nitishxp we do have this error popping out in our CloudRun pipelines.

Many thanks for your feedback

Cheers

[Skorpyon]

Any update here? 3 years of issue present...

Traceback (most recent call last):
  File "/usr/src/app/notifications/services/notification_service/providers/mobile.py", line 94, in _send_notification
    firebase.send(fb_message, app=firebase_app)
  File "/usr/local/lib/python3.10/site-packages/firebase_admin/messaging.py", line 121, in send
    return _get_messaging_service(app).send(message, dry_run)
  File "/usr/local/lib/python3.10/site-packages/firebase_admin/messaging.py", line 411, in send
    resp = self._client.body(
  File "/usr/local/lib/python3.10/site-packages/firebase_admin/_http_client.py", line 131, in body
    resp = self.request(method, url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/firebase_admin/_http_client.py", line 118, in request
    resp = self._session.request(method, self.base_url + url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/google/auth/transport/requests.py", line 577, in request
    self.credentials.refresh(auth_request)
  File "/usr/local/lib/python3.10/site-packages/google/oauth2/service_account.py", line 445, in refresh
    access_token, expiry, _ = _client.jwt_grant(
  File "/usr/local/lib/python3.10/site-packages/google/oauth2/_client.py", line 308, in jwt_grant
    response_data = _token_endpoint_request(
  File "/usr/local/lib/python3.10/site-packages/google/oauth2/_client.py", line 268, in _token_endpoint_request
    response_status_ok, response_data, retryable_error = _token_endpoint_request_no_throw(
  File "/usr/local/lib/python3.10/site-packages/google/oauth2/_client.py", line 215, in _token_endpoint_request_no_throw
    request_succeeded, response_data, retryable_error = _perform_request()
  File "/usr/local/lib/python3.10/site-packages/google/oauth2/_client.py", line 191, in _perform_request
    response = request(
  File "/usr/local/lib/python3.10/site-packages/google/auth/transport/requests.py", line 192, in __call__
    raise new_exc from caught_exc
google.auth.exceptions.TransportError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Firebase Python SDK still affected, because connection aborted didn't included in retryable list.