Spanner: Error message for non-existent spanner instance is hard to debug.

[crwilcox]

When an invalid (non-existent) instance id is specified the error you receive isn't intuitive.

spanner_client = spanner.Client()

# Get an instance, but provide a non-existent instance id
instance = spanner_client.instance('non_existent_spanner_instance')

database = instance.database('real_database_id_on_different_instance')
with database.snapshot() as snapshot:
  results = snapshot.execute_sql('SELECT 1')

  for row in results:
      print(row)

Expected

An error indicating my instance doesn't exist.

Actual

google.api_core.exceptions.InvalidArgument: 400 Invalid CreateSession request.

 File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/database.py", line 499, in __enter__
    session = self._session = self._database._pool.get()
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/pool.py", line 268, in get
    session.create()
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/session.py", line 116, in create
    session_pb = api.create_session(self._database.name, metadata=metadata, **kw)
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 290, in create_session
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/google/home/donghuiz/.local/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/usr/lib/python2.7/dist-packages/six.py", line 737, in raise_from
    raise value
google.api_core.exceptions.InvalidArgument: 400 Invalid CreateSession request.

[tseaver]

@crwilcox I'm not sure how we can fix that: is there a more useful message in the details of the exception we can surface? We don't want to implicitly generated an 'instance.get' call early to enforce instance existence.

[mf2199]

A simple check whether the instance exists results in another error:
InvalidArgument: 400 Invalid GetInstance request.

This shouldn't happen, as Instance.exists() method is supposed to simply return False if the instance does not exists. Instead, it throws an exception. Could it be a matter for another issue?

[tseaver]

@mf2199 I can't reproduce that with a "legal" instance ID:

Python 3.7.1 (default, Oct 22 2018, 11:31:38) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud.spanner_v1 import Client
>>> client = Client()
>>> instance = client.instance('nonesuch-instance-exists')
>>> instance.exists()
False

Hyphens are allowed in instance names, but underscores are not, per the Instance message API docs.

[mf2199]

@tseaver Got it, thanks! The issue can be circumvented by adding some checkpoints inside the Database class' batch() and snapshot() methods. However, as was noted in #220, this can also be readdressed as a feature request and solved at the server side. What do you think?

[tseaver]

Making an unasked-for API call to validate existence of the instance / database before returning a session has undesirable performance / billing consequences.

Trying to reproduce with a valid (hyphens, not underscores) but non-existent instance id:

>>> from google.cloud import spanner
>>> client = spanner.Client()
>>> instance = client.instance('nonesuch-instance-exists')
>>> instance.exists()
False
>>> database = instance.database('valid-database-name')
>>> database.exists()
False
>>> with database.snapshot() as snapshot:
...     results = snapshot.execute_sql('SELECT 1')
...     for row in results:
...         print(row)
... 
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/pool.py", line 264, in get
    session = self._sessions.get_nowait()
  File "/opt/Python-3.7.1/lib/python3.7/queue.py", line 198, in get_nowait
    return self.get(block=False)
  File "/opt/Python-3.7.1/lib/python3.7/queue.py", line 167, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.NOT_FOUND
	details = "Instance not found: projects/citric-celerity-697/instances/nonesuch-instance-exists"
	debug_error_string = "{"created":"@1564156732.715693901","description":"Error received from peer ipv6:[2607:f8b0:4004:801::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Instance not found: projects/citric-celerity-697/instances/nonesuch-instance-exists","grpc_status":5}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/database.py", line 499, in __enter__
    session = self._session = self._database._pool.get()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/pool.py", line 267, in get
    session.create()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/session.py", line 117, in create
    session_pb = api.create_session(self._database.name, metadata=metadata, **kw)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/gapic/spanner_client.py", line 307, in create_session
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.NotFound: 404 Instance not found: projects/citric-celerity-697/instances/nonesuch-instance-exists

Which seems like a pretty helpful messsage. I think making the message for and invalid instance ID more useful is an issue for the back-end API, not for google-cloud-spanner. It is true for other RPCs, as well, e.g.:

>>> invalid_instance.exists()
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/.nox/system-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Invalid GetInstance request."
	debug_error_string = "{"created":"@1564156898.102875705","description":"Error received from peer ipv6:[2607:f8b0:4004:801::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Invalid GetInstance request.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_v1/instance.py", line 236, in exists
    api.get_instance(self.name, metadata=metadata)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/spanner/google/cloud/spanner_admin_instance_v1/gapic/instance_admin_client.py", line 609, in get_instance
    request, retry=retry, timeout=timeout, metadata=metadata
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 273, in retry_wrapped_func
    on_error=on_error,
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/retry.py", line 182, in retry_target
    return target()
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/gcp/api_core/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Invalid GetInstance request.

[mf2199]

@tseaver, I suspected so, thanks for the informative response. I'd suggest this issue be redirected to the back-end team(s), given all the above argumentation.

[tseaver]

@crwilcox Can you raise the issue of the error messages for invalid (not missing) instance IDs to the Spanner team?