list_models() of dataset containing a BigQuery AutoML models errors with Invalid enum value AUTOML_REGRESSOR for enum type

[romainr]

Listing standard BigQuery models (e.g. LINEAR_REGRESSION or https://cloud.google.com/bigquery-ml/docs/bigqueryml-mf-explicit-tutorial#run_the_create_model_query) works, but AutoML model (https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-create-automl#create_model_syntax) seems to fail?

Environment details

• OS type and version: Ubuntu 20.04
• Python version: Python 3.8.2 (default, Jul 16 2020, 14:00:26)
• pip version: pip 20.2

pip3 freeze | grep goo
google-api-core==1.22.3
google-auth==1.22.0
google-cloud-bigquery==2.0.0
google-cloud-core==1.4.2
google-crc32c==1.0.0
google-resumable-media==1.0.0
googleapis-common-protos==1.6.0

Steps to reproduce

1. Create a BigQuery AutoML model, e.g. https://cloud.google.com/bigquery-ml/docs/bigqueryml-mf-explicit-tutorial
2. Use the API to list the models

Code example

from google.cloud import bigquery

client = bigquery.Client()                
                
for d in client.list_datasets():
  for m in client.list_models(d.reference):
    print(m.friendly_name)

Stack trace

Traceback (most recent call last):
  File " python3.8/site-packages/google/protobuf/json_format.py", line 745, in _ConvertScalarFieldValue
    number = int(value)
ValueError: invalid literal for int() with base 10: 'AUTOML_REGRESSOR'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File " python3.8/site-packages/google/protobuf/json_format.py", line 589, in _ConvertFieldValuePair
    setattr(message, field.name, _ConvertScalarFieldValue(value, field))
  File " python3.8/site-packages/google/protobuf/json_format.py", line 748, in _ConvertScalarFieldValue
    raise ParseError('Invalid enum value {0} for enum type {1}.'.format(
google.protobuf.json_format.ParseError: Invalid enum value AUTOML_REGRESSOR for enum type google.cloud.bigquery.v2.Model.ModelType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File " python3.8/site-packages/google/api_core/page_iterator.py", line 213, in _items_iter
    for item in page:
  File " python3.8/site-packages/google/api_core/page_iterator.py", line 133, in next
    result = self._item_to_value(self._parent, item)
  File " python3.8/site-packages/google/cloud/bigquery/client.py", line 3241, in _item_to_model
    return Model.from_api_repr(resource)
  File " python3.8/site-packages/google/cloud/bigquery/model.py", line 308, in from_api_repr
    this._proto = json_format.ParseDict(
  File " python3.8/site-packages/google/protobuf/json_format.py", line 452, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File " python3.8/site-packages/google/protobuf/json_format.py", line 483, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File " python3.8/site-packages/google/protobuf/json_format.py", line 592, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse modelType field: Invalid enum value AUTOML_REGRESSOR for enum type google.cloud.bigquery.v2.Model.ModelType..

[HemangChothani]

@romainr Thanks for the report, I am able to reproduce it.
@tswast This is is related to gapic (autogen) files, I found the enum mapping for LINEAR_REGRESSION and others in types/model.py, but not able to found for the AUTOML_REGRESSOR so, when protobuf trying to fetch enum value it gets None value and failed at the time of int conversion.

I think it's external issue.

[tswast]

We're working on refreshing the protocol buffers at https://github.com/googleapis/googleapis/tree/master/google/cloud/bigquery/v2 which are used to generate this code.

@romainr does this happen at the time of list_models() or when the enum property is accessed? I wonder if it would be a useful workaround in the meantime to add some lazy parsing logic to this generated code to not fail on missing enum values?

CC @shollyman

[HemangChothani]

It happen when we execute list(list_models()) line, I have assigned a static number when it tried to fetch enum_value by the name and it works fine.

[romainr]

Thanks for confirming!

In the meantime, @HemangChothani, would you be able to share your "workaround"?

[HemangChothani]

@romainr What i have done is just to check the actual cause of error, it's not the right way to solve it like to edit generated code files and proto files.