An error about Protobuf size limits : tensorflow.GraphDef exceeds maximum protobuf size of 2GB

[zhaiyubo]

Describe the bug
I'm using the following command to convert Tensorflow model to ONNX, then an error occurred

   python3 -m tf2onnx.convert --saved-model ./tfModel/ --output ./onnxModel

System information

• OS Platform and Distribution : Red Hat 4.8.2-16
• Tensorflow Version: 2.2
• Python version: 3.6.8

[arthurzhai@VM-44-159-centos ~/onnxTest]$ python3 -m tf2onnx.convert --saved-model ./tfModel/ --output ./
2020-08-05 17:30:24,174 - WARNING - '--tag' not specified for saved_model. Using --tag serve
WARNING:tensorflow:From /usr/local/lib64/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
2020-08-05 17:30:27,334 - WARNING - From /usr/local/lib64/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
2020-08-05 17:30:34,886 - INFO - Signatures found in model: [serving_default].
2020-08-05 17:30:34,887 - WARNING - '--signature_def' not specified, using first signature: serving_default
WARNING:tensorflow:Issue encountered when serializing global_step.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2020-08-05 17:30:35,158 - WARNING - Issue encountered when serializing global_step.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
WARNING:tensorflow:Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2020-08-05 17:30:35,158 - WARNING - Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
WARNING:tensorflow:Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2020-08-05 17:30:35,158 - WARNING - Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/site-packages/tf2onnx/convert.py", line 171, in
main()
File "/usr/local/lib/python3.6/site-packages/tf2onnx/convert.py", line 132, in main
args.saved_model, args.inputs, args.outputs, args.tag, args.signature_def, args.concrete_function)
File "/usr/local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 288, in from_saved_model
_from_saved_model_v2(model_path, input_names, output_names, tag, signatures, concrete_function)
File "/usr/local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 277, in _from_saved_model_v2
frozen_graph = from_function(concrete_func, inputs, outputs)
File "/usr/local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 100, in from_function
frozen_func = convert_variables_to_constants_v2(func, lower_control_flow=False)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 680, in convert_variables_to_constants_v2
return _construct_concrete_function(func, output_graph_def, converted_inputs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 406, in _construct_concrete_function
new_output_names)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 633, in function_from_graph_def
wrapped_import = wrap_function(_imports_graph_def, [])
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 611, in wrap_function
collections={}),
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 86, in call
return self.call_with_variable_creator_scope(self._fn)(*args, **kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 92, in wrapped
return fn(*args, **kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 631, in _imports_graph_def
importer.import_graph_def(graph_def, name="")
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def
producer_op_list=producer_op_list)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/framework/importer.py", line 494, in _import_graph_def_internal
with c_api_util.tf_buffer(graph_def.SerializeToString()) as serialized:
ValueError: Message tensorflow.GraphDef exceeds maximum protobuf size of 2GB: 2159033923

Additional context
How can I fix it? Please help me

[TomWildenhain-Microsoft]

Unfortunately we can't convert models that exceed the maximum protobuf size. This is a known issue and is on our todo list.

[zhaiyubo]

@TomWildenhain-Microsoft Thanks for your reply. I wonder if only tf2onnx has a 2GB size limit, or pytorch2onnx , caffe2onnx also have the limit.

[guschmue]

pytorch exporter does support >2GB but you need to call it with torch.onnx.export(..., use_external_data_format=True).
caffe2onnx has the 2GB issue.

tf2onnx doesn't do it yet for 2 reasons:

• we have not added the support for the external data format (that should not be to hard to add)
• we call tensorflow convert_variables_to_constants_v2() and that itself seems to have an issue

[zhaiyubo]

@guschmue Very clear explanation. Thanks a lot

[guschmue]

fixed with #1090

[josephrocca]

For those who missed it like I did the first time, the above-referenced pull request introduced the large_model parameter/flag which needs to be used to bypass the 2GB limit. If you're using the command line tool, add --large_model to your command, if you're using tf2onnx.convert.from_function, add the large_model=True parameter.

(I originally assumed that the above pull request raised the limit by default)

[lsmliborio]

The model was successfully exported after converting using --large_model, but now it says it should be unzipped before using onnxruntime. How should I unzip it?