ValueError: Input 0 of node StatefulPartitionedCall was passed float from Func incompatible with expected resource.

[hoaquocphan]

Dear All,

I would like to report the issue when converting tensorflow saved model to onnx model

Environment:

$pip3 list
Package                       Version
----------------------------- -------------------
tb-nightly                    2.4.0a20201026
tensorboard                   2.3.0
tensorboard-plugin-wit        1.7.0
tensorflow                    2.3.1
tensorflow-addons             0.11.2
tensorflow-datasets           4.0.1
tensorflow-estimator          2.3.0
tensorflow-hub                0.9.0
tensorflow-metadata           0.24.0
tensorflow-model-optimization 0.5.0
tf2onnx                       1.7.1

Procedure:
1/ download saved model imagenet/inception_v1/classification
2/ run command to convert saved model to onnx model:

python3 -m tf2onnx.convert --output model.onnx --concrete_function 0 --saved-model imagenet_inception_v1_classification_4

error log:

python3 -m tf2onnx.convert --output model.onnx  --concrete_function 0 --saved-model imagenet_inception_v1_classification_4
2020-10-29 15:39:13.569001: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-10-29 15:39:13.569130: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-10-29 15:39:15,970 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2020-10-29 15:39:16.282882: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-10-29 15:39:16.283089: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-10-29 15:39:16.283213: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (hp): /proc/driver/nvidia/version does not exist
2020-10-29 15:39:16.283569: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-29 15:39:16.303386: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2712000000 Hz
2020-10-29 15:39:16.306374: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x62bbec0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-29 15:39:16.306519: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-29 15:39:20,079 - INFO - Signatures found in model: [].
2020-10-29 15:39:20.404054: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-10-29 15:39:20.404305: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-10-29 15:39:20.668518: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: graph_to_optimize
2020-10-29 15:39:20.668666: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: Graph size after: 2054 nodes (1820), 3308 edges (3074), time = 109.225ms.
2020-10-29 15:39:20.668718: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 1.878ms.
Traceback (most recent call last):
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 497, in _import_graph_def_internal
    graph._c_graph, serialized, options)  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input 0 of node StatefulPartitionedCall/StatefulPartitionedCall/train/InceptionV1/InceptionV1/Conv2d_1a_7x7/BatchNorm/AssignMovingAvg/AssignSubVariableOp was passed float from Func/StatefulPartitionedCall/StatefulPartitionedCall/input/_239:0 incompatible with expected resource.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/convert.py", line 185, in <module>
    main()
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/convert.py", line 136, in main
    args.signature_def, args.concrete_function, args.large_model)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 330, in from_saved_model
    _from_saved_model_v2(model_path, input_names, output_names, tag, signatures, concrete_function, large_model)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 317, in _from_saved_model_v2
    raise e
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 313, in _from_saved_model_v2
    frozen_graph = from_function(concrete_func, inputs, outputs, large_model)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 124, in from_function
    frozen_func = convert_variables_to_constants_v2(func, lower_control_flow=False)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1075, in convert_variables_to_constants_v2
    converted_input_indices)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1001, in _construct_concrete_function
    new_output_names)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 650, in function_from_graph_def
    wrapped_import = wrap_function(_imports_graph_def, [])
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 628, in wrap_function
    collections={}),
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 87, in __call__
    return self.call_with_variable_creator_scope(self._fn)(*args, **kwargs)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 93, in wrapped
    return fn(*args, **kwargs)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 648, in _imports_graph_def
    importer.import_graph_def(graph_def, name="")
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def
    producer_op_list=producer_op_list)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 501, in _import_graph_def_internal
    raise ValueError(str(e))
ValueError: Input 0 of node StatefulPartitionedCall/StatefulPartitionedCall/train/InceptionV1/InceptionV1/Conv2d_1a_7x7/BatchNorm/AssignMovingAvg/AssignSubVariableOp was passed float from Func/StatefulPartitionedCall/StatefulPartitionedCall/input/_239:0 incompatible with expected resource.

could anyone please help me to check this issue,
Thanks.

[TomWildenhain-Microsoft]

Concrete function 0 in that model is messed up. Try concrete function 1.
python3 -m tf2onnx.convert --output model.onnx --concrete_function 1 --saved-model imagenet_inception_v1_classification_4

[hoaquocphan]

Dear Mr @TomWildenhain-Microsoft

Thanks very much for your support
I have tried your method (update concrete function from 0 to 1;2;3) and it can convert many models,
But there are another issue with other models: imagenet/nasnet_large/classification

Error log:

python3 -m tf2onnx.convert --output model.onnx --concrete_function 2 --saved-model imagenet_nasnet_large_classification_4
2020-10-30 08:24:50.740873: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-10-30 08:24:50.741000: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-10-30 08:24:56,093 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2020-10-30 08:25:02.800544: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-10-30 08:25:02.800705: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-10-30 08:25:02.800774: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (hp): /proc/driver/nvidia/version does not exist
2020-10-30 08:25:02.801172: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-30 08:25:02.835980: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2712000000 Hz
2020-10-30 08:25:02.836193: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xbeedc80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-30 08:25:02.836214: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-30 08:25:25,687 - INFO - Signatures found in model: [].
2020-10-30 08:25:27.108574: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-10-30 08:25:27.109762: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-10-30 08:25:27.873460: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:816] Optimization results for grappler item: graph_to_optimize
2020-10-30 08:25:27.873612: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: Graph size after: 8115 nodes (6565), 13017 edges (11467), time = 416.284ms.
2020-10-30 08:25:27.873661: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:818]   function_optimizer: function_optimizer did nothing. time = 18.679ms.
WARNING:tensorflow:From /home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py:413: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2020-10-30 08:26:05,291 - WARNING - From /home/hoaphan/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py:413: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2020-10-30 08:26:06.473269: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-10-30 08:26:06.480821: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
Killed

So, could your have a check and give me your solution

Thanks very much

[TomWildenhain-Microsoft]

Interesting. I think I have previously been able to convert nasnet_large with those exact settings. Maybe your computer is running out of memory and the process is being killed? If you have enough disk space, try increasing the amount of virtual memory on your machine. Make sure in Task Manager the committed memory doesn't exceed the amount you set:

You can change virtual memory in Windows through Start -> search "Advanced System Settings" -> Performance Settings -> Advanced -> Virtual Memory -> Change -> Custom size

[TomWildenhain-Microsoft]

Converted model:
https://www.dropbox.com/s/5aw4pb4u1oxbce9/imagenet-nasnet_large-classification.onnx?dl=0

[hoaquocphan]

Hi Mr @TomWildenhain-Microsoft

I just searched this issue then find your comment in other post 1120#issuecomment-706273362 and I have checked all my models and see that this issue occur on all model that bigger than 160MB.
I think that this issue is because of lack of memory when conversion.

Thanks very much for your support.
I would like to close this ticket.

[hoaquocphan]

Hi Mr @TomWildenhain-Microsoft

Currently, I have the task related to optimize saved model. and it have the same error log with the previous task as below:
ValueError: Input 1 of node StatefulPartitionedCall was passed float from conv1/kernel:0 incompatible with expected resource.

How to reproduce:

1/ I download saved model from tensorflow hub: resnet_50_classification_1/
2/ I freeze model with below code:

def freeze_model(saved_model_dir, output_node_names, output_filename):
    output_graph_filename = os.path.join(saved_model_dir, output_filename)
    initializer_nodes = ''
    freeze_graph.freeze_graph(
        input_saved_model_dir=saved_model_dir,
        output_graph=output_graph_filename,
        saved_model_tags = tag_constants.SERVING,
        output_node_names=output_node_names,
        initializer_nodes=initializer_nodes,
        input_graph=None,
        input_saver=False,
        input_binary=False,
        input_checkpoint=None,
        restore_op_name=None,
        filename_tensor_name=None,
        clear_devices=False,
        input_meta_graph=False,
    )
    print('graph freezed!')

3/ optimize model by transform_graph tool

4/ I convert frozen model from step2 to saved model with below code:

def convert_graph_def_to_saved_model(export_dir, graph_filepath):
    if tf.io.gfile.exists(export_dir):
        tf.compat.v1.gfile.DeleteRecursively(export_dir)
    graph_def = get_graph_def_from_file(graph_filepath)
    k.clear_session()
    k.set_learning_phase(0)
    with tf.compat.v1.Session(graph=tf.Graph()) as session:
        tf.import_graph_def(graph_def, name='')
        tf.compat.v1.saved_model.simple_save(
            session,
            #k.get_session(),
            export_dir,
            inputs={
                node.name: session.graph.get_tensor_by_name(
                    '{}:0'.format(node.name))
                for node in graph_def.node if node.op=='serving_default_input_1'},
                #for node in graph_def.node if node.op=='Placeholder'},
            outputs={
                node.name: session.graph.get_tensor_by_name(
                    '{}:0'.format(node.name))
                for node in graph_def.node if node.op=='StatefulPartitionedCall'},
        )
        print('Optimized graph converted to SavedModel!')

full error log as below:

WARNING:tensorflow:From optimize_saved_model.py:146: set_learning_phase (from tensorflow.python.keras.backend) is deprecated and will be removed after 2020-10-11.
Instructions for updating:
Simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
Traceback (most recent call last):
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 497, in _import_graph_def_internal
    graph._c_graph, serialized, options)  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input 1 of node StatefulPartitionedCall was passed float from conv1/kernel:0 incompatible with expected resource.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "optimize_saved_model.py", line 216, in <module>
    convert_graph_def_to_saved_model(optimized_savedmodel_dir, optimized_graphdef_filepath)
  File "optimize_saved_model.py", line 148, in convert_graph_def_to_saved_model
    tf.import_graph_def(graph_def, name='')
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 405, in import_graph_def
    producer_op_list=producer_op_list)
  File "/home/hoaphan/.local/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 501, in _import_graph_def_internal
    raise ValueError(str(e))
ValueError: Input 1 of node StatefulPartitionedCall was passed float from conv1/kernel:0 incompatible with expected resource.

so could you have any solution to fix it?
Please help me

[TomWildenhain-Microsoft]

Hi @hoaquocphan, try downloading the model from TFHub, unzip it, and then convert it directly with:
python -m tf2onnx.convert --saved-model path/to/model --concrete_function 1 --opset 13

We will freeze the model automatically. The --concrete_function 1 is because this model has a few signatures and it turns out signature 1 is the one for inference, not training. I got the function index using the saved model cli that comes with tensorflow

saved_model_cli show --dir --all path/to/model

The second option has training set to false, and the functions are 0-indexed so that is --concrete_function 1.

[hoaquocphan]

Hi @TomWildenhain-Microsoft

currently, I'm doing the task to convert tensorflow saved model to onnx model and optimize it. I break the task to 3 parts as below:
Part A/ optimize saved model

• I follow the guide in https://medium.com/google-cloud/optimizing-tensorflow-models-for-serving-959080e9ddbf as my previous post and have the error while convert from graphdef file to saved model
  ValueError: Input 1 of node StatefulPartitionedCall was passed float from conv1/kernel:0 incompatible with expected resource.

Part B/ optimize model while convert saved model to onnx

• I use tf2onnx tool and add some options to optimize model

Part C/ optimize onnx model.

• I haven't do this part.

So I have some questions need the feedback from you:
1/ as your comment: "We will freeze the model automatically....", this mean the tf2onnx tool will optimize model as part A and when I do the part B, I can skip part A?

2/ do you know the rootcause of error code in Part A? as I refer in some forum, it because of the model train on tensorflow v1 and it conflict with tensorflow v2.

3/ when I use the tf2onnx to convert and optimize model as below command:
python3 -m tf2onnx.convert --saved-model tf2-preview_inception_v3_classification_4 --output tf2-preview_inception_v3_classification_4.onnx --concrete_function 2 --opset 13 --inputs inputs:0[1,416,416,3]
I see that option --concrete_function 2 will help to optimize model and reduce the output note, but option --concrete_function 1 as your comment do nothing to optimize model. so could you explain it?

[TomWildenhain-Microsoft]

1. Yes, you can skip step A and jump straight to B
2. Not all graphs can be frozen. If you try to freeze the training graph instead of the inference graph, you likely will fail because the training graph performs variable assignments in addition to variable reads. The error is saying that a node (likely a variable assignment node) was given a float (the frozen value of the variable) but was expecting a resource (the mutable variable).
3. The model may contain multiple graphs of different sizes. --concrete_function 2 might be significantly smaller because it is not the primary graph in the model. Optimization should make the model faster, but it won't necessarily make it smaller (the majority of the model size is from the weights, and they won't change). tf2onnx is primarily a model converter, not optimizer.

[TomWildenhain-Microsoft]

@hoaquocphan were you able to get the model converted?

[debloisg]

1. Yes, you can skip step A and jump straight to B
2. Not all graphs can be frozen. If you try to freeze the training graph instead of the inference graph, you likely will fail because the training graph performs variable assignments in addition to variable reads. The error is saying that a node (likely a variable assignment node) was given a float (the frozen value of the variable) but was expecting a resource (the mutable variable).
3. The model may contain multiple graphs of different sizes. --concrete_function 2 might be significantly smaller because it is not the primary graph in the model. Optimization should make the model faster, but it won't necessarily make it smaller (the majority of the model size is from the weights, and they won't change). tf2onnx is primarily a model converter, not optimizer.

@TomWildenhain-Microsoft Hi, you comment make sense, thanks ! The problem is that I have added on purpose a tf.Variable in my inference model in order to make it stateful (I'm using LSTMs).
Do you see any way of still converting the inference graph to frozen graph / ONNX / tensorRT to benefit from their nice runtime optimizations ?

[TomWildenhain-Microsoft]

All onnx models are stateless, so you will always get the same results given the same model and input (aside from random ops). To simulate state, make the model have an additional input and output for the state you would like to maintain. Then pass that state into the model with every inference. If you don't do this, the model will just always use the initial state, which should be fine for your LSTM if you have the entire sequence up front, but will cause issues if you are doing a "streaming" sort of application.

The latest tf2onnx will attempt to recover from freezing failures by stripping variable assignments from the graph. I'd recommend first converting the model with the latest tf2onnx and make sure it produces consistent results with tf on the first inference. Subsequent inferences might not match since the keras LSTM is stateful. Inspect the model to find where you need to add inputs and outputs (probably the while loop) and edit it accordingly. Then pass in the state with inference and everything should work.