Skip to content

Unable to convert TF model generated from Tensorflow for Poets #1259

@Chindianese

Description

@Chindianese

Describe the bug
A clear and concise description of what the bug is.

Urgency
None

System information

  • Windows x64
  • Tensorflow Version(Docker): 1.7.0
  • Tensorflow Version(PC): 2.4.0
  • Python version(Docker): 2.7.12
  • Python version(Docker): 2.8.7

Additional context
Saved model generated from Tensorflow for Poets / Inception model, through docker. Running convert for saved model and graph def returns
ValueError: make_sure failure: non-const default value for PlaceholderWithDefault is not supported.
Not sure how to update existing graph to const value.

Input
Training
python -m scripts.retrain
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=10
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/output/retrained_graph.pb
--output_labels=tf_files/output/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--saved_model_dir=tf_files/saved_models/1
--image_dir=tf_files/test_photos

Convert
python -m tf2onnx.convert --saved-model [path]\saved_models\1 --output D:/model.onnx

Log
2021-01-07 12:16:24.810137: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:24.810336: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-01-07 12:16:26,984 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2021-01-07 12:16:27.640053: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:27.641384: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-01-07 12:16:27.666394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.695GHz coreCount: 16 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2021-01-07 12:16:27.667932: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:27.669378: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-01-07 12:16:27.670706: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-01-07 12:16:27.672055: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-01-07 12:16:27.673398: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-01-07 12:16:27.674723: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-01-07 12:16:27.676072: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-01-07 12:16:27.677528: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-01-07 12:16:27.677688: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-01-07 12:16:27.678292: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-07 12:16:27.678887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:27.679025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-01-07 12:16:27.679226: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,682 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,683 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,950 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,951 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27.955182: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,972 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,973 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,345 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,346 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,490 - INFO - Signatures found in model: [serving_default].
2021-01-07 12:16:28,490 - WARNING - '--signature_def' not specified, using first signature: serving_default
2021-01-07 12:16:28.578491: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-01-07 12:16:28.578861: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-01-07 12:16:28.583051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.695GHz coreCount: 16 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2021-01-07 12:16:28.584847: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:28.586490: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-01-07 12:16:28.587977: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-01-07 12:16:28.590111: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-01-07 12:16:28.591883: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-01-07 12:16:28.593605: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-01-07 12:16:28.595185: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-01-07 12:16:28.596754: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-01-07 12:16:28.596928: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-01-07 12:16:28.700158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:28.700298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-07 12:16:28.701793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-07 12:16:28.702331: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:28.782523: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:928] Optimization results for grappler item: graph_to_optimize
function_optimizer: function_optimizer did nothing. time = 0.001ms.
function_optimizer: function_optimizer did nothing. time = 0ms.

WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,934 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,936 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,942 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,943 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:29.893283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:29.893453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-01-07 12:16:29.895201: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:From C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\tf_loader.py:416: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2021-01-07 12:16:30,091 - WARNING - From C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\tf_loader.py:416: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2021-01-07 12:16:30.182976: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-01-07 12:16:30.183349: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-01-07 12:16:30.186431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.695GHz coreCount: 16 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2021-01-07 12:16:30.188367: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:30.190052: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-01-07 12:16:30.191547: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-01-07 12:16:30.193002: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-01-07 12:16:30.194486: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-01-07 12:16:30.196219: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-01-07 12:16:30.197825: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-01-07 12:16:30.199318: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-01-07 12:16:30.199491: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-01-07 12:16:30.200295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:30.200942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-07 12:16:30.201543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-07 12:16:30.202066: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:30.407555: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:928] Optimization results for grappler item: graph_to_optimize
constant_folding: Graph size after: 177 nodes (-383), 176 edges (-410), time = 47.866ms.
function_optimizer: function_optimizer did nothing. time = 1.596ms.
constant_folding: Graph size after: 177 nodes (0), 176 edges (0), time = 10.236ms.
function_optimizer: function_optimizer did nothing. time = 0.547ms.

2021-01-07 12:16:34.373394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:34.373613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-01-07 12:16:34.375303: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:34,375 - INFO - Using tensorflow=2.4.0, onnx=1.8.0, tf2onnx=1.7.2/995bd6
2021-01-07 12:16:34,376 - INFO - Using opset <onnx, 8>
2021-01-07 12:16:38,073 - INFO - Computed 0 values for constant folding
2021-01-07 12:16:52,823 - INFO - Optimizing ONNX model
2021-01-07 12:16:53,201 - INFO - After optimization: Add -27 (28->1), Cast -1 (1->0), Const -13 (72->59), Identity -2 (2->0), Mul -13 (13->0), Transpose -57 (58->1)
Traceback (most recent call last):
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\convert.py", line 188, in
main()
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\convert.py", line 171, in main
model_proto = onnx_graph.make_model("converted from {}".format(model_path), external_tensor_storage=tensor_storage)
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1122, in make_model
graph = self.make_graph(graph_doc, graph_name, external_tensor_storage)
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1072, in make_graph
utils.make_sure(op.inputs[0].is_const(),
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\utils.py", line 213, in make_sure
raise ValueError("make_sure failure: " + error_msg % args)
ValueError: make_sure failure: non-const default value for PlaceholderWithDefault is not supported.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions