-
Notifications
You must be signed in to change notification settings - Fork 455
Description
Describe the bug
A clear and concise description of what the bug is.
Urgency
None
System information
- Windows x64
- Tensorflow Version(Docker): 1.7.0
- Tensorflow Version(PC): 2.4.0
- Python version(Docker): 2.7.12
- Python version(Docker): 2.8.7
Additional context
Saved model generated from Tensorflow for Poets / Inception model, through docker. Running convert for saved model and graph def returns
ValueError: make_sure failure: non-const default value for PlaceholderWithDefault is not supported.
Not sure how to update existing graph to const value.
Input
Training
python -m scripts.retrain
--bottleneck_dir=tf_files/bottlenecks
--how_many_training_steps=10
--model_dir=tf_files/models/
--summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}"
--output_graph=tf_files/output/retrained_graph.pb
--output_labels=tf_files/output/retrained_labels.txt
--architecture="${ARCHITECTURE}"
--saved_model_dir=tf_files/saved_models/1
--image_dir=tf_files/test_photos
Convert
python -m tf2onnx.convert --saved-model [path]\saved_models\1 --output D:/model.onnx
Log
2021-01-07 12:16:24.810137: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:24.810336: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-01-07 12:16:26,984 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2021-01-07 12:16:27.640053: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:27.641384: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-01-07 12:16:27.666394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.695GHz coreCount: 16 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2021-01-07 12:16:27.667932: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:27.669378: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-01-07 12:16:27.670706: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-01-07 12:16:27.672055: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-01-07 12:16:27.673398: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-01-07 12:16:27.674723: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-01-07 12:16:27.676072: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-01-07 12:16:27.677528: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-01-07 12:16:27.677688: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-01-07 12:16:27.678292: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-07 12:16:27.678887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:27.679025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-01-07 12:16:27.679226: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,682 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,683 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,950 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,951 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27.955182: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,972 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:27,973 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,345 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,346 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,490 - INFO - Signatures found in model: [serving_default].
2021-01-07 12:16:28,490 - WARNING - '--signature_def' not specified, using first signature: serving_default
2021-01-07 12:16:28.578491: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-01-07 12:16:28.578861: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-01-07 12:16:28.583051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.695GHz coreCount: 16 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2021-01-07 12:16:28.584847: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:28.586490: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-01-07 12:16:28.587977: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-01-07 12:16:28.590111: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-01-07 12:16:28.591883: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-01-07 12:16:28.593605: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-01-07 12:16:28.595185: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-01-07 12:16:28.596754: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-01-07 12:16:28.596928: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-01-07 12:16:28.700158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:28.700298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-07 12:16:28.701793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-07 12:16:28.702331: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:28.782523: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:928] Optimization results for grappler item: graph_to_optimize
function_optimizer: function_optimizer did nothing. time = 0.001ms.
function_optimizer: function_optimizer did nothing. time = 0ms.
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,934 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,936 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,942 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/weights/final_weights:0' shape=(1001, 2) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:28,943 - WARNING - Unable to create a python object for variable <tf.Variable 'final_training_ops/biases/final_biases:0' shape=(2,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-01-07 12:16:29.893283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:29.893453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-01-07 12:16:29.895201: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:From C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\tf_loader.py:416: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2021-01-07 12:16:30,091 - WARNING - From C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\tf_loader.py:416: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2021-01-07 12:16:30.182976: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-01-07 12:16:30.183349: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-01-07 12:16:30.186431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.695GHz coreCount: 16 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 238.66GiB/s
2021-01-07 12:16:30.188367: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-01-07 12:16:30.190052: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-01-07 12:16:30.191547: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-01-07 12:16:30.193002: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-01-07 12:16:30.194486: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-01-07 12:16:30.196219: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-01-07 12:16:30.197825: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-01-07 12:16:30.199318: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-01-07 12:16:30.199491: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-01-07 12:16:30.200295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:30.200942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-07 12:16:30.201543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-07 12:16:30.202066: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:30.407555: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:928] Optimization results for grappler item: graph_to_optimize
constant_folding: Graph size after: 177 nodes (-383), 176 edges (-410), time = 47.866ms.
function_optimizer: function_optimizer did nothing. time = 1.596ms.
constant_folding: Graph size after: 177 nodes (0), 176 edges (0), time = 10.236ms.
function_optimizer: function_optimizer did nothing. time = 0.547ms.
2021-01-07 12:16:34.373394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-07 12:16:34.373613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-01-07 12:16:34.375303: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-07 12:16:34,375 - INFO - Using tensorflow=2.4.0, onnx=1.8.0, tf2onnx=1.7.2/995bd6
2021-01-07 12:16:34,376 - INFO - Using opset <onnx, 8>
2021-01-07 12:16:38,073 - INFO - Computed 0 values for constant folding
2021-01-07 12:16:52,823 - INFO - Optimizing ONNX model
2021-01-07 12:16:53,201 - INFO - After optimization: Add -27 (28->1), Cast -1 (1->0), Const -13 (72->59), Identity -2 (2->0), Mul -13 (13->0), Transpose -57 (58->1)
Traceback (most recent call last):
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\convert.py", line 188, in
main()
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\convert.py", line 171, in main
model_proto = onnx_graph.make_model("converted from {}".format(model_path), external_tensor_storage=tensor_storage)
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1122, in make_model
graph = self.make_graph(graph_doc, graph_name, external_tensor_storage)
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1072, in make_graph
utils.make_sure(op.inputs[0].is_const(),
File "C:\Users\Cheng\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\utils.py", line 213, in make_sure
raise ValueError("make_sure failure: " + error_msg % args)
ValueError: make_sure failure: non-const default value for PlaceholderWithDefault is not supported.