Onnx conversion fails when ConvLSTM2D layers are used #1678
Description
OS: Windows 10
Python: 3.8
tfonnx: 1.9.1
tensorflow-gpu: 2.2
Tensorflow Model Project : [(https://github.com/keras-team/keras-io/blob/master/examples/vision/conv_lstm.py]((https://github.com/keras-team/keras-io/blob/master/examples/vision/conv_lstm.py)
I'm attempting to convert a tensorflow a SavedModel to ONNX. The model is taken from keras-io examples above.
Onnx conversion with tf2onnx fails. ONNX file is created, but it can't be processed by TensorRT (7.1.3, 7.2, 8).
I've tried other custom models too, that have ConvLSTM2D layers - and it fails in a similar manner. Once the ConvLSTM2D layers are removed, tf2onnx is able to convert the model w/o asserting, and TensorRT is able to process the Onnx.
This issue is similar to https://github.com/onnx/tensorflow-onnx/issues/1222 - that is currently closed, so I'm opening a new issue.
When I run Onnx conversion (repeatedly) with this model it will always fail in one of 2 distinct manners:
- ValueError: make_sure failure: Cannot find node with output 'TensorArrayV2Write/TensorListSetItem:0' in graph 'model_1_conv_lst_m2d_5_while_body_1626_frozen'
- ValueError("graph output " + o + " not exist")
ValueError: graph output Identity_4:0 not exist
The full conversion logs for both failure types:
First failure
Model: "model_1"
Layer (type) Output Shape Param #
input_2 (InputLayer) [(None, None, 64, 64, 1)] 0
conv_lst_m2d_3 (ConvLSTM2D) (None, None, 64, 64, 64) 416256
batch_normalization_2 (Batch (None, None, 64, 64, 64) 256
conv_lst_m2d_4 (ConvLSTM2D) (None, None, 64, 64, 64) 295168
batch_normalization_3 (Batch (None, None, 64, 64, 64) 256
conv_lst_m2d_5 (ConvLSTM2D) (None, None, 64, 64, 64) 33024
conv3d_1 (Conv3D) (None, None, 64, 64, 1) 1729
Total params: 746,689
Trainable params: 746,433
Non-trainable params: 256
2021-08-11 07:35:16.697614: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2021-08-11 07:35:16.698201: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-08-11 07:35:16.699894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2000M computeCapability: 5.0
coreClock: 1.137GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 74.65GiB/s
2021-08-11 07:35:16.701657: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-08-11 07:35:16.702304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-08-11 07:35:16.702836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-08-11 07:35:16.703342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-08-11 07:35:16.703841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-08-11 07:35:16.704531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-08-11 07:35:16.705781: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-08-11 07:35:16.706172: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-08-11 07:35:16.777461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:35:16.778099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2021-08-11 07:35:16.778373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2021-08-11 07:35:16.780866: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1b63919d130 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-08-11 07:35:16.781566: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro M2000M, Compute Capability 5.0
2021-08-11 07:35:17.611097: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: graph_to_optimize
2021-08-11 07:35:17.611496: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 338 nodes (0), 399 edges (0), time = 7.787ms.
2021-08-11 07:35:17.612028: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 338 nodes (0), 399 edges (0), time = 8.739ms.
2021-08-11 07:35:17.612762: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_body_1142
2021-08-11 07:35:17.613323: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:35:17.613923: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:35:17.614442: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_body_1626
2021-08-11 07:35:17.615006: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:35:17.615644: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:35:17.616170: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_cond_1625
2021-08-11 07:35:17.616704: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:35:17.617276: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:35:17.617959: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_cond_1141
2021-08-11 07:35:17.618576: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:35:17.619138: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:35:17.619663: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_body_1384
2021-08-11 07:35:17.620319: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:35:17.620888: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:35:17.621369: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_cond_1383
2021-08-11 07:35:17.621896: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:35:17.622275: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:35:21.115799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:35:21.116323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
WARNING:tensorflow:From C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\tf_loader.py:662: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2021-08-11 07:35:22.141997: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2021-08-11 07:35:22.142580: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-08-11 07:35:22.144827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2000M computeCapability: 5.0
coreClock: 1.137GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 74.65GiB/s
2021-08-11 07:35:22.146788: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-08-11 07:35:22.147410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-08-11 07:35:22.147803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-08-11 07:35:22.148133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-08-11 07:35:22.148699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-08-11 07:35:22.149080: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-08-11 07:35:22.150124: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-08-11 07:35:22.150608: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-08-11 07:35:22.151430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:35:22.151776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2021-08-11 07:35:22.151988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2021-08-11 07:35:22.795426: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: graph_to_optimize
2021-08-11 07:35:22.795977: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 84 nodes (-50), 92 edges (-52), time = 53.932ms.
2021-08-11 07:35:22.796561: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 84 nodes (0), 92 edges (0), time = 7.58ms.
2021-08-11 07:35:22.797146: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 84 nodes (0), 92 edges (0), time = 8.453ms.
2021-08-11 07:35:22.797585: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 84 nodes (0), 92 edges (0), time = 7.192ms.
2021-08-11 07:35:22.798103: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_body_1626_frozen
2021-08-11 07:35:22.798780: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (-3), 102 edges (0), time = 2.732ms.
2021-08-11 07:35:22.799679: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.09ms.
2021-08-11 07:35:22.800429: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (0), 102 edges (0), time = 1.257ms.
2021-08-11 07:35:22.800837: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.053ms.
2021-08-11 07:35:22.801294: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_body_1142_frozen
2021-08-11 07:35:22.801833: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (-3), 102 edges (0), time = 2.998ms.
2021-08-11 07:35:22.802454: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.088ms.
2021-08-11 07:35:22.802842: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (0), 102 edges (0), time = 1.326ms.
2021-08-11 07:35:22.803262: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.039ms.
2021-08-11 07:35:22.803727: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_body_1384_frozen
2021-08-11 07:35:22.804155: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (-3), 102 edges (0), time = 2.829ms.
2021-08-11 07:35:22.804660: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.049ms.
2021-08-11 07:35:22.805040: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (0), 102 edges (0), time = 1.559ms.
2021-08-11 07:35:22.805464: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.041ms.
2021-08-11 07:35:22.805853: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_cond_1141_frozen
2021-08-11 07:35:22.806336: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.387ms.
2021-08-11 07:35:22.806849: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.006ms.
2021-08-11 07:35:22.807194: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.171ms.
2021-08-11 07:35:22.807657: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.005ms.
2021-08-11 07:35:22.808033: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_cond_1383_frozen
2021-08-11 07:35:22.808395: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.336ms.
2021-08-11 07:35:22.808814: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.005ms.
2021-08-11 07:35:22.809211: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.164ms.
2021-08-11 07:35:22.809674: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.009ms.
2021-08-11 07:35:22.809935: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_cond_1625_frozen
2021-08-11 07:35:22.810392: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.446ms.
2021-08-11 07:35:22.810820: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.005ms.
2021-08-11 07:35:22.811128: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.216ms.
2021-08-11 07:35:22.811803: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.005ms.
2021-08-11 07:35:23.601070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:35:23.601725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
2021-08-11 07:35:23.632679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:35:23.633334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
2021-08-11 07:35:23.650469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:35:23.650921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
Failed to apply loop_optimizer
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer_init_.py", line 62, in optimize_graph
graph = opt.optimize(current, iteration) or graph
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer\optimizer_base.py", line 42, in optimize
graph.update_proto()
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 827, in update_proto
node.update_proto(external_tensor_storage)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 383, in update_proto
graph_proto = sub_graph.make_graph("graph for " + self.name + " " + attr_name,
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1067, in make_graph
self.delete_unused_nodes(self.outputs)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1549, in delete_unused_nodes
self.reset_nodes(related_nodes)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 747, in reset_nodes
raise ValueError("graph output " + o + " not exist")
ValueError: graph output Identity_4:0 not exist
Failed to apply loop_optimizer
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer_init_.py", line 62, in optimize_graph
graph = opt.optimize(current, iteration) or graph
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer\optimizer_base.py", line 42, in optimize
graph.update_proto()
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 827, in update_proto
node.update_proto(external_tensor_storage)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 383, in update_proto
graph_proto = sub_graph.make_graph("graph for " + self.name + " " + attr_name,
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1068, in make_graph
self.topological_sort(self.get_nodes())
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1028, in topological_sort
utils.make_sure(j is not None, "Cannot find node with output %r in graph %r", inp, self.graph_name)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\utils.py", line 260, in make_sure
raise ValueError("make_sure failure: " + error_msg % args)
ValueError: make_sure failure: Cannot find node with output 'TensorArrayV2Write/TensorListSetItem:0' in graph 'model_1_conv_lst_m2d_5_while_body_1626_frozen'
Second failure
Model: "model_1"
Layer (type) Output Shape Param #
input_2 (InputLayer) [(None, None, 64, 64, 1)] 0
conv_lst_m2d_3 (ConvLSTM2D) (None, None, 64, 64, 64) 416256
batch_normalization_2 (Batch (None, None, 64, 64, 64) 256
conv_lst_m2d_4 (ConvLSTM2D) (None, None, 64, 64, 64) 295168
batch_normalization_3 (Batch (None, None, 64, 64, 64) 256
conv_lst_m2d_5 (ConvLSTM2D) (None, None, 64, 64, 64) 33024
conv3d_1 (Conv3D) (None, None, 64, 64, 1) 1729
Total params: 746,689
Trainable params: 746,433
Non-trainable params: 256
2021-08-11 07:27:03.216299: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2021-08-11 07:27:03.216932: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-08-11 07:27:03.218882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2000M computeCapability: 5.0
coreClock: 1.137GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 74.65GiB/s
2021-08-11 07:27:03.220704: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-08-11 07:27:03.221263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-08-11 07:27:03.221635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-08-11 07:27:03.222008: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-08-11 07:27:03.222335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-08-11 07:27:03.222739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-08-11 07:27:03.224030: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-08-11 07:27:03.224473: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-08-11 07:27:03.296120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:27:03.296599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2021-08-11 07:27:03.296797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2021-08-11 07:27:03.298650: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1da43e90d50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-08-11 07:27:03.299164: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro M2000M, Compute Capability 5.0
2021-08-11 07:27:04.105015: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: graph_to_optimize
2021-08-11 07:27:04.105455: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 338 nodes (0), 399 edges (0), time = 7.435ms.
2021-08-11 07:27:04.105882: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 338 nodes (0), 399 edges (0), time = 6.974ms.
2021-08-11 07:27:04.106317: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_body_1142
2021-08-11 07:27:04.106744: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:27:04.107135: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:27:04.107530: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_body_1626
2021-08-11 07:27:04.107936: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:04.108328: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:04.108739: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_cond_1625
2021-08-11 07:27:04.109179: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:27:04.109590: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:04.109942: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_cond_1141
2021-08-11 07:27:04.110342: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:27:04.110793: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:04.111147: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_body_1384
2021-08-11 07:27:04.111534: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:04.111897: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:04.112271: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_cond_1383
2021-08-11 07:27:04.112704: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2021-08-11 07:27:04.113129: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0ms.
2021-08-11 07:27:07.710219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:27:07.710620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
WARNING:tensorflow:From C:\Users\E012761\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\tf_loader.py:662: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2021-08-11 07:27:08.672009: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2021-08-11 07:27:08.672698: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-08-11 07:27:08.674737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro M2000M computeCapability: 5.0
coreClock: 1.137GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 74.65GiB/s
2021-08-11 07:27:08.676848: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-08-11 07:27:08.677456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-08-11 07:27:08.677874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-08-11 07:27:08.678088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-08-11 07:27:08.678476: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-08-11 07:27:08.678825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-08-11 07:27:08.680115: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
2021-08-11 07:27:08.680591: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-08-11 07:27:08.681554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:27:08.681924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2021-08-11 07:27:08.682224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2021-08-11 07:27:09.341269: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: graph_to_optimize
2021-08-11 07:27:09.341819: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 84 nodes (-50), 92 edges (-52), time = 49.449ms.
2021-08-11 07:27:09.342663: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 84 nodes (0), 92 edges (0), time = 7.55ms.
2021-08-11 07:27:09.343327: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 84 nodes (0), 92 edges (0), time = 7.991ms.
2021-08-11 07:27:09.343889: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: Graph size after: 84 nodes (0), 92 edges (0), time = 7.987ms.
2021-08-11 07:27:09.344595: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_body_1626_frozen
2021-08-11 07:27:09.345207: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (-3), 102 edges (0), time = 2.684ms.
2021-08-11 07:27:09.345829: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.066ms.
2021-08-11 07:27:09.346346: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (0), 102 edges (0), time = 1.272ms.
2021-08-11 07:27:09.346853: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.04ms.
2021-08-11 07:27:09.347355: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_body_1142_frozen
2021-08-11 07:27:09.347839: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (-3), 102 edges (0), time = 2.828ms.
2021-08-11 07:27:09.348351: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.052ms.
2021-08-11 07:27:09.348801: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (0), 102 edges (0), time = 1.271ms.
2021-08-11 07:27:09.349337: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.039ms.
2021-08-11 07:27:09.349769: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_body_1384_frozen
2021-08-11 07:27:09.350234: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (-3), 102 edges (0), time = 2.999ms.
2021-08-11 07:27:09.350703: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.078ms.
2021-08-11 07:27:09.351206: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 90 nodes (0), 102 edges (0), time = 1.491ms.
2021-08-11 07:27:09.351675: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.048ms.
2021-08-11 07:27:09.352162: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_3_while_cond_1141_frozen
2021-08-11 07:27:09.352567: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.474ms.
2021-08-11 07:27:09.353093: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.007ms.
2021-08-11 07:27:09.353550: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.281ms.
2021-08-11 07:27:09.354122: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.006ms.
2021-08-11 07:27:09.354636: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_4_while_cond_1383_frozen
2021-08-11 07:27:09.355137: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.422ms.
2021-08-11 07:27:09.355644: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.006ms.
2021-08-11 07:27:09.356115: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.214ms.
2021-08-11 07:27:09.356802: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.005ms.
2021-08-11 07:27:09.357367: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: model_1_conv_lst_m2d_5_while_cond_1625_frozen
2021-08-11 07:27:09.358046: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.576ms.
2021-08-11 07:27:09.358614: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.008ms.
2021-08-11 07:27:09.359045: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] constant_folding: Graph size after: 14 nodes (0), 4 edges (0), time = 0.294ms.
2021-08-11 07:27:09.359534: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799] function_optimizer: function_optimizer did nothing. time = 0.006ms.
2021-08-11 07:27:10.203873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:27:10.204362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
2021-08-11 07:27:10.244898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:27:10.245432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
2021-08-11 07:27:10.264687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-11 07:27:10.265098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
Failed to apply loop_optimizer
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer_init_.py", line 62, in optimize_graph
graph = opt.optimize(current, iteration) or graph
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer\optimizer_base.py", line 42, in optimize
graph.update_proto()
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 827, in update_proto
node.update_proto(external_tensor_storage)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 383, in update_proto
graph_proto = sub_graph.make_graph("graph for " + self.name + " " + attr_name,
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1067, in make_graph
self.delete_unused_nodes(self.outputs)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1549, in delete_unused_nodes
self.reset_nodes(related_nodes)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 747, in reset_nodes
raise ValueError("graph output " + o + " not exist")
ValueError: graph output Identity_4:0 not exist
Failed to apply loop_optimizer
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer_init_.py", line 62, in optimize_graph
graph = opt.optimize(current, iteration) or graph
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\optimizer\optimizer_base.py", line 42, in optimize
graph.update_proto()
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 827, in update_proto
node.update_proto(external_tensor_storage)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 383, in update_proto
graph_proto = sub_graph.make_graph("graph for " + self.name + " " + attr_name,
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1067, in make_graph
self.delete_unused_nodes(self.outputs)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 1549, in delete_unused_nodes
self.reset_nodes(related_nodes)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\site-packages\tf2onnx\graph.py", line 747, in reset_nodes
raise ValueError("graph output " + o + " not exist")
ValueError: graph output Identity_4:0 not exist