Converting SSD-MobilenetV2 to ONNX: no change after optimization

[anoushsepehri]

Describe the bug
I am trying to convert a SSD-mobilenetv2 model from TensorFlow's object detection API to ONNX for use with TensorRT, however, I have been running into several problems. An onnx file gets saved however in the terminal the optimization seems to fail.

I am trying to follow the nvidia blog on optimizing tensorflow models with tensorRT. I included a link of it below. When trying to use the ONNX files even though there is no optimization, I get an error saying at least one output node is required. ( i have created a different post for that error).

https://devblogs.nvidia.com/speeding-up-deep-learning-inference-using-tensorflow-onnx-and-tensorrt/https://devblogs.nvidia.com/speeding-up-deep-learning-inference-using-tensorflow-onnx-and-tensorrt/

Urgency
This is very urgent for hard project deadlines

2020-06-12 13:14:38.976453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:14:43.232209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-06-12 13:14:43.249407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2020-06-12 13:15:02,703 - INFO - tensorflow: Saver not created because there are no variables in the graph to restore
2020-06-12 13:15:07.104291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-06-12 13:15:07.131981: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.132348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 23.84GiB/s
2020-06-12 13:15:07.132533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:15:07.132868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-06-12 13:15:07.151975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-06-12 13:15:07.183043: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-06-12 13:15:07.209363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-06-12 13:15:07.225522: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-06-12 13:15:07.225758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-12 13:15:07.226234: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.226618: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.226782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-06-12 13:15:07.251852: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-12 13:15:07.252395: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2a8799b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-12 13:15:07.252446: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-06-12 13:15:07.383170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.383460: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2db61aa0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-12 13:15:07.383514: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-12 13:15:07.383998: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.384120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 23.84GiB/s
2020-06-12 13:15:07.384197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:15:07.384247: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-06-12 13:15:07.384312: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-06-12 13:15:07.384359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-06-12 13:15:07.384398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-06-12 13:15:07.384441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-06-12 13:15:07.384477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-12 13:15:07.384640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.384838: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:07.384912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-06-12 13:15:07.384998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:15:17.317440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-12 13:15:17.317822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-06-12 13:15:17.317967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-06-12 13:15:17.321422: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:17.322350: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:17.325828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 91 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-12 13:15:36.220556: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:36.220775: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-06-12 13:15:36.222241: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-06-12 13:15:36.232537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:36.232755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 23.84GiB/s
2020-06-12 13:15:36.232945: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:15:36.233045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-06-12 13:15:36.233161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-06-12 13:15:36.233227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-06-12 13:15:36.233283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-06-12 13:15:36.233336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-06-12 13:15:36.233384: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-12 13:15:36.233592: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:36.233829: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:36.233926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-06-12 13:15:36.234104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-12 13:15:36.234190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-06-12 13:15:36.234218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-06-12 13:15:36.234482: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:36.234824: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:15:36.234983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 91 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-12 13:15:38.875298: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:841] Optimization results for grappler item: graph_to_optimize
2020-06-12 13:15:38.875400: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:843] function_optimizer: function_optimizer did nothing. time = 0.012ms.
2020-06-12 13:15:38.875427: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:843] function_optimizer: function_optimizer did nothing. time = 0.002ms.
2020-06-12 13:16:08.219007: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:08.219348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 23.84GiB/s
2020-06-12 13:16:08.219720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:16:08.219799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-06-12 13:16:08.219888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-06-12 13:16:08.219945: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-06-12 13:16:08.219991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-06-12 13:16:08.220030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-06-12 13:16:08.220067: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-12 13:16:08.220279: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:08.220523: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:08.220646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-06-12 13:16:08.220731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-12 13:16:08.220764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-06-12 13:16:08.220794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-06-12 13:16:08.221018: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:08.221271: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:08.221607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 91 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
WARNING:tensorflow:From /home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py:324: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2020-06-12 13:16:15,038 - WARNING - tensorflow: From /home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py:324: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2020-06-12 13:16:19.547058: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:19.547224: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-06-12 13:16:19.547889: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-06-12 13:16:19.549435: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:19.549579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 23.84GiB/s
2020-06-12 13:16:19.549676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:16:19.549733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-06-12 13:16:19.549796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-06-12 13:16:19.549838: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-06-12 13:16:19.549878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-06-12 13:16:19.549919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-06-12 13:16:19.549954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-12 13:16:19.550114: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:19.550372: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:19.550446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-06-12 13:16:19.550515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-12 13:16:19.550548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-06-12 13:16:19.550574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-06-12 13:16:19.550750: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:19.550962: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:16:19.551071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 91 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-12 13:16:26.169866: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:841] Optimization results for grappler item: graph_to_optimize
2020-06-12 13:16:26.169994: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:843] constant_folding: Graph size after: 1145 nodes (-1401), 1266 edges (-1572), time = 1506.3ms.
2020-06-12 13:16:26.170033: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:843] function_optimizer: function_optimizer did nothing. time = 9.512ms.
2020-06-12 13:16:26.170068: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:843] constant_folding: Graph size after: 1145 nodes (0), 1266 edges (0), time = 166.712ms.
2020-06-12 13:16:26.170098: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:843] function_optimizer: function_optimizer did nothing. time = 7.143ms.
2020-06-12 13:17:40,106 - INFO - tf2onnx: inputs: ['image_tensor:0']
2020-06-12 13:17:40,107 - INFO - tf2onnx: outputs: ['detection_boxes:0', 'detection_classes:0', 'detection_multiclass_scores:0', 'detection_scores:0', 'num_detections:0', 'raw_detection_boxes:0', 'raw_detection_scores:0']
2020-06-12 13:17:50.730974: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:17:50.775618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 23.84GiB/s
2020-06-12 13:17:50.776233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-06-12 13:17:50.805733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-06-12 13:17:50.813606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-06-12 13:17:50.813805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-06-12 13:17:50.813887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-06-12 13:17:50.813957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-06-12 13:17:50.838794: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-12 13:17:50.839162: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:17:50.839483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:17:50.839756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-06-12 13:17:50.841203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-12 13:17:50.841336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-06-12 13:17:50.841537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-06-12 13:17:50.841959: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:17:50.842432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-06-12 13:17:50.842651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 91 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-12 13:17:50,858 - INFO - tf2onnx.tfonnx: Using tensorflow=2.1.0, onnx=1.7.0, tf2onnx=1.7.0/b73231
2020-06-12 13:17:50,859 - INFO - tf2onnx.tfonnx: Using opset <onnx, 11>
2020-06-12 13:22:56,748 - VERBOSE - tf2onnx.tfonnx: Mapping TF node to ONNX node(s)
2020-06-12 13:23:08,815 - VERBOSE - tf2onnx.tfonnx: Summay Stats:
tensorflow ops: Counter({'Const': 597, 'Identity': 109, 'FusedBatchNorm': 64, 'Conv2D': 55, 'Relu6': 47, 'GatherV2': 26, 'Sub': 21, 'DepthwiseConv2dNative': 21, 'Reshape': 20, 'StridedSlice': 18, 'Add': 15, 'Mul': 14, 'Shape': 14, 'ExpandDims': 12, 'BiasAdd': 10, 'Select': 10, 'ConcatV2': 9, 'Pack': 9, 'Greater': 8, 'Slice': 7, 'Unpack': 6, 'Squeeze': 5, 'Minimum': 5, 'Split': 4, 'Maximum': 4, 'Cast': 4, 'Pad': 4, 'Assert': 3, 'Fill': 3, 'Transpose': 2, 'Exp': 2, 'NonMaxSuppressionV3': 2, 'Less': 2, 'Equal': 2, 'TopKV2': 2, 'Range': 2, 'Placeholder': 1, 'ResizeBilinear': 1, 'Sigmoid': 1, 'Where': 1, 'Size': 1, 'GreaterEqual': 1, 'Sum': 1})
tensorflow attr: Counter({'dtype': 598, 'value': 597, 'T': 515, 'data_format': 150, 'dilations': 76, 'strides': 76, 'padding': 76, 'epsilon': 64, 'is_training': 64, 'explicit_paddings': 55, 'use_cudnn_on_gpu': 55, 'Taxis': 26, 'batch_dims': 26, 'Tindices': 26, 'Tparams': 26, 'Index': 25, 'Tshape': 20, 'N': 18, 'shrink_axis_mask': 18, 'ellipsis_mask': 18, 'begin_mask': 18, 'new_axis_mask': 18, 'end_mask': 18, 'axis': 15, 'out_type': 15, 'Tdim': 12, 'Tidx': 12, 'num': 6, 'squeeze_dims': 5, 'num_split': 4, 'SrcT': 4, 'Truncate': 4, 'DstT': 4, 'Tpaddings': 4, 'summarize': 3, 'index_type': 3, 'Tperm': 2, 'T_threshold': 2, 'incompatible_shape_error': 2, 'sorted': 2, 'shape': 1, 'align_corners': 1, 'half_pixel_centers': 1, 'keep_dims': 1})
onnx mapped: Counter({'Const': 556, 'Identity': 109, 'FusedBatchNorm': 64, 'Conv2D': 55, 'Relu6': 47, 'GatherV2': 26, 'Sub': 21, 'DepthwiseConv2dNative': 21, 'Reshape': 20, 'StridedSlice': 18, 'Add': 15, 'Mul': 14, 'Shape': 14, 'ExpandDims': 12, 'Select': 10, 'ConcatV2': 9, 'Pack': 9, 'Greater': 8, 'Slice': 7, 'Unpack': 6, 'Squeeze': 5, 'Minimum': 5, 'Split': 4, 'Maximum': 4, 'Cast': 4, 'Pad': 4, 'Fill': 3, 'Transpose': 2, 'Exp': 2, 'NonMaxSuppressionV3': 2, 'Less': 2, 'TopKV2': 2, 'Range': 2, 'Placeholder': 1, 'ResizeBilinear': 1, 'Sigmoid': 1, 'Where': 1, 'GreaterEqual': 1, 'Sum': 1})
onnx unmapped: Counter()
2020-06-12 13:23:08,818 - INFO - tf2onnx.optimizer: Optimizing ONNX model
2020-06-12 13:23:08,843 - VERBOSE - tf2onnx.optimizer: Apply optimize_transpose
2020-06-12 13:23:09,907 - WARNING - tf2onnx.optimizer: Failed to apply optimize_transpose
Traceback (most recent call last):
File "/home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-12 13:23:09,942 - VERBOSE - tf2onnx.optimizer: Apply fold_constants
2020-06-12 13:23:11,047 - WARNING - tf2onnx.optimizer: Failed to apply fold_constants
Traceback (most recent call last):
File "/home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-12 13:23:11,072 - VERBOSE - tf2onnx.optimizer: Apply loop_optimizer
2020-06-12 13:23:11,671 - WARNING - tf2onnx.optimizer: Failed to apply loop_optimizer
Traceback (most recent call last):
File "/home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-12 13:23:11,696 - VERBOSE - tf2onnx.optimizer: Apply merge_duplication
2020-06-12 13:23:12,768 - WARNING - tf2onnx.optimizer: Failed to apply merge_duplication
Traceback (most recent call last):
File "/home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-12 13:23:12,793 - VERBOSE - tf2onnx.optimizer: Apply remove_identity
2020-06-12 13:23:13,397 - WARNING - tf2onnx.optimizer: Failed to apply remove_identity
Traceback (most recent call last):
File "/home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-12 13:23:13,423 - VERBOSE - tf2onnx.optimizer: Apply remove_back_to_back
2020-06-12 13:23:14,465 - WARNING - tf2onnx.optimizer: Failed to apply remove_back_to_back
Traceback (most recent call last):
File "/home/anoushs/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-12 13:23:14,693 - INFO - tf2onnx.optimizer: After optimization: no change
2020-06-12 13:23:16,197 - INFO - tf2onnx:
2020-06-12 13:23:16,197 - INFO - tf2onnx: Successfully converted TensorFlow model saved_model2 to ONNX
2020-06-12 13:23:16,600 - INFO - tf2onnx: ONNX model is saved at mobilenetv2_updated11_from_src.onnx

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• Tensorflow Version: 2.1.0
• Python version: 3.6.9

To Reproduce
python3 -m tf2onnx.convert --saved-model saved_model_dir --opset 11 --verbose --output output_file.onnx

[guschmue]

This is a known issue that we have trouble reproducing ourselfs.
But I think @jignparm was able to reproduce it now and is working on it.

[anoushsepehri]

Thank you for your response @guschmue

Just one question, would the failure to optimize the ONNX file result in failures converting the ONNX to a TensorRT engine? When I try to take the ONNX file with no optimizations, I end up with an error stating at least one output is necessary?

Furthermore, if tf2onnx cant optimizes the network, does that mean I won't be able to get any benefits from running the network using TensorRT?

Thanks

[guschmue]

Without optimizer it should be still runable, just slower. tensorrt will do its own optimizations and might make good for some of it but its hard to say what impact it has.
You could check with netron on the model outputs.

[jignparm]

@anoushsepehri , PR #972 should resolve this issue. Could you try it out and let us if it works for your model?

[jignparm]

Either re-install the repo using pip install git+https://github.com/onnx/tensorflow-onnx, or else you can make the one-line change manually, on top of your installed version (since the fix is tiny).

[anoushsepehri]

@jignparm Thank you for your help, I reinstalled the repo however I get the same results and no optimization is done. I attached the terminal thread below. I am using a saved model format to do the conversion.

python3 -m tf2onnx.convert --saved-model saved_model2 --opset 11 --output mobilenetv2.onnx
2020-06-15 11:25:57.872003: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:From /home/anoush/.local/lib/python3.6/site-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-15 11:26:08,943 - WARNING - From /home/anoush/.local/lib/python3.6/site-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-15 11:26:08.948906: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-15 11:26:08.954460: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:08.954618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 11:26:08.954694: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 11:26:08.958677: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 11:26:08.978049: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 11:26:09.018963: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 11:26:09.038269: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 11:26:09.056200: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 11:26:09.056723: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 11:26:09.057683: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:09.057995: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:09.058106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 11:26:09.084289: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-15 11:26:09.084917: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x42e54c20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-15 11:26:09.084976: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-06-15 11:26:09.173003: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:09.173304: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x42fc18e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-15 11:26:09.173358: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-15 11:26:09.173737: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:09.173842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 11:26:09.173916: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 11:26:09.173992: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 11:26:09.174045: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 11:26:09.174091: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 11:26:09.174139: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 11:26:09.174184: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 11:26:09.174229: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 11:26:09.174400: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:09.174597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:09.174668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 11:26:09.174754: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 11:26:13.529891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-15 11:26:13.530018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-15 11:26:13.530066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-15 11:26:13.530659: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:13.531037: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:13.531306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 154 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-15 11:26:45.479443: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:45.479706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 11:26:45.500313: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 11:26:45.522232: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 11:26:45.522562: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 11:26:45.522740: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 11:26:45.522801: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 11:26:45.522850: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 11:26:45.522901: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 11:26:45.523110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:45.523341: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:45.523424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 11:26:45.523786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-15 11:26:45.523838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-15 11:26:45.523874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-15 11:26:45.524182: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:45.524580: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 11:26:45.524916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 154 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-15 11:26:45,525 - INFO - Using tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.7.0/b73231
2020-06-15 11:26:45,525 - INFO - Using opset <onnx, 11>
2020-06-15 11:27:02,517 - INFO - folding node type=Pack, name=Preprocessor/stack_1
2020-06-15 11:27:02,548 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast
2020-06-15 11:27:02,577 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_1
2020-06-15 11:27:02,606 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_2
2020-06-15 11:27:02,635 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_3
2020-06-15 11:27:02,663 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_4
2020-06-15 11:27:02,691 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_5
2020-06-15 11:27:02,717 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_6
2020-06-15 11:27:02,745 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_7
2020-06-15 11:27:02,773 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_8
2020-06-15 11:27:02,801 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_9
2020-06-15 11:27:02,829 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_10
2020-06-15 11:27:02,857 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_11
2020-06-15 11:27:02,885 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_12
2020-06-15 11:27:02,913 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_13
2020-06-15 11:27:02,942 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt
2020-06-15 11:27:02,970 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_15
2020-06-15 11:27:02,998 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range
2020-06-15 11:27:03,026 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_14
2020-06-15 11:27:03,054 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_1
2020-06-15 11:27:03,082 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_15
2020-06-15 11:27:03,112 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_1
2020-06-15 11:27:03,140 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_23
2020-06-15 11:27:03,168 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_2
2020-06-15 11:27:03,195 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_16
2020-06-15 11:27:03,224 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_3
2020-06-15 11:27:03,251 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_17
2020-06-15 11:27:03,279 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_2
2020-06-15 11:27:03,306 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_31
2020-06-15 11:27:03,332 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_4
2020-06-15 11:27:03,359 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_18
2020-06-15 11:27:03,386 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_5
2020-06-15 11:27:03,412 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_19
2020-06-15 11:27:03,440 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_3
2020-06-15 11:27:03,466 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_39
2020-06-15 11:27:03,493 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_6
2020-06-15 11:27:03,520 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_20
2020-06-15 11:27:03,547 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_7
2020-06-15 11:27:03,573 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_21
2020-06-15 11:27:03,601 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_4
2020-06-15 11:27:03,628 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_47
2020-06-15 11:27:03,655 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_8
2020-06-15 11:27:03,681 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_22
2020-06-15 11:27:03,708 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_9
2020-06-15 11:27:03,735 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_23
2020-06-15 11:27:03,763 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_5
2020-06-15 11:27:03,790 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_55
2020-06-15 11:27:03,816 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_10
2020-06-15 11:27:03,843 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_24
2020-06-15 11:27:03,870 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_11
2020-06-15 11:27:03,896 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_25
2020-06-15 11:27:03,927 - INFO - folding node type=Cast, name=Postprocessor/Cast_1
2020-06-15 11:27:03,953 - INFO - folding node type=Cast, name=Postprocessor/Cast_2
2020-06-15 11:27:03,980 - INFO - folding node type=Cast, name=Postprocessor/Cast_3
2020-06-15 11:27:04,006 - INFO - folding node type=Mul, name=Postprocessor/BatchMultiClassNonMaxSuppression/mul
2020-06-15 11:27:04,034 - INFO - folding node type=Add, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_1
2020-06-15 11:27:04,061 - INFO - folding node type=Add, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_3
2020-06-15 11:27:04,088 - INFO - folding node type=ConcatV2, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Concatenate/concat_5
2020-06-15 11:27:53,048 - INFO - Optimizing ONNX model
2020-06-15 11:27:54,524 - WARNING - Failed to apply optimize_transpose
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-15 11:27:56,900 - WARNING - Failed to apply fold_constants
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-15 11:27:59,116 - WARNING - Failed to apply loop_optimizer
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-15 11:28:00,438 - WARNING - Failed to apply merge_duplication
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-15 11:28:02,871 - WARNING - Failed to apply remove_identity
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-15 11:28:04,498 - WARNING - Failed to apply remove_back_to_back
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 50, in optimize_graph
current = copy.deepcopy(graph)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
copier = getattr(x, "deepcopy", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-15 11:28:05,112 - INFO - After optimization: no change
2020-06-15 11:28:08,422 - INFO -
2020-06-15 11:28:08,423 - INFO - Successfully converted TensorFlow model saved_model2 to ONNX
2020-06-15 11:28:09,215 - INFO - ONNX model is saved at mobilenetv2.onnx

[jignparm]

Can you confirm that you are using the corrected Graph.py:set_tensor_value() function? A easy way would be to add a print('sometoken') at line 304, just above the fix, and see that the statement is being printed in a conversion run.

This will rule out the possibility of using the wrong module (due to multiple Python installations or environment variable settings etc.).

[jignparm]

Also, can you provide a link to your saved_model2. Is there a location to download the model directly (the links above refer to Keras models in hd5 format, not TF saved_model format)?

[anoushsepehri]

It seems that the pip install did not update the installation with your changes, so I went in to change it myself. This time I get the following error, however I think it has began optimizing so that is an improvement. I included the error thread below.

As for the saved model, it is an ssd-mobilenetv2 that I trained using tensorflows object detection API on a host computer. I retrained it for different classes and exported the model to deploy on the jetson nano. I can send you the saved model privatly if you would like to test it yourself.

python3 -m tf2onnx.convert --saved-model saved_model2 --opset 11 --output mobilenetv2.onnx
2020-06-15 12:13:35.358670: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.6/runpy.py", line 109, in _get_module_details
import(pkg_name)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/init.py", line 14, in
from tf2onnx import tfonnx, utils, graph, graph_builder, graph_matcher, shape_inference, schemas # pylint: disable=wrong-import-order
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/tfonnx.py", line 22, in
from tf2onnx.graph import Graph
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/graph.py", line 304
print('checkpoint')
^
TabError: inconsistent use of tabs and spaces in indentation
anoush@anoush-desktop:~/Tensorflow_code$ python3 -m tf2onnx.convert --saved-model saved_model2 --opset 11 --output mobilenetv2.onnx
2020-06-15 12:14:12.021503: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:From /home/anoush/.local/lib/python3.6/site-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-15 12:14:19,451 - WARNING - From /home/anoush/.local/lib/python3.6/site-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-15 12:14:19.487066: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-15 12:14:19.493127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.493273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 12:14:19.493342: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:19.523316: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 12:14:19.540454: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 12:14:19.582085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 12:14:19.601861: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 12:14:19.620986: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 12:14:19.621555: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 12:14:19.621832: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.622093: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.622182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 12:14:19.649569: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-15 12:14:19.650256: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13882310 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-15 12:14:19.650324: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-06-15 12:14:19.726864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.727358: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x139ee850 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-15 12:14:19.727431: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-15 12:14:19.727828: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.727957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 12:14:19.728080: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:19.728169: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 12:14:19.728237: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 12:14:19.728300: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 12:14:19.728363: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 12:14:19.728423: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 12:14:19.728487: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 12:14:19.728672: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.728945: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.729047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 12:14:19.729125: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:22.306261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-15 12:14:22.306372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-15 12:14:22.306413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-15 12:14:22.306955: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:22.307264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:22.307479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 130 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-15 12:14:52.676885: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.677041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 12:14:52.677132: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:52.677227: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 12:14:52.677285: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 12:14:52.677329: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 12:14:52.677375: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 12:14:52.677418: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 12:14:52.677468: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 12:14:52.677659: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.677859: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.677933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 12:14:52.678037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-15 12:14:52.678067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-15 12:14:52.678091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-15 12:14:52.678300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.678518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.678637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 130 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-15 12:14:52,679 - INFO - Using tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.7.0/b73231
2020-06-15 12:14:52,679 - INFO - Using opset <onnx, 11>
2020-06-15 12:15:10,350 - INFO - folding node type=Pack, name=Preprocessor/stack_1
2020-06-15 12:15:10,380 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast
2020-06-15 12:15:10,407 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_1
2020-06-15 12:15:10,433 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_2
2020-06-15 12:15:10,458 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_3
2020-06-15 12:15:10,484 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_4
2020-06-15 12:15:10,510 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_5
2020-06-15 12:15:10,536 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_6
2020-06-15 12:15:10,562 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_7
2020-06-15 12:15:10,588 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_8
2020-06-15 12:15:10,614 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_9
2020-06-15 12:15:10,641 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_10
2020-06-15 12:15:10,667 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_11
2020-06-15 12:15:10,694 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_12
2020-06-15 12:15:10,721 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_13
2020-06-15 12:15:10,750 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt
2020-06-15 12:15:10,776 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_15
2020-06-15 12:15:10,803 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range
2020-06-15 12:15:10,830 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_14
2020-06-15 12:15:10,856 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_1
2020-06-15 12:15:10,882 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_15
2020-06-15 12:15:10,910 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_1
2020-06-15 12:15:10,936 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_23
2020-06-15 12:15:10,963 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_2
2020-06-15 12:15:10,990 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_16
2020-06-15 12:15:11,017 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_3
2020-06-15 12:15:11,044 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_17
2020-06-15 12:15:11,074 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_2
2020-06-15 12:15:11,101 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_31
2020-06-15 12:15:11,127 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_4
2020-06-15 12:15:11,154 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_18
2020-06-15 12:15:11,181 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_5
2020-06-15 12:15:11,207 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_19
2020-06-15 12:15:11,235 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_3
2020-06-15 12:15:11,261 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_39
2020-06-15 12:15:11,288 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_6
2020-06-15 12:15:11,315 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_20
2020-06-15 12:15:11,343 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_7
2020-06-15 12:15:11,370 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_21
2020-06-15 12:15:11,398 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_4
2020-06-15 12:15:11,425 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_47
2020-06-15 12:15:11,451 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_8
2020-06-15 12:15:11,478 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_22
2020-06-15 12:15:11,504 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_9
2020-06-15 12:15:11,531 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_23
2020-06-15 12:15:11,559 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_5
2020-06-15 12:15:11,586 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_55
2020-06-15 12:15:11,613 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_10
2020-06-15 12:15:11,641 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_24
2020-06-15 12:15:11,668 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_11
2020-06-15 12:15:11,694 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_25
2020-06-15 12:15:11,725 - INFO - folding node type=Cast, name=Postprocessor/Cast_1
2020-06-15 12:15:11,751 - INFO - folding node type=Cast, name=Postprocessor/Cast_2
2020-06-15 12:15:11,777 - INFO - folding node type=Cast, name=Postprocessor/Cast_3
2020-06-15 12:15:11,804 - INFO - folding node type=Mul, name=Postprocessor/BatchMultiClassNonMaxSuppression/mul
2020-06-15 12:15:11,831 - INFO - folding node type=Add, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_1
2020-06-15 12:15:11,859 - INFO - folding node type=Add, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_3
2020-06-15 12:15:11,886 - INFO - folding node type=ConcatV2, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Concatenate/concat_5
2020-06-15 12:15:58,268 - INFO - Optimizing ONNX model
2020-06-15 12:22:33,970 - WARNING - Failed to apply optimize_transpose
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 52, in optimize_graph
graph = opt.optimize(current) or graph
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 41, in optimize
graph = self._optimize(graph)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 135, in _optimize
return self._apply_optimization(graph, self._optimize_at_current_graph_level)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 62, in _apply_optimization
graph = optimize_func(graph)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 168, in _optimize_at_current_graph_level
self.post_optimize_action()
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 105, in post_optimize_action
if (is_nchw_transpose(op) and (input_shape[3] == 1 or (input_shape[1:3] == [1, 1])))
IndexError: list index out of range

[anoushsepehri]

@jignparm my apologies, so the optimizer did complete and save the model, however one of the optimizers failed as per my message above. The error thread below contains the entire output from the program. It seems that all the operations were handled except for the one error for the optimize_transpose

python3 -m tf2onnx.convert --saved-model saved_model2 --opset 11 --output mobilenetv2.onnx
2020-06-15 12:14:12.021503: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:From /home/anoush/.local/lib/python3.6/site-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-15 12:14:19,451 - WARNING - From /home/anoush/.local/lib/python3.6/site-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-15 12:14:19.487066: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-15 12:14:19.493127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.493273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 12:14:19.493342: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:19.523316: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 12:14:19.540454: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 12:14:19.582085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 12:14:19.601861: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 12:14:19.620986: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 12:14:19.621555: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 12:14:19.621832: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.622093: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.622182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 12:14:19.649569: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-15 12:14:19.650256: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13882310 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-15 12:14:19.650324: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-06-15 12:14:19.726864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.727358: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x139ee850 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-15 12:14:19.727431: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-15 12:14:19.727828: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.727957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 12:14:19.728080: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:19.728169: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 12:14:19.728237: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 12:14:19.728300: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 12:14:19.728363: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 12:14:19.728423: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 12:14:19.728487: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 12:14:19.728672: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.728945: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:19.729047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 12:14:19.729125: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:22.306261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-15 12:14:22.306372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-15 12:14:22.306413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-15 12:14:22.306955: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:22.307264: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:22.307479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 130 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-15 12:14:52.676885: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.677041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-15 12:14:52.677132: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-15 12:14:52.677227: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-15 12:14:52.677285: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-15 12:14:52.677329: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-15 12:14:52.677375: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-15 12:14:52.677418: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-15 12:14:52.677468: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-15 12:14:52.677659: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.677859: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.677933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-15 12:14:52.678037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-15 12:14:52.678067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0
2020-06-15 12:14:52.678091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N
2020-06-15 12:14:52.678300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.678518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-15 12:14:52.678637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 130 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-15 12:14:52,679 - INFO - Using tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.7.0/b73231
2020-06-15 12:14:52,679 - INFO - Using opset <onnx, 11>
2020-06-15 12:15:10,350 - INFO - folding node type=Pack, name=Preprocessor/stack_1
2020-06-15 12:15:10,380 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast
2020-06-15 12:15:10,407 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_1
2020-06-15 12:15:10,433 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_2
2020-06-15 12:15:10,458 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_3
2020-06-15 12:15:10,484 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_4
2020-06-15 12:15:10,510 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_5
2020-06-15 12:15:10,536 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_6
2020-06-15 12:15:10,562 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_7
2020-06-15 12:15:10,588 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_8
2020-06-15 12:15:10,614 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_9
2020-06-15 12:15:10,641 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_10
2020-06-15 12:15:10,667 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_11
2020-06-15 12:15:10,694 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_12
2020-06-15 12:15:10,721 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_13
2020-06-15 12:15:10,750 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt
2020-06-15 12:15:10,776 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_15
2020-06-15 12:15:10,803 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range
2020-06-15 12:15:10,830 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_14
2020-06-15 12:15:10,856 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_1
2020-06-15 12:15:10,882 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_15
2020-06-15 12:15:10,910 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_1
2020-06-15 12:15:10,936 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_23
2020-06-15 12:15:10,963 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_2
2020-06-15 12:15:10,990 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_16
2020-06-15 12:15:11,017 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_3
2020-06-15 12:15:11,044 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_17
2020-06-15 12:15:11,074 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_2
2020-06-15 12:15:11,101 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_31
2020-06-15 12:15:11,127 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_4
2020-06-15 12:15:11,154 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_18
2020-06-15 12:15:11,181 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_5
2020-06-15 12:15:11,207 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_19
2020-06-15 12:15:11,235 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_3
2020-06-15 12:15:11,261 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_39
2020-06-15 12:15:11,288 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_6
2020-06-15 12:15:11,315 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_20
2020-06-15 12:15:11,343 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_7
2020-06-15 12:15:11,370 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_21
2020-06-15 12:15:11,398 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_4
2020-06-15 12:15:11,425 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_47
2020-06-15 12:15:11,451 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_8
2020-06-15 12:15:11,478 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_22
2020-06-15 12:15:11,504 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_9
2020-06-15 12:15:11,531 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_23
2020-06-15 12:15:11,559 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_5
2020-06-15 12:15:11,586 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_55
2020-06-15 12:15:11,613 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_10
2020-06-15 12:15:11,641 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_24
2020-06-15 12:15:11,668 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_11
2020-06-15 12:15:11,694 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/Cast_25
2020-06-15 12:15:11,725 - INFO - folding node type=Cast, name=Postprocessor/Cast_1
2020-06-15 12:15:11,751 - INFO - folding node type=Cast, name=Postprocessor/Cast_2
2020-06-15 12:15:11,777 - INFO - folding node type=Cast, name=Postprocessor/Cast_3
2020-06-15 12:15:11,804 - INFO - folding node type=Mul, name=Postprocessor/BatchMultiClassNonMaxSuppression/mul
2020-06-15 12:15:11,831 - INFO - folding node type=Add, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_1
2020-06-15 12:15:11,859 - INFO - folding node type=Add, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_3
2020-06-15 12:15:11,886 - INFO - folding node type=ConcatV2, name=Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Concatenate/concat_5
2020-06-15 12:15:58,268 - INFO - Optimizing ONNX model
2020-06-15 12:22:33,970 - WARNING - Failed to apply optimize_transpose
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 52, in optimize_graph
graph = opt.optimize(current) or graph
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 41, in optimize
graph = self._optimize(graph)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 135, in _optimize
return self._apply_optimization(graph, self._optimize_at_current_graph_level)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 62, in _apply_optimization
graph = optimize_func(graph)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 168, in _optimize_at_current_graph_level
self.post_optimize_action()
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 105, in post_optimize_action
if (is_nchw_transpose(op) and (input_shape[3] == 1 or (input_shape[1:3] == [1, 1])))
IndexError: list index out of range
2020-06-15 12:23:18,525 - WARNING - Failed to apply optimize_transpose
Traceback (most recent call last):
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 52, in optimize_graph
graph = opt.optimize(current) or graph
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 41, in optimize
graph = self._optimize(graph)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 135, in _optimize
return self._apply_optimization(graph, self._optimize_at_current_graph_level)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 62, in _apply_optimization
graph = optimize_func(graph)
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 168, in _optimize_at_current_graph_level
self.post_optimize_action()
File "/home/anoush/.local/lib/python3.6/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 105, in post_optimize_action
if (is_nchw_transpose(op) and (input_shape[3] == 1 or (input_shape[1:3] == [1, 1])))
IndexError: list index out of range
2020-06-15 12:23:31,817 - INFO - After optimization: Add -140 (197->57), BatchNormalization -57 (64->7), Cast -256 (335->79), Concat -19 (91->72), Const -1152 (1420->268), Div -5 (34->29), Equal -70 (81->11), Expand -41 (49->8), Identity -120 (120->0), Min -1 (8->7), Mul -83 (146->63), Range -1 (4->3), Reshape -56 (135->79), Shape -1 (15->14), Slice -42 (114->72), Squeeze -10 (58->48), Transpose -289 (311->22), Unsqueeze -55 (118->63)
2020-06-15 12:23:32,574 - INFO -
2020-06-15 12:23:32,575 - INFO - Successfully converted TensorFlow model saved_model2 to ONNX
2020-06-15 12:23:32,882 - INFO - ONNX model is saved at mobilenetv2.onnx

[jignparm]

so the optimizer did complete and save the model,

Great to know that the deepcopy error is no longer present.

It seems that all the operations were handled except for the one error for the optimize_transpose

Ok, this seems like a slightly different error.

I can send you the saved model privatly if you would like to test it yourself.

Sure. Feel free to email me details of your model location at my user ID @microsoft.com

[anoushsepehri]

@jignparm I sent you an email with the saved model

Thanks

[jignparm]

@anoushsepehri , thanks for sending model and for opening a separate issue #973 to track the orthogonal problem.

Since the deepcopy() error seems to be addressed, I'll close out this issue. Please reopen if needed.