Failed optimization when converting SSD Mobilenet V2 Coco

[turowicz]

Describe the bug
I have an SSD Mobilenet V2 COCO from TF model zoo that I have re-exported using TF 1.15.3 on GPU. I am now trying to convert that model to ONNX. I get the same error with the original network that comes from TF model zoo without re-export to latest TF 1.x.

Urgency
Blocking production workflow.

System information

• Nvidia Jetson Nano JP44
• Ubuntu 18.04
• Python 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] on linux
• tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.6.0/4a3c49

To Reproduce
I run the following command

python3 -m tf2onnx.convert --saved-model /workspace/ssd_mobilenet_v2_coco_2018_03_29/export_1/saved_model/ --output /workspace/ssd-im.tf15.onnx --opset 11 --fold_const

Expected behavior
I expect the network to convert successfully and an .onnx be created file without issues.

Screenshots
N/A

Additional context
Full log below

root@15891574462c:/workspace# python3 -m tf2onnx.convert --saved-model /workspace/ssd_mobilenet_v2_coco_2018_03_29/export_1/saved_model/ --output /workspace/ssd-im.tf15.onnx --opset 11 --fold_const
2020-06-04 12:07:47.778799: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-04 12:07:54,009 - WARNING - From /usr/local/lib/python3.6/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-04 12:07:54.014732: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-04 12:07:54.031229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.031367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-04 12:07:54.031428: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-04 12:07:54.035127: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-04 12:07:54.038234: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-04 12:07:54.038998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-04 12:07:54.043749: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-04 12:07:54.047692: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-04 12:07:54.048185: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-04 12:07:54.048403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.048652: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.048768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-04 12:07:54.070342: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-04 12:07:54.071357: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d24ae80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-04 12:07:54.071428: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-06-04 12:07:54.141120: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.141518: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d3b78d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-04 12:07:54.141572: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-04 12:07:54.142154: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.142288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-04 12:07:54.142402: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-04 12:07:54.142482: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-04 12:07:54.142533: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-04 12:07:54.142567: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-04 12:07:54.142615: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-04 12:07:54.142654: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-04 12:07:54.142689: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-04 12:07:54.142935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.143154: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:54.143245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-04 12:07:54.143344: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-04 12:07:56.479318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-04 12:07:56.479395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-06-04 12:07:56.479424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-06-04 12:07:56.479887: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:56.480179: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:07:56.480343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 683 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-04 12:09:00.281640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:09:00.281823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-04 12:09:00.281907: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-04 12:09:00.281979: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-04 12:09:00.282030: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-04 12:09:00.282075: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-04 12:09:00.282117: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-04 12:09:00.282163: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-04 12:09:00.282207: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-04 12:09:00.282375: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:09:00.282580: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:09:00.282667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-04 12:09:00.282730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-04 12:09:00.282761: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-06-04 12:09:00.282784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-06-04 12:09:00.282946: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:09:00.283149: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-04 12:09:00.283265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 683 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-04 12:09:00,283 - INFO - Using tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.6.0/4a3c49
2020-06-04 12:09:00,284 - INFO - Using opset <onnx, 11>
2020-06-04 12:10:18,072 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat
2020-06-04 12:10:18,136 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_1
2020-06-04 12:10:18,202 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_2
2020-06-04 12:10:18,268 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_3
2020-06-04 12:10:18,333 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_4
2020-06-04 12:10:18,398 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_5
2020-06-04 12:10:18,463 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_6
2020-06-04 12:10:18,528 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_7
2020-06-04 12:10:18,593 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_8
2020-06-04 12:10:18,659 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_9
2020-06-04 12:10:18,724 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_10
2020-06-04 12:10:18,788 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_11
2020-06-04 12:10:18,854 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_12
2020-06-04 12:10:18,920 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_13
2020-06-04 12:10:18,986 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt
2020-06-04 12:10:19,051 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_15
2020-06-04 12:10:19,116 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range
2020-06-04 12:10:19,182 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_14
2020-06-04 12:10:19,247 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_1
2020-06-04 12:10:19,312 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_15
2020-06-04 12:10:19,378 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_1
2020-06-04 12:10:19,443 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_23
2020-06-04 12:10:19,509 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_2
2020-06-04 12:10:19,574 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_16
2020-06-04 12:10:19,640 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_3
2020-06-04 12:10:19,705 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_17
2020-06-04 12:10:19,771 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_2
2020-06-04 12:10:19,836 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_31
2020-06-04 12:10:19,902 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_4
2020-06-04 12:10:19,968 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_18
2020-06-04 12:10:20,033 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_5
2020-06-04 12:10:20,099 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_19
2020-06-04 12:10:20,165 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_3
2020-06-04 12:10:20,230 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_39
2020-06-04 12:10:20,295 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_6
2020-06-04 12:10:20,360 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_20
2020-06-04 12:10:20,425 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_7
2020-06-04 12:10:20,490 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_21
2020-06-04 12:10:20,556 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_4
2020-06-04 12:10:20,621 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_47
2020-06-04 12:10:20,687 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_8
2020-06-04 12:10:20,752 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_22
2020-06-04 12:10:20,817 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_9
2020-06-04 12:10:20,881 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_23
2020-06-04 12:10:20,948 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_5
2020-06-04 12:10:21,013 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_55
2020-06-04 12:10:21,079 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_10
2020-06-04 12:10:21,144 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_24
2020-06-04 12:10:21,209 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_11
2020-06-04 12:10:21,274 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_25
2020-06-04 12:10:21,343 - INFO - folding node type=Cast, name=Postprocessor/ToFloat_1
2020-06-04 12:10:21,408 - INFO - folding node type=Cast, name=Postprocessor/ToFloat_2
2020-06-04 12:13:02,580 - INFO - Optimizing ONNX model
2020-06-04 12:13:07,559 - WARNING - Failed to apply optimize_transpose
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-04 12:13:12,292 - WARNING - Failed to apply fold_constants
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-04 12:13:15,257 - WARNING - Failed to apply loop_optimizer
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-04 12:13:21,546 - WARNING - Failed to apply merge_duplication
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-04 12:13:24,506 - WARNING - Failed to apply remove_identity
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-04 12:13:28,998 - WARNING - Failed to apply remove_back_to_back
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-04 12:13:30,180 - INFO - After optimization: no change
2020-06-04 12:13:42,188 - INFO - 
2020-06-04 12:13:42,188 - INFO - Successfully converted TensorFlow model /workspace/ssd_mobilenet_v2_coco_2018_03_29/export_1/saved_model/ to ONNX
2020-06-04 12:13:44,192 - INFO - ONNX model is saved at /workspace/ssd-im.tf15.onnx

[guschmue]

this is some known issue seen by others:
#886
Unfortunately we have a hard time to reproduce this so far which makes it hard to debug.
We'll try your model, see if that fails for us.

[jignparm]

There is a suspected problem in Conv2D, which may be leading to these intermittent deep-copy failures for some users, and other users cannot reproduce them.

@turowicz , would you be able to try PR # 953 to see if that resolves the deepcopy issue for you?

[turowicz]

@jignparm I got this error when trying to convert on Jetson Nano (JP44).

On a desktop PC with a GPU it does work correctly.

[turowicz]

I will try @jignparm

[turowicz]

@jignparm No unfortunately the PR #953 does not help on Nano

root@1ac34358a8cc:/workspace# python3 -m tf2onnx.convert --saved-model /workspace/ssd_mobilenet_v2_coco_2018_03_29/saved_model --output /workspace/ssd.nano.onnx --opset 11 --fold_const
2020-06-05 09:55:13.423976: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-05 09:55:22,462 - WARNING - From /usr/local/lib/python3.6/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-05 09:55:22.467949: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-05 09:55:22.482605: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.482764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-05 09:55:22.482838: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:55:22.512646: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-05 09:55:22.575763: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-05 09:55:22.635672: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-05 09:55:22.794437: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-05 09:55:22.846730: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-05 09:55:22.847836: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-05 09:55:22.848224: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.848712: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.848898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-05 09:55:22.874040: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-05 09:55:22.874829: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41a9c40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-05 09:55:22.874881: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-06-05 09:55:22.967476: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.967861: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4316670 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-05 09:55:22.967910: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-05 09:55:22.968277: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.968433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-05 09:55:22.968504: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:55:22.968573: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-05 09:55:22.968627: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-05 09:55:22.968668: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-05 09:55:22.968718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-05 09:55:22.968756: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-05 09:55:22.968804: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-05 09:55:22.968954: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.969176: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:22.969250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-05 09:55:22.969358: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:55:32.419388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-05 09:55:32.419459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-06-05 09:55:32.419489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-06-05 09:55:32.419860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:32.420139: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:32.420373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 348 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/convert.py", line 169, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/convert.py", line 130, in main
    args.saved_model, args.inputs, args.outputs, args.signature_def)
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/tf_loader.py", line 244, in from_saved_model
    _from_saved_model_v1(sess, model_path, input_names, output_names, signatures)
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/tf_loader.py", line 184, in _from_saved_model_v1
    imported = tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], model_path)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 268, in load
    loader = SavedModelLoader(export_dir)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 284, in __init__
    self._saved_model = parse_saved_model(export_dir)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/loader_impl.py", line 83, in parse_saved_model
    constants.SAVED_MODEL_FILENAME_PB))
OSError: SavedModel file does not exist at: /workspace/ssd_mobilenet_v2_coco_2018_03_29/saved_model/{saved_model.pbtxt|saved_model.pb}
root@1ac34358a8cc:/workspace# python3 -m tf2onnx.convert --saved-model /workspace/ssd_mobilenet_v2_coco_2018_03_29/export_1/saved_model --output /workspace/ssd.nano.onnx --opset 11 --fold_const
2020-06-05 09:55:45.113495: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-05 09:55:51,324 - WARNING - From /usr/local/lib/python3.6/dist-packages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-06-05 09:55:51.330256: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-06-05 09:55:51.344383: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.344521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-05 09:55:51.344579: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:55:51.348301: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-05 09:55:51.351720: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-05 09:55:51.352501: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-05 09:55:51.357325: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-05 09:55:51.361348: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-05 09:55:51.361842: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-05 09:55:51.362062: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.362338: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.362436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-05 09:55:51.383774: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-06-05 09:55:51.384897: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xd1959b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-05 09:55:51.384953: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-06-05 09:55:51.465036: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.465346: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xd301fd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-05 09:55:51.465396: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-06-05 09:55:51.466481: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.467087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-05 09:55:51.467430: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:55:51.467593: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-05 09:55:51.467648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-05 09:55:51.467691: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-05 09:55:51.467745: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-05 09:55:51.467790: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-05 09:55:51.467832: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-05 09:55:51.468098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.468295: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:51.468364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-05 09:55:51.468455: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:55:53.931403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-05 09:55:53.931477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-06-05 09:55:53.931538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-06-05 09:55:53.931886: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:53.932153: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:55:53.932341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 313 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-05 09:57:00.746567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:57:00.746723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-06-05 09:57:00.746796: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-06-05 09:57:00.746871: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-06-05 09:57:00.746926: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-06-05 09:57:00.746972: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-06-05 09:57:00.747017: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-06-05 09:57:00.747059: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-06-05 09:57:00.747092: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-06-05 09:57:00.747277: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:57:00.747500: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:57:00.747587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-06-05 09:57:00.747657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-05 09:57:00.747688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-06-05 09:57:00.747711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-06-05 09:57:00.747896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:57:00.748118: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-06-05 09:57:00.748238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 313 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-06-05 09:57:00,748 - INFO - Using tensorflow=1.15.2, onnx=1.7.0, tf2onnx=1.6.0/ed1c17
2020-06-05 09:57:00,749 - INFO - Using opset <onnx, 11>
2020-06-05 09:58:19,597 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat
2020-06-05 09:58:19,661 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_1
2020-06-05 09:58:19,724 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_2
2020-06-05 09:58:19,787 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_3
2020-06-05 09:58:19,850 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_4
2020-06-05 09:58:19,913 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_5
2020-06-05 09:58:19,976 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_6
2020-06-05 09:58:20,039 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_7
2020-06-05 09:58:20,102 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_8
2020-06-05 09:58:20,166 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_9
2020-06-05 09:58:20,228 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_10
2020-06-05 09:58:20,293 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_11
2020-06-05 09:58:20,357 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_12
2020-06-05 09:58:20,423 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_13
2020-06-05 09:58:20,491 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt
2020-06-05 09:58:20,555 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_15
2020-06-05 09:58:20,619 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range
2020-06-05 09:58:20,683 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_14
2020-06-05 09:58:20,746 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_1
2020-06-05 09:58:20,810 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_15
2020-06-05 09:58:20,875 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_1
2020-06-05 09:58:20,938 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_23
2020-06-05 09:58:21,002 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_2
2020-06-05 09:58:21,066 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_16
2020-06-05 09:58:21,130 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_3
2020-06-05 09:58:21,193 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_17
2020-06-05 09:58:21,257 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_2
2020-06-05 09:58:21,321 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_31
2020-06-05 09:58:21,386 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_4
2020-06-05 09:58:21,450 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_18
2020-06-05 09:58:21,514 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_5
2020-06-05 09:58:21,577 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_19
2020-06-05 09:58:21,642 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_3
2020-06-05 09:58:21,705 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_39
2020-06-05 09:58:21,769 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_6
2020-06-05 09:58:21,833 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_20
2020-06-05 09:58:21,896 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_7
2020-06-05 09:58:21,960 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_21
2020-06-05 09:58:22,025 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_4
2020-06-05 09:58:22,089 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_47
2020-06-05 09:58:22,153 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_8
2020-06-05 09:58:22,217 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_22
2020-06-05 09:58:22,280 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_9
2020-06-05 09:58:22,344 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_23
2020-06-05 09:58:22,409 - INFO - folding node type=Sqrt, name=MultipleGridAnchorGenerator/Sqrt_5
2020-06-05 09:58:22,473 - INFO - folding node type=Mul, name=MultipleGridAnchorGenerator/mul_55
2020-06-05 09:58:22,538 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_10
2020-06-05 09:58:22,602 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_24
2020-06-05 09:58:22,665 - INFO - folding node type=Range, name=MultipleGridAnchorGenerator/range_11
2020-06-05 09:58:22,728 - INFO - folding node type=Cast, name=MultipleGridAnchorGenerator/ToFloat_25
2020-06-05 09:58:22,795 - INFO - folding node type=Cast, name=Postprocessor/ToFloat_1
2020-06-05 09:58:22,859 - INFO - folding node type=Cast, name=Postprocessor/ToFloat_2
2020-06-05 10:01:05,742 - INFO - Optimizing ONNX model
2020-06-05 10:01:10,723 - WARNING - Failed to apply optimize_transpose
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-05 10:01:15,463 - WARNING - Failed to apply fold_constants
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-05 10:01:18,444 - WARNING - Failed to apply loop_optimizer
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-05 10:01:24,742 - WARNING - Failed to apply merge_duplication
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-05 10:01:27,716 - WARNING - Failed to apply remove_identity
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-05 10:01:32,226 - WARNING - Failed to apply remove_back_to_back
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/optimizer/__init__.py", line 50, in optimize_graph
    current = copy.deepcopy(graph)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 159, in deepcopy
    copier = getattr(x, "__deepcopy__", None)
ReferenceError: weakly-referenced object no longer exists
2020-06-05 10:01:33,401 - INFO - After optimization: no change
2020-06-05 10:01:45,336 - INFO - 
2020-06-05 10:01:45,337 - INFO - Successfully converted TensorFlow model /workspace/ssd_mobilenet_v2_coco_2018_03_29/export_1/saved_model to ONNX
2020-06-05 10:01:47,363 - INFO - ONNX model is saved at /workspace/ssd.nano.onnx

[turowicz]

I installed the PR with:

RUN pip3 install git+https://github.com/gerbenvv/tensorflow-onnx.git@feat/make-3d-convolutions-work

[turowicz]

Converting from frozen_graph also fails with the same error on Jetson Nano (JP44).

[turowicz]

Anyway it doesn't matter as I am able to do this conversion on a Desktop computer.

[jignparm]

@turowicz, PR #972 should resolve this issue. Let me know if you see any errors.