Unknown output shape for model with FusedBatchNormV3

[l0stpenguin]

I am trying to convert the following freezed tensorflow model to onnx:
deeplens.pb

python3 -m tf2onnx.convert --graphdef deeplens.pb --output deeplens.onnx --inputs image_320:0[320,320,3] --outputs Network/div:0 --opset 11

Although i get plenty of warnings that it cannot infer the shape of FusedBatchNormV3, the model still gets converted. But when i inspect the model in Netron, the output node shows unknown dimensions float32[1,unk__477,unk__478,1]

How can i make the output node gets the correct shape?

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mac OS X 10.14.6
• Tensorflow Version: 1.15.0
• Python version: 3.7

Logs:

WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tf2onnx/verbose_logging.py:72: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-05-06 18:16:06,068 - WARNING - From /usr/local/lib/python3.7/site-packages/tf2onnx/verbose_logging.py:72: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-05-06 18:16:10,156 - INFO - Using tensorflow=1.15.0, onnx=1.6.0, tf2onnx=1.5.6/80edd7
2020-05-06 18:16:10,156 - INFO - Using opset <onnx, 11>
2020-05-06 18:16:10,156 - INFO - Apply shape override:
2020-05-06 18:16:10,156 - INFO -        Set image_320:0 shape to [320, 320, 3]
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn_conv1/FusedBatchNormV3: Network/Network/Depth/bn_conv1/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2a_branch1/FusedBatchNormV3: Network/Network/Depth/bn2a_branch1/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2a_branch2a/FusedBatchNormV3: Network/Network/Depth/bn2a_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2a_branch2b/FusedBatchNormV3: Network/Network/Depth/bn2a_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2a_branch2c/FusedBatchNormV3: Network/Network/Depth/bn2a_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2b_branch2a/FusedBatchNormV3: Network/Network/Depth/bn2b_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2b_branch2b/FusedBatchNormV3: Network/Network/Depth/bn2b_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2b_branch2c/FusedBatchNormV3: Network/Network/Depth/bn2b_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2c_branch2a/FusedBatchNormV3: Network/Network/Depth/bn2c_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2c_branch2b/FusedBatchNormV3: Network/Network/Depth/bn2c_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn2c_branch2c/FusedBatchNormV3: Network/Network/Depth/bn2c_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn3a_branch1/FusedBatchNormV3: Network/Network/Depth/bn3a_branch1/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn3a_branch2a/FusedBatchNormV3: Network/Network/Depth/bn3a_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,696 - WARNING - Cannot infer shape for Network/Network/Depth/bn3a_branch2b/FusedBatchNormV3: Network/Network/Depth/bn3a_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3a_branch2c/FusedBatchNormV3: Network/Network/Depth/bn3a_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3b_branch2a/FusedBatchNormV3: Network/Network/Depth/bn3b_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3b_branch2b/FusedBatchNormV3: Network/Network/Depth/bn3b_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3b_branch2c/FusedBatchNormV3: Network/Network/Depth/bn3b_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3c_branch2a/FusedBatchNormV3: Network/Network/Depth/bn3c_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3c_branch2b/FusedBatchNormV3: Network/Network/Depth/bn3c_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3c_branch2c/FusedBatchNormV3: Network/Network/Depth/bn3c_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3d_branch2a/FusedBatchNormV3: Network/Network/Depth/bn3d_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3d_branch2b/FusedBatchNormV3: Network/Network/Depth/bn3d_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn3d_branch2c/FusedBatchNormV3: Network/Network/Depth/bn3d_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn4a_branch1/FusedBatchNormV3: Network/Network/Depth/bn4a_branch1/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn4a_branch2a/FusedBatchNormV3: Network/Network/Depth/bn4a_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn4a_branch2b/FusedBatchNormV3: Network/Network/Depth/bn4a_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,697 - WARNING - Cannot infer shape for Network/Network/Depth/bn4a_branch2c/FusedBatchNormV3: Network/Network/Depth/bn4a_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,706 - WARNING - Cannot infer shape for Network/Network/Depth/bn4b_branch2a/FusedBatchNormV3: Network/Network/Depth/bn4b_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,706 - WARNING - Cannot infer shape for Network/Network/Depth/bn4b_branch2b/FusedBatchNormV3: Network/Network/Depth/bn4b_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,706 - WARNING - Cannot infer shape for Network/Network/Depth/bn4b_branch2c/FusedBatchNormV3: Network/Network/Depth/bn4b_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,706 - WARNING - Cannot infer shape for Network/Network/Depth/bn4c_branch2a/FusedBatchNormV3: Network/Network/Depth/bn4c_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,706 - WARNING - Cannot infer shape for Network/Network/Depth/bn4c_branch2b/FusedBatchNormV3: Network/Network/Depth/bn4c_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,707 - WARNING - Cannot infer shape for Network/Network/Depth/bn4c_branch2c/FusedBatchNormV3: Network/Network/Depth/bn4c_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,814 - WARNING - Cannot infer shape for Network/Network/Depth/bn4d_branch2a/FusedBatchNormV3: Network/Network/Depth/bn4d_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,814 - WARNING - Cannot infer shape for Network/Network/Depth/bn4d_branch2b/FusedBatchNormV3: Network/Network/Depth/bn4d_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,814 - WARNING - Cannot infer shape for Network/Network/Depth/bn4d_branch2c/FusedBatchNormV3: Network/Network/Depth/bn4d_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,815 - WARNING - Cannot infer shape for Network/Network/Depth/bn4e_branch2a/FusedBatchNormV3: Network/Network/Depth/bn4e_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,815 - WARNING - Cannot infer shape for Network/Network/Depth/bn4e_branch2b/FusedBatchNormV3: Network/Network/Depth/bn4e_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,815 - WARNING - Cannot infer shape for Network/Network/Depth/bn4e_branch2c/FusedBatchNormV3: Network/Network/Depth/bn4e_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,818 - WARNING - Cannot infer shape for Network/Network/Depth/bn4f_branch2a/FusedBatchNormV3: Network/Network/Depth/bn4f_branch2a/FusedBatchNormV3:5
2020-05-06 18:16:10,818 - WARNING - Cannot infer shape for Network/Network/Depth/bn4f_branch2b/FusedBatchNormV3: Network/Network/Depth/bn4f_branch2b/FusedBatchNormV3:5
2020-05-06 18:16:10,818 - WARNING - Cannot infer shape for Network/Network/Depth/bn4f_branch2c/FusedBatchNormV3: Network/Network/Depth/bn4f_branch2c/FusedBatchNormV3:5
2020-05-06 18:16:10,818 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn1/FusedBatchNormV3: Network/Network/Depth/spp_bn1/FusedBatchNormV3:5
2020-05-06 18:16:10,818 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn_a/FusedBatchNormV3: Network/Network/Depth/spp_bn_a/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn_b/FusedBatchNormV3: Network/Network/Depth/spp_bn_b/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn_c/FusedBatchNormV3: Network/Network/Depth/spp_bn_c/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn_d/FusedBatchNormV3: Network/Network/Depth/spp_bn_d/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn2/FusedBatchNormV3: Network/Network/Depth/spp_bn2/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn3_1/FusedBatchNormV3: Network/Network/Depth/spp_bn3_1/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn3_2/FusedBatchNormV3: Network/Network/Depth/spp_bn3_2/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn4_1/FusedBatchNormV3: Network/Network/Depth/spp_bn4_1/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/spp_bn4_2/FusedBatchNormV3: Network/Network/Depth/spp_bn4_2/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/depth_bn1/FusedBatchNormV3: Network/Network/Depth/depth_bn1/FusedBatchNormV3:5
2020-05-06 18:16:10,819 - WARNING - Cannot infer shape for Network/Network/Depth/depth_bn2/FusedBatchNormV3: Network/Network/Depth/depth_bn2/FusedBatchNormV3:5
2020-05-06 18:16:14,353 - INFO - Optimizing ONNX model
2020-05-06 18:16:31,542 - INFO - After optimization: Add -2 (3->1), Cast -1 (10->9), Const -140 (441->301), Identity -2 (2->0), Reshape +2 (0->2), Squeeze -2 (2->0), Transpose -210 (275->65), Unsqueeze -2 (3->1)
2020-05-06 18:16:31,920 - INFO - 
2020-05-06 18:16:31,922 - INFO - Successfully converted TensorFlow model deeplens.pb to ONNX
2020-05-06 18:16:32,081 - INFO - ONNX model is saved at deeplens.onnx

[guschmue]

I assume the original model does have H,W = None and the model carries this through to the output ? When setting the shape on the input I think we keep this internal and do not force tensorflow to re-run the shape inference (we'd need to check on this).
This input shape override is there because in earlier onnx opsets we hit many cases where we needed to know H,W to calculate attributes for onnx ops. Newer opsets (--opset 11) improved this a lot so you might not even need this anymore.
If the output has H,W = None - the model might be still perfectly good, just that H,W are dynamic and not static, should not be a problem.
But if the model doesn't work, we can take a look.

[l0stpenguin]

Yes the H,W is dynamic. I thought passing the shape along with the input would have propagated it. I intend to use it with a fixed size, do you know a way that i can enforce the shape on this model?

[jignparm]

@guschmue, is this the right location for shape inference and shape_override logic ? If true, then it looks like the downstream shapes should be re-inferred if the input shapes are overridden.
https://github.com/onnx/tensorflow-onnx/blob/master/tf2onnx/shape_inference.py#L29

I tested on a small model (see attached reverse.zip) with input shape= [? , 2, ?, 3]. If we convert the model without any shape override, the output shape is computed as [unk__34,2,unk__35,4].

However, if we override the shape, and convert the model with input shape=[1, 2, 3, 4], the output shape is updated with a more specific shape of [1, 2, 3, 4].

So shape-override is working on this small model -- for deeplens.pb, since it's larger, I wonder if TF loses track of shapes in some interior ops and propagates the unknown shapes to the output?

reverse.zip

[guschmue]

We try to do the right thing but obviously something wrong.
Looking at tf_reload_graph() - I think there is a tf.reset_graph() missing.

[jignparm]

I edited the deeplens.pb model, to make the input placeholder shape value fixed at image_320=[320, 320, 3], and ran TF shape inference on the edited model. It looks like TF is losing shape information at the two StridedSlice operators, which is leading to a shape of (1, ?, ?, 1) for 'Network/ResizeBilinear_1' just below them.

It doesn't look like tf2onnx can do much in this case, since TF is not able to propagate the shape all the way down to the outputs.

The warnings for FusedBatchNormV3 are misleading -- removed them in PR #916 to avoid confusion.

###TF Inferred Shapes

StridedSlice ,  Network/strided_slice
        Inputs
         Tensor("Network/Shape:0", shape=(4,), dtype=int32)
         Tensor("Network/strided_slice/stack:0", shape=(1,), dtype=int32)
         Tensor("Network/strided_slice/stack_1:0", shape=(1,), dtype=int32)
         Tensor("Network/strided_slice/stack_2:0", shape=(1,), dtype=int32)
        Outputs
         Tensor("Network/strided_slice:0", shape=(), dtype=int32)
StridedSlice ,  Network/strided_slice_1
        Inputs
         Tensor("Network/Shape:0", shape=(4,), dtype=int32)
         Tensor("Network/strided_slice_1/stack:0", shape=(1,), dtype=int32)
         Tensor("Network/strided_slice_1/stack_1:0", shape=(1,), dtype=int32)
         Tensor("Network/strided_slice_1/stack_2:0", shape=(1,), dtype=int32)
        Outputs
         Tensor("Network/strided_slice_1:0", shape=(), dtype=int32)
Pack ,  Network/ResizeBilinear_1/size
        Inputs
         Tensor("Network/strided_slice:0", shape=(), dtype=int32)
         Tensor("Network/strided_slice_1:0", shape=(), dtype=int32)
        Outputs
         Tensor("Network/ResizeBilinear_1/size:0", shape=(2,), dtype=int32)
ResizeBilinear ,  Network/ResizeBilinear_1
        Inputs
         Tensor("Network/Network/Depth/depth_pre/BiasAdd:0", shape=(1, 320, 320, 1), dtype=float32)
         Tensor("Network/ResizeBilinear_1/size:0", shape=(2,), dtype=int32)
        Outputs
         Tensor("Network/ResizeBilinear_1:0", shape=(1, ?, ?, 1), dtype=float32)

[l0stpenguin]

@jignparm Do you know any way i can edit the graph to make the shape size static?

[jignparm]

Yes, you can always edit the Onnx model.

See a short snippet of loading a model, editing the input shape and re-saving it here: microsoft/onnxruntime#2264

The official Python API is here, although it's a longer read https://github.com/onnx/onnx/blob/master/docs/PythonAPIOverview.md

[l0stpenguin]

ok thanks