Merge pull request #1 from onnx/master

Merge base branch changes

Author: buddhapuneeth

README.md

@@ -2,29 +2,24 @@
 
 | Build Type | OS | Python | Tensorflow | Onnx opset | Status |
 | ---        | ---    | ---    | ---        | ---        | ---    |
-| Unit Test - Basic | Linux, MacOS<sup>\*</sup>, Windows<sup>\*</sup> | 3.6, 3.7 | 1.12-1.15, 2.1-2.2 | 7-12 | [![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build/latest?definitionId=16&branchName=master) |
-| Unit Test - Full | Linux, MacOS, Windows | 3.6, 3.7 | 1.12-1.15, 2.1-2.2 | 7-12 | [![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test-matrix?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build/latest?definitionId=18&branchName=master) | |
+| Unit Test - Basic | Linux, MacOS<sup>\*</sup>, Windows<sup>\*</sup> | 3.6, 3.7, 3.8 | 1.12-1.15, 2.1-2.4 | 7-12 | [![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build/latest?definitionId=16&branchName=master) |
+| Unit Test - Full | Linux, MacOS, Windows | 3.6, 3.7, 3.8 | 1.12-1.15, 2.1-2.3 | 7-12 | [![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test-matrix?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build/latest?definitionId=18&branchName=master) | |
 
 ## Supported Versions
 
 ### ONNX
 
 tensorflow-onnx will use the ONNX version installed on your system and installs the latest ONNX version if none is found.
 
-We support ONNX opset-6 to opset-12. By default we use opset-8 for the resulting ONNX graph since most runtimes will support opset-8.
+We support ONNX opset-6 to opset-12. By default we use opset-9 for the resulting ONNX graph since most runtimes will support opset-9.
 Support for future opsets add added as they are released.
 
-If you want the graph to be generated with a specific opset, use ```--opset``` in the command line, for example ```--opset 11```.
+If you want the graph to be generated with a specific opset, use ```--opset``` in the command line, for example ```--opset 12```.
 
 ### TensorFlow
 
 We support all ```tf-1.x graphs```. To keep our test matrix manageable we test tf2onnx running on top of ```tf-1.12 and up```. tf2onnx-1.5.4 was the last version that was tested all the way back to tf-1.4.
 
-There is now ```experimental support for tf-2.x```. 
-With the exception of LSTM unit tests, all unit tests are enabled and passing.
-Unit tests that we still need to fix are marked with ```@skip_tf2```.
-GRU/LSTM's are converting but not runnable due to type/shape inference issues at runtime (working on that one).
-All unit tests are running in eager mode. After execution we take the python function, make it a graph and convert it to ONNX.
 When running under tf-2.x tf2onnx will use the tensorflow V2 controlflow.
 
 You can install tf2onnx on top of tf-1.x or tf-2.x.
@@ -139,11 +134,16 @@ python -m tf2onnx.convert
     [--outputs GRAPH_OUTPUS]
     [--inputs-as-nchw inputs_provided_as_nchw]
     [--opset OPSET]
+    [--tag TAG]
+    [--signature_def SIGNATURE_DEF]
+    [--concrete_function CONCRETE_FUNCTION]
     [--target TARGET]
     [--custom-ops list-of-custom-ops]
     [--fold_const]
+    [--large_model]
     [--continue_on_error]
     [--verbose]
+    [--output_frozen_graph]
 ```
 
 ### Parameters
@@ -176,13 +176,44 @@ By default we preserve the image format of inputs (`nchw` or `nhwc`) as given in
 
 By default we use the opset 8 to generate the graph. By specifying ```--opset``` the user can override the default to generate a graph with the desired opset. For example ```--opset 5``` would create a onnx graph that uses only ops available in opset 5. Because older opsets have in most cases fewer ops, some models might not convert on a older opset.
 
+#### --tag
+
+Only valid with parameter `--saved_model`. Specifies the tag in the saved_model to be used. Typical value is 'serve'.
+
+#### --signature_def
+
+Only valid with parameter `--saved_model`. Specifies which signature to use within the specified --tag value. Typical value is 'serving_default'.
+
+#### --concrete_function
+
+(This is experimental, valid only for TF2.x models)
+
+Only valid with parameter `--saved_model`. If a model contains a list of concrete functions, under the function name `__call__` (as can be viewed using the command `saved_model_cli show --all`), this parameter is a 0-based integer specifying which function in that list should be converted. This parameter takes priority over `--signature_def`, which will be ignored.
+
+#### --large_model
+
+(This is experimental, valid only for TF2.x models)
+
+Only valid with parameter `--saved_model`. When set, creates a zip file containing the ONNX protobuf model and large tensor values stored externally. This allows for converting models that exceed the 2 GB protobuf limit.
+
+#### --output_frozen_graph
+
+Saves the frozen tensorflow graph to file.
+
+#### --custom-ops
+
+If a model contains ops not recognized by onnx runtime, you can tag these ops with a custom op domain so that the
+runtime can still open the model. The format is a comma-separated map of tf op names to domains in the format
+OpName:domain. If only an op name is provided (no colon), the default domain of `ai.onnx.converters.tensorflow`
+will be used.
+
 #### --target
 
 Some models require special handling to run on some runtimes. In particular, the model may use unsupported data types. Workarounds are activated with ```--target TARGET```. Currently supported values are listed on this [wiki](https://github.com/onnx/tensorflow-onnx/wiki/target). If your model will be run on Windows ML, you should specify the appropriate target value.
 
 #### --fold_const
 
-When set, TensorFlow fold_constants transformation is applied before conversion. This benefits features including Transpose optimization (e.g. Transpose operations introduced during tf-graph-to-onnx-graph conversion will be removed), and RNN unit conversion (for example LSTM). Older TensorFlow version might run into issues with this option depending on the model.
+Deprecated. Constant folding is always enabled.
 
 ### <a name="summarize_graph"></a>Tool to get Graph Inputs & Outputs
 
@@ -257,7 +288,8 @@ tf2onnx.tfonnx.process_tf_graph(tf_graph,
             opset=None, custom_op_handlers=None,
             custom_rewriter=None, extra_opset=None,
             shape_override=None, inputs_as_nchw=None,
-            input_names=None, output_names=None):
+            input_names=None, output_names=None,
+            const_node_values=None):
     """Convert tensorflow graph to onnx graph.
         Args:
             tf_graph: tensorflow graph
@@ -272,11 +304,12 @@ tf2onnx.tfonnx.process_tf_graph(tf_graph,
             inputs_as_nchw: transpose inputs in list from nchw to nchw
             input_names: list of input node names in graph, input name format as node_name:port_id
             output_names: list of output node names in graph, output name format as node_name:port_id
+            const_node_values: an optional dict mapping node names to tensor values
         Return:
             onnx graph
     """
 ```
-For example in [examples/call_coverter_via_python.py]():
+For example in [examples/call_converter_via_python.py]():
 ```
 import tensorflow as tf
 import tf2onnx

VERSION_NUMBER

@@ -1 +1 @@
-1.7.0
+1.8.0

ci_build/azure_pipelines/pretrained_model_test.yml

@@ -4,7 +4,7 @@ jobs:
 - template: 'templates/job_generator.yml'
   parameters:
     python_versions: ['3.7']
-    tf_versions: ['1.15.2','2.1.0']
+    tf_versions: ['1.15.2','2.3.0']
     job:
       steps:
       - template: 'pretrained_model_test.yml'
@@ -24,3 +24,4 @@ jobs:
     job:
       steps:
       - template: 'pretrained_model_test.yml'
+

ci_build/azure_pipelines/pylint.yml

@@ -12,3 +12,4 @@ jobs:
           pip freeze
           pylint --rcfile=tools/pylintrc --ignore=version.py --disable=cyclic-import tf2onnx tests/*.py tools -j 0
         displayName: 'Pylint'
+

ci_build/azure_pipelines/templates/job_generator.yml

@@ -5,8 +5,8 @@ parameters:
   python_versions: ['3.7']
   tf_versions: ['']
   onnx_versions: ['']
-  onnx_opsets: ['12', '11', '10', '9', '8', '7']
-  onnx_backends: {onnxruntime: ['1.3.0']}
+  onnx_opsets: ['13', '12', '11', '10', '9', '8', '7']
+  onnx_backends: {onnxruntime: ['1.6.0']}
   job: {}
   run_setup: 'True'
   report_coverage: 'False'

ci_build/azure_pipelines/templates/unit_test.yml

@@ -1,7 +1,7 @@
 # Run unit test
 
 parameters:
-    onnx_opsets: ['12', '11', '10', '9', '8', '7']
+    onnx_opsets: ['13', '12', '11', '10', '9', '8', '7']
 
 steps:
 - ${{ each onnx_opset in parameters.onnx_opsets }}:

ci_build/azure_pipelines/unit_test.yml

@@ -3,16 +3,26 @@
 stages:
   - stage:
     jobs:
+    - template: 'templates/job_generator.yml'
+      parameters:
+        python_versions: ['3.8']
+        tf_versions: ['2.4.0']
+        onnx_opsets: ['']
+        job:
+          steps:
+          - template: 'unit_test.yml'
+        report_coverage: 'True'
+
     - template: 'templates/job_generator.yml'
       parameters:
         python_versions: ['3.7']
-        tf_versions: ['1.14.0','1.15.2','2.1.0','2.2.0']
+        tf_versions: ['1.14.0','1.15.2','2.2.0','2.3.0']
         onnx_opsets: ['']
         job:
           steps:
           - template: 'unit_test.yml'
         report_coverage: 'True'
-    
+
     - template: 'templates/job_generator.yml'
       parameters:
         python_versions: [3.6']
@@ -34,3 +44,4 @@ stages:
         report_coverage: 'True'
 
   - template: 'templates/combine_test_coverage.yml'
+

examples/benchmark_tfmodel_ort.py

@@ -0,0 +1,47 @@
+"""
+The following code compares the speed of tensorflow against onnxruntime
+with a model downloaded from Tensorflow Hub.
+"""
+import time
+import numpy
+from tqdm import tqdm
+import tensorflow_hub as hub
+import onnxruntime as ort
+
+
+def generate_random_images(shape=(100, 100), n=10):
+    imgs = []
+    for i in range(n):
+        sh = (1,) + shape + (3,)
+        img = numpy.clip(numpy.abs(numpy.random.randn(*sh)), 0, 1) * 255
+        img = img.astype(numpy.float32)
+        imgs.append(img)
+    return imgs
+
+
+def measure_time(fct, imgs):
+    results = []
+    times = []
+    for img in tqdm(imgs):
+        begin = time.perf_counter()
+        result = fct(img)
+        end = time.perf_counter()
+        results.append(result)
+        times.append(end - begin)
+    return results, times
+
+
+imgs = generate_random_images()
+
+# Download model from https://tfhub.dev/captain-pool/esrgan-tf2/1
+# python -m tf2onnx.convert --saved-model esrgan --output "esrgan-tf2.onnx" --opset 12
+ort = ort.InferenceSession('esrgan-tf2.onnx')
+fct_ort = lambda img: ort.run(None, {'input_0:0': img})
+results_ort, duration_ort = measure_time(fct_ort, imgs)
+print(len(imgs), duration_ort)
+
+model = hub.load("https://tfhub.dev/captain-pool/esrgan-tf2/1")
+results_tf, duration_tf = measure_time(model, imgs)
+print(len(imgs), duration_tf)
+
+print("ratio ORT / TF", sum(duration_ort) / sum(duration_tf))

examples/call_coverter_via_python.py renamed to examples/call_converter_via_python.py

@@ -0,0 +1,75 @@
+"""
+This example retrieves a model from tensorflowhub.
+It is converted into ONNX. Predictions are compared to
+the predictions from tensorflow to check there is no
+discrepencies. Inferencing time is also compared between
+*onnxruntime*, *tensorflow* and *tensorflow.lite*.
+"""
+from onnxruntime import InferenceSession
+import os
+import sys
+import subprocess
+import timeit
+import numpy as np
+import tensorflow as tf
+from tensorflow import keras
+from tensorflow.keras import Input
+try:
+    import tensorflow_hub as tfhub
+except ImportError:
+    # no tensorflow_hub
+    print("tensorflow_hub not installed.")
+    sys.exit(0)
+
+########################################
+# Downloads the model.
+hub_layer = tfhub.KerasLayer(
+    "https://tfhub.dev/google/efficientnet/b0/classification/1")
+model = keras.Sequential()
+model.add(Input(shape=(224, 224, 3), dtype=tf.float32))
+model.add(hub_layer)
+print(model.summary())
+
+########################################
+# Saves the model.
+if not os.path.exists("efficientnetb0clas"):
+    os.mkdir("efficientnetb0clas")
+tf.keras.models.save_model(model, "efficientnetb0clas")
+
+input_names = [n.name for n in model.inputs]
+output_names = [n.name for n in model.outputs]
+print('inputs:', input_names)
+print('outputs:', output_names)
+
+########################################
+# Testing the model.
+input = np.random.randn(2, 224, 224, 3).astype(np.float32)
+expected = model.predict(input)
+print(expected)
+
+########################################
+# Run the command line.
+proc = subprocess.run(
+    'python -m tf2onnx.convert --saved-model efficientnetb0clas '
+    '--output efficientnetb0clas.onnx --opset 12'.split(),
+    capture_output=True)
+print(proc.returncode)
+print(proc.stdout.decode('ascii'))
+print(proc.stderr.decode('ascii'))
+
+########################################
+# Runs onnxruntime.
+session = InferenceSession("efficientnetb0clas.onnx")
+got = session.run(None, {'input_1:0': input})
+print(got[0])
+
+########################################
+# Measures the differences.
+print(np.abs(got[0] - expected).max())
+
+########################################
+# Measures processing time.
+print('tf:', timeit.timeit('model.predict(input)',
+                           number=10, globals=globals()))
+print('ort:', timeit.timeit("session.run(None, {'input_1:0': input})",
+                            number=10, globals=globals()))