[Rewriter]: fuse successive Relu/Clip nodes #2410

AyoubMDL · 2025-06-22T12:46:43Z

This PR adds the following transformation:

Relu(Relu(X)) -> Relu
Relu(Clip(X)) -> Clip
Clip(Relu(X)) -> Clip
Clip(Clip(X)) -> Clip

codecov · 2025-06-22T15:15:29Z

❌ 13 Tests Failed:

Tests completed	Failed	Passed	Skipped
1450	13	1437	453

View the top 2 failed test(s) by shortest run time

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0245_test_clip_example

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_clip_example'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_clip_example' (e=No module named 'tests.onnx_backend_test_code.test_clip_example') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_clip_example.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_clip_example.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_clip_example(x: FLOAT[3], min: FLOAT, max: FLOAT) -> (FLOAT[3]):
E       y = opset13.Clip(x, min, max)
E       return y

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1126_test_slice_end_out_of_bounds

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_slice_end_out_of_bounds'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_slice_end_out_of_bounds' (e=No module named 'tests.onnx_backend_test_code.test_slice_end_out_of_bounds') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_slice_end_out_of_bounds.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_slice_end_out_of_bounds.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT, INT64
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_slice_end_out_of_bounds(x: FLOAT[20,10,5], starts: INT64[1], ends: INT64[1], axes: INT64[1], steps: INT64[1]) -> (FLOAT[20,9,5]):
E       y = opset13.Slice(x, starts, ends, axes, steps)
E       return y

View the full list of 1 ❄️ flaky tests

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1134_test_softmax_axis_1

Flake rate in main: 8.70% (Passed 21 times, Failed 2 times)

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_softmax_axis_1'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_softmax_axis_1' (e=No module named 'tests.onnx_backend_test_code.test_softmax_axis_1') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_softmax_axis_1.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_softmax_axis_1.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_softmax_axis_1(x: FLOAT[3,4,5]) -> (FLOAT[3,4,5]):
E       y = opset13.Softmax(x, axis=1)
E       return y

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

Copilot

Pull Request Overview

This PR adds graph rewrite rules to fuse consecutive Relu and Clip operations, updates the test harness to control ONNX Runtime’s optimization level, and provides unit tests to validate the new transformations.

Introduce four fusion rules (Relu(Relu), Relu(Clip), Clip(Relu), Clip(Clip)) in fuse_relus_clips.py
Extend assert_numerically_equal in testing.py to accept an ort_optimization_level argument
Add comprehensive tests in fuse_relus_clips_test.py to cover valid and invalid fusion scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
onnxscript/rewriter/fuse_relus_clips.py	Implement new `RewriteRule` classes and assemble them into a set.
onnxscript/rewriter/testing.py	Update test helper to pass through ONNX Runtime optimization level.
onnxscript/rewriter/fuse_relus_clips_test.py	Add unit tests for each fusion pattern and edge‐case validations.

Comments suppressed due to low confidence (2)

onnxscript/rewriter/fuse_relus_clips.py:161

The variable name fuse_sucessive_relu_clip_rule has a typo (sucessive vs. successive). Rename it to fuse_successive_relu_clip_rule for consistency with the other rules, and update any references.

fuse_sucessive_relu_clip_rule = FuseSuccessiveReluClip().rule()

onnxscript/rewriter/testing.py:27

[nitpick] The Args: section in the docstring does not match the parameter order of the function signature. Consider reordering the entries so they follow (original_model_proto, rewritten_model_proto, args, ort_optimization_level, rtol, atol).

        ort_optimization_level: Onnxruntime optimization level.

justinchuby · 2025-06-23T16:09:24Z

onnxscript/rewriter/fuse_relus_clips_test.py

+        expected_op_type: str,
+        dtype: str = "float",
+    ):
+        base_model = ir.serde.deserialize_model(base_model)


Suggested change

base_model = ir.serde.deserialize_model(base_model)

justinchuby · 2025-06-23T16:09:31Z

onnxscript/rewriter/fuse_relus_clips_test.py

+
+    def run_test(
+        self,
+        base_model: onnx.ModelProto | ir.Model,


Suggested change

base_model: onnx.ModelProto | ir.Model,

base_model: ir.Model,

justinchuby · 2025-06-23T16:09:57Z

onnxscript/rewriter/fuse_relus_clips_test.py

+        model_proto = onnx.parser.parse_model("""
+            < ir_version: 10, opset_import: ["" : 20] >
+            test_model (float[N, 32, 14] X) => (float [N, ?, ?] Y)
+            {
+                x1 = Relu(X)
+                x2 = Relu(x1)
+                Y = Relu(x2)
+            }
+        """)
+        self.run_test(model_proto, expected_op_type="Relu")


Suggested change

model_proto = onnx.parser.parse_model("""

< ir_version: 10, opset_import: ["" : 20] >

test_model (float[N, 32, 14] X) => (float [N, ?, ?] Y)

{

x1 = Relu(X)

x2 = Relu(x1)

Y = Relu(x2)

}

""")

self.run_test(model_proto, expected_op_type="Relu")

model = ir.from_onnx_text("""

< ir_version: 10, opset_import: ["" : 20] >

test_model (float[N, 32, 14] X) => (float [N, ?, ?] Y)

{

x1 = Relu(X)

x2 = Relu(x1)

Y = Relu(x2)

}

""")

self.run_test(model, expected_op_type="Relu")

Here and below, thanks!

justinchuby

Thank you - I think this can be part of the default rewrite rules

cc @gramalingam

- Relu(Relu(X)) -> Relu - Relu(Clip(X)) -> Clip - Clip(Relu(X)) -> Clip - Clip(Clip(X)) -> Clip

github-project-automation bot added this to ONNX Script Review Board Jun 22, 2025

github-project-automation bot moved this to Todo in ONNX Script Review Board Jun 22, 2025

justinchuby requested a review from Copilot June 22, 2025 15:17

Copilot AI reviewed Jun 22, 2025

View reviewed changes

justinchuby reviewed Jun 23, 2025

View reviewed changes

AyoubMDL added 3 commits June 24, 2025 01:41

[Rewriter]: fuse successive Relu/Clip nodes

d827d63

- Relu(Relu(X)) -> Relu - Relu(Clip(X)) -> Clip - Clip(Relu(X)) -> Clip - Clip(Clip(X)) -> Clip

review: use ir APIs

be812d5

review: add rewriter to default rules

e0f6332

AyoubMDL force-pushed the fuse-sucessive-relus-clips branch from ab90aaf to e0f6332 Compare June 23, 2025 23:41

AyoubMDL mentioned this pull request Jun 24, 2025

Optimizer Suggestions for Common Graph Fusions #2301

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Rewriter]: fuse successive Relu/Clip nodes #2410

[Rewriter]: fuse successive Relu/Clip nodes #2410

Uh oh!

AyoubMDL commented Jun 22, 2025

Uh oh!

codecov bot commented Jun 22, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

justinchuby Jun 23, 2025

Uh oh!

justinchuby Jun 23, 2025

Uh oh!

justinchuby Jun 23, 2025

Uh oh!

justinchuby Jun 23, 2025

Uh oh!

justinchuby left a comment •

edited

Loading

Uh oh!

Uh oh!

	base_model: onnx.ModelProto \| ir.Model,
	base_model: ir.Model,

[Rewriter]: fuse successive Relu/Clip nodes #2410

Are you sure you want to change the base?

[Rewriter]: fuse successive Relu/Clip nodes #2410

Uh oh!

Conversation

AyoubMDL commented Jun 22, 2025

Uh oh!

codecov bot commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 13 Tests Failed:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

justinchuby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

justinchuby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

justinchuby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

justinchuby Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

justinchuby left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jun 22, 2025 •

edited

Loading

justinchuby left a comment •

edited

Loading