Skip to content

Support gqa in aten spda #2408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Support gqa in aten spda #2408

wants to merge 4 commits into from

Conversation

justinchuby
Copy link
Collaborator

@justinchuby justinchuby commented Jun 20, 2025

Signed-off-by: Justin Chu <[email protected]>
@justinchuby justinchuby changed the title Support gpa in aten spda Support gqa in aten spda Jun 20, 2025
Copy link

codecov bot commented Jun 20, 2025

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
15549 6 15543 1881
View the top 3 failed test(s) by shortest run time
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0199_test_cast_FLOAT_to_STRING
Stack Traces | 0.003s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_cast_FLOAT_to_STRING'

The above exception was the direct cause of the following exception:
.nox\test_onnx_ir_git\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_cast_FLOAT_to_STRING' (e=No module named 'tests.onnx_backend_test_code.test_cast_FLOAT_to_STRING') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_cast_FLOAT_to_STRING.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_cast_FLOAT_to_STRING.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT, STRING
E   from onnxscript.onnx_opset import opset21
E   
E   @script()
E   def bck_test_cast_FLOAT_to_STRING(input: FLOAT[3,4]) -> (STRING[3,4]):
E       output = opset21.Cast(input, to=8)
E       return output
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0303_test_cumsum_1d
Stack Traces | 0.003s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_cumsum_1d'

The above exception was the direct cause of the following exception:
.nox\test\lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_cumsum_1d' (e=No module named 'tests.onnx_backend_test_code.test_cumsum_1d') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_cumsum_1d.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_cumsum_1d.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import DOUBLE, INT32
E   from onnxscript.onnx_opset import opset14
E   
E   @script()
E   def bck_test_cumsum_1d(x: DOUBLE[5], axis: INT32) -> (DOUBLE[5]):
E       y = opset14.CumSum(x, axis)
E       return y
onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0665_test_mod_mixed_sign_float16
Stack Traces | 0.003s run time
onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_mod_mixed_sign_float16'

The above exception was the direct cause of the following exception:
.nox\test_onnx_ir_git\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_mod_mixed_sign_float16' (e=No module named 'tests.onnx_backend_test_code.test_mod_mixed_sign_float16') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_mod_mixed_sign_float16.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_mod_mixed_sign_float16.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT16
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_mod_mixed_sign_float16(x: FLOAT16[6], y: FLOAT16[6]) -> (FLOAT16[6]):
E       z = opset13.Mod(x, y, fmod=1)
E       return z

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Signed-off-by: Justin Chu <[email protected]>
Signed-off-by: Justin Chu <[email protected]>
Comment on lines +1994 to +1996
return _aten_scaled_dot_product_attention_bool_mask_onnx(
query, key, value, attn_mask, scale, dropout_p, enable_gqa=enable_gqa
)

Check failure

Code scanning / CodeQL

Wrong name for an argument in a call

Keyword argument 'enable_gqa' is not a supported parameter name of [function _aten_scaled_dot_product_attention_bool_mask_onnx](1).

Copilot Autofix

AI 5 days ago

To fix the issue, the keyword argument enable_gqa should be removed from the call to _aten_scaled_dot_product_attention_bool_mask_onnx on line 1994. This ensures that the function is called with only the parameters it supports. The removal of enable_gqa will not affect the functionality of _aten_scaled_dot_product_attention_bool_mask_onnx, as it does not use this argument.

Suggested changeset 1
onnxscript/function_libs/torch_lib/ops/nn.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/onnxscript/function_libs/torch_lib/ops/nn.py b/onnxscript/function_libs/torch_lib/ops/nn.py
--- a/onnxscript/function_libs/torch_lib/ops/nn.py
+++ b/onnxscript/function_libs/torch_lib/ops/nn.py
@@ -1994,3 +1994,3 @@
         return _aten_scaled_dot_product_attention_bool_mask_onnx(
-            query, key, value, attn_mask, scale, dropout_p, enable_gqa=enable_gqa
+            query, key, value, attn_mask, scale, dropout_p
         )
EOF
@@ -1994,3 +1994,3 @@
return _aten_scaled_dot_product_attention_bool_mask_onnx(
query, key, value, attn_mask, scale, dropout_p, enable_gqa=enable_gqa
query, key, value, attn_mask, scale, dropout_p
)
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
@justinchuby justinchuby marked this pull request as draft June 20, 2025 19:14
Signed-off-by: Justin Chu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

[ONNX] Support for grouped query attention
1 participant