DISABLED test_simple_multi_arch_embed_kernel_binary_True_cuda (__main__.AOTInductorTestABICompatibleGpu)

[pytorch-bot]

Platforms: inductor

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 3 hours, it has been determined flaky in 6 workflow(s) with 12 failures and 6 successes.

Debugging instructions (after clicking on the recent samples link):
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:

1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for test_simple_multi_arch_embed_kernel_binary_True_cuda
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.

Sample error message

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/test/inductor/test_aot_inductor.py", line 208, in test_simple_multi_arch
    self.check_model(model, example_inputs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/jenkins/workspace/test/inductor/test_aot_inductor_utils.py", line 241, in check_model
    actual = AOTIRunnerUtil.run(
        model,
    ...<2 lines>...
        dynamic_shapes,
    )
  File "/var/lib/jenkins/workspace/test/inductor/test_aot_inductor_utils.py", line 174, in run
    package_path = AOTIRunnerUtil.compile(
        model,
    ...<2 lines>...
        dynamic_shapes=dynamic_shapes,
    )
  File "/var/lib/jenkins/workspace/test/inductor/test_aot_inductor_utils.py", line 162, in compile
    package_path = torch._inductor.aoti_compile_and_package(
        ep, inductor_configs=inductor_configs
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/__init__.py", line 151, in aoti_compile_and_package
    return aot_inductor_minifier_wrapper(
        _aoti_compile_and_package_inner,
    ...<2 lines>...
        inductor_configs=inductor_configs,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/debug.py", line 961, in aot_inductor_minifier_wrapper
    raise e
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/debug.py", line 931, in aot_inductor_minifier_wrapper
    return func(
        gm,
    ...<4 lines>...
        load_and_run=use_minifier,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/__init__.py", line 194, in _aoti_compile_and_package_inner
    aoti_files = aot_compile(gm, args, kwargs, options=inductor_configs)
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/__init__.py", line 301, in aot_compile
    return compile_fx_aot(
        gm,
        flat_example_inputs,  # type: ignore[arg-type]
        config_patches=options,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1821, in compile_fx_aot
    compiled_artifacts = compile_fx(
        model_,
    ...<5 lines>...
        config_patches=config_patches,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2003, in compile_fx
    return compile_fx(
        model_,
    ...<4 lines>...
        ignore_shape_env=ignore_shape_env,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2060, in compile_fx
    return compile_fx(
        patched_mod,
    ...<3 lines>...
        ignore_shape_env=ignore_shape_env,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2410, in compile_fx
    return inference_compiler(unlifted_gm, example_inputs_)
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_functorch/aot_autograd.py", line 482, in __call__
    return self.compiler_fn(gm, example_inputs)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2251, in fw_compiler_base
    return inner_compile(
        gm,
    ...<5 lines>...
        boxed_forward_device_index=forward_device,
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/contextlib.py", line 85, in inner
    return func(*args, **kwds)
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 745, in compile_fx_inner
    return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        gm,
        ^^^
        example_inputs,
        ^^^^^^^^^^^^^^^
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_dynamo/repro/after_aot.py", line 124, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 897, in _compile_fx_inner
    mb_compiled_graph = fx_codegen_and_compile(
        gm, example_inputs, inputs_to_check, **graph_kwargs
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1579, in fx_codegen_and_compile
    return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1439, in codegen_and_compile
    compiled_fn = AotCodeCompiler.compile(
        graph,
    ...<13 lines>...
        ],
    )
  File "/opt/conda/envs/py_3.13/lib/python3.13/site-packages/torch/_inductor/codecache.py", line 2146, in compile
    subprocess.run(
    ~~~~~~~~~~~~~~^
        cmd.split(), capture_output=True, text=True, check=True
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/conda/envs/py_3.13/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['nvcc', '-fatbin', '/tmp/tmp5a9m9w_a/cnz4ulmnfd7mraahh23lgc2lmejzgx67etxgjpcfh3h7yn6pu5h5/cxyzylofvhajtg4araeujr4c3nexhkpxayedxun2tqcdsjlzfsrd.ptx', '-o', '/tmp/tmp5a9m9w_a/cnz4ulmnfd7mraahh23lgc2lmejzgx67etxgjpcfh3h7yn6pu5h5/cxyzylofvhajtg4araeujr4c3nexhkpxayedxun2tqcdsjlzfsrd.fatbin', '-gencode', 'arch=compute_86,code=compute_86', '-gencode', 'arch=compute_86,code=sm_86']' returned non-zero exit status 1.

Test file path: inductor/test_aot_inductor.py

For all disabled tests (by GitHub issue), see https://hud.pytorch.org/disabled.

cc @clee2000 @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @aakhundov

[pytorch-bot]

Hello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disable a test in PyTorch CI. The information I have parsed is below:

• Test name: test_simple_multi_arch_embed_kernel_binary_True_cuda (__main__.AOTInductorTestABICompatibleGpu)
• Platforms for which to skip the test: inductor
• Disabled by pytorch-bot[bot]

Within ~15 minutes, test_simple_multi_arch_embed_kernel_binary_True_cuda (__main__.AOTInductorTestABICompatibleGpu) will be disabled in PyTorch CI for these platforms: inductor. Please verify that your test name looks correct, e.g., test_cuda_assert_async (__main__.TestCuda).

To modify the platforms list, please include a line in the issue body, like below. The default action will disable the test for all platforms if no platforms list is specified.

Platforms: case-insensitive, list, of, platforms

We currently support the following platforms: asan, dynamo, inductor, linux, mac, macos, rocm, slow, win, windows.

How to re-enable a test

To re-enable the test globally, close the issue. To re-enable a test for only a subset of platforms, remove the platforms from the list in the issue body. This may take some time to propagate. To re-enable a test only for a PR, put Fixes #156930 in the PR body and rerun the test jobs. Note that if a test is flaky, it maybe be difficult to tell if the test is still flaky on the PR.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[jansel]

@desertfire this looks like an AOTI issue, can you take a look at it?

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

[pytorch-bot]

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.