NotImplementedError: The operator 'aten::_conv_depthwise2d' is not currently implemented for the XPU device.

[mengfei25]

🐛 Describe the bug

python benchmarks/dynamo/huggingface.py --accuracy --float16 -d xpu -n10 --inference --only YituTechConvBert --backend=inductor --cold-start-latency --timeout=10800

loading model: 0it [00:01, ?it/s]
xpu  eval  YituTechConvBert
ERROR:common:
Traceback (most recent call last):
  File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/common.py", line 2270, in check_accuracy
    new_result = self.run_n_iterations(
  File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/common.py", line 1957, in run_n_iterations
    model_iter_fn(mod, inputs, collect_outputs=False)
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 658, in _fn
    return fn(*args, **kwargs)
  File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/huggingface.py", line 523, in forward_pass
    def forward_pass(self, mod, inputs, collect_outputs=True):
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 850, in _fn
    return fn(*args, **kwargs)
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1207, in forward
    return compiled_fn(full_args)
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 331, in runtime_wrapper
    all_outs = call_func_at_runtime_with_args(
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
    out = normalize_as_list(f(args))
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 692, in inner_fn
    outs = compiled_fn(args)
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 498, in wrapper
    return compiled_fn(runtime_args)
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 561, in __call__
    return self.current_callable(inputs)
  File "/home/sdp/.conda/envs/e2e_ci/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2449, in run
    return model(new_inputs)
  File "/tmp/tmpd7imunxp/hh/chhfzkj7uoru46ifjhp5vzlulb5ejwuxtzv4n5vxcysnzadaehek.py", line 1113, in call
    buf14 = extern_kernels.convolution(reinterpret_tensor(buf4, (1, 768, 1, 512), (0, 1, 0, 768), 0), reinterpret_tensor(arg14_1, (768, 1, 1, 9), (9, 9, 9, 1), 0), stride=(1, 1), padding=(0, 4), dilation=(1, 1), transposed=False, output_padding=(0, 0), groups=768, bias=None)
NotImplementedError: The operator 'aten::_conv_depthwise2d' is not currently implemented for the XPU device. Please open a feature on https://github.com/intel/torch-xpu-ops/issues. You can set the environment variable `PYTORCH_ENABLE_XPU_FALLBACK=1` to use the CPU implementation as a fallback for XPU unimplemented operators. WARNING: this will bring unexpected performance compared with running natively on XPU.
TorchDynamo optimized model failed to run because of following error
fail_to_run

Versions

pytorch: 8497491f38cb80aed0e91de9d0d7c3abd83ebfee
torch-xpu-ops: 87906ac

[dvrogozh]

I am seeing the same issue started to happen on multiple Huggingface native Transformers tests in multiple models. Compare:

• https://github.com/intel/torch-xpu-ops/actions/runs/14550731463
  • transformers==4.47.0, accelerate==1.6.0, torch==2.8.0.dev20250419+xpu
  • 316 failed in tests/models
• https://github.com/intel/torch-xpu-ops/actions/runs/14421217689
  • transformers==4.47.0, accelerate==1.6.0, 2.8.0.dev20250412+xpu
  • 9 failed in tests/models

As HF side is the same, that's a regression caused by some pytorch changes. And that's quite a huge regression. @ZhiweiYan-96, did you identify the root cause by now? Please, help to comment in this issue. If that's some change in pytorch, then we must report it to pytorch rather than hold on torch-xpu-ops side.

[dvrogozh]

Still happens on the latest pytorch and HF cloned today (huggingface/transformers@a42ba80, pytorch/pytorch@bf28d1c). Here is a log from one of the tests which fail on HF transformers test suite side:

$ python3 -m pytest tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_different_timm_backbone
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/dvrogozh/git/huggingface/transformers
configfile: pyproject.toml
plugins: hypothesis-6.129.4, xdist-3.6.1, order-1.3.0, rerunfailures-15.0, asyncio-0.23.8, timeout-2.3.1, rich-0.2.0, anyio-4.8.0, typeguard-4.3.0
asyncio: mode=Mode.STRICT
collected 1 item

tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_different_timm_backbone FAILED                                                                      [100%]

================================================================================================ FAILURES =================================================================================================
__________________________________________________________________________ ConditionalDetrModelTest.test_different_timm_backbone __________________________________________________________________________

self = <tests.models.conditional_detr.test_modeling_conditional_detr.ConditionalDetrModelTest testMethod=test_different_timm_backbone>

    def test_different_timm_backbone(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

        # let's pick a random timm backbone
        config.backbone = "tf_mobilenetv3_small_075"
        config.backbone_config = None
        config.use_timm_backbone = True
        config.backbone_kwargs = {"out_indices": [2, 3, 4]}

        for model_class in self.all_model_classes:
            model = model_class(config)
            model.to(torch_device)
            model.eval()
            with torch.no_grad():
>               outputs = model(**self._prepare_for_class(inputs_dict, model_class))

tests/models/conditional_detr/test_modeling_conditional_detr.py:458:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
src/transformers/models/conditional_detr/modeling_conditional_detr.py:1534: in forward
    features, object_queries_list = self.backbone(pixel_values, pixel_mask)
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
src/transformers/models/conditional_detr/modeling_conditional_detr.py:415: in forward
    out = self.conv_encoder(pixel_values, pixel_mask)
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
src/transformers/models/conditional_detr/modeling_conditional_detr.py:392: in forward
    features = self.model(pixel_values) if self.config.use_timm_backbone else self.model(pixel_values).feature_maps
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
../../../pytorch.xpu/lib/python3.12/site-packages/timm/models/mobilenetv3.py:378: in forward
    x = b(x)
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/container.py:245: in forward
    input = module(input)
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
../../../pytorch.xpu/lib/python3.12/site-packages/timm/models/_efficientnet_blocks.py:190: in forward
    x = self.conv_dw(x)
../../pytorch/pytorch/torch/nn/modules/module.py:1751: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
../../pytorch/pytorch/torch/nn/modules/module.py:1762: in _call_impl
    return forward_call(*args, **kwargs)
../../../pytorch.xpu/lib/python3.12/site-packages/timm/layers/conv2d_same.py:51: in forward
    return conv2d_same(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = tensor([[[[ 2.0218e-02, -4.2653e-03,  2.2434e-02,  ...,  1.4470e-02,
            9.8226e-03,  0.0000e+00],
          [...    [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
            0.0000e+00,  0.0000e+00]]]], device='xpu:0')
weight = Parameter containing:
tensor([[[[ 1.2045e-02,  1.2690e-02, -7.7089e-03],
          [-2.3422e-03,  6.4745e-03, -3.7017e...084e-02, -7.1098e-03],
          [-4.9985e-02, -2.4990e-02, -1.0419e-02]]]], device='xpu:0',
       requires_grad=True)
bias = None, stride = (2, 2), padding = (0, 0), dilation = (1, 1), groups = 16

    def conv2d_same(
            x,
            weight: torch.Tensor,
            bias: Optional[torch.Tensor] = None,
            stride: Tuple[int, int] = (1, 1),
            padding: Tuple[int, int] = (0, 0),
            dilation: Tuple[int, int] = (1, 1),
            groups: int = 1,
    ):
        x = pad_same(x, weight.shape[-2:], stride, dilation)
>       return F.conv2d(x, weight, bias, stride, (0, 0), dilation, groups)
E       NotImplementedError: The operator 'aten::_conv_depthwise2d' is not currently implemented for the XPU device. Please open a feature on https://github.com/intel/torch-xpu-ops/issues. You can set the environment variable `PYTORCH_ENABLE_XPU_FALLBACK=1` to use the CPU implementation as a fallback for XPU unimplemented operators. WARNING: this will bring unexpected performance compared with running natively on XPU.

../../../pytorch.xpu/lib/python3.12/site-packages/timm/layers/conv2d_same.py:27: NotImplementedError
========================================================================================= short test summary info =========================================================================================
FAILED tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_different_timm_backbone - NotImplementedError: The operator 'aten::_conv_depthwise2d' is not currently implemented for the XPU device. Please open a feature on https://github.com/intel/torch-xpu-ops/issues. You can set the e...
============================================================================================ 1 failed in 3.75s ============================================================================================

[dvrogozh]

The culprit is pytorch/pytorch@08831f3 by @yucai-intel. I guess that this commit actually depends on aten kernels being implemented for XPU backend which is not the case. Effectively this commit should not have been merged without commit pin update for torch-xpu-ops in the pytorch at the first place. This highlights 2 problems:

1. PR [Intel GPU] Allow XPU backend in Depthwise_conv2d&3d operators pytorch/pytorch#149114 which introduced the feature did not reference that it requires an update on torch-xpu-ops side and respective commit pin update. This way pytorch maintainers and reviewers were not notified on the dependency. @yucai-intel
2. PR [Intel GPU] Allow XPU backend in Depthwise_conv2d&3d operators pytorch/pytorch#149114 was merged ignoring CI failures and this issue was actually caught by CI. @guangyey

2025-04-10T09:52:51.1212443Z FAILED [0.0028s] inductor/test_torchinductor.py::GPUTests::test_convolution4_xpu - NotImplementedError: The operator 'aten::_conv_depthwise2d' is not currently implemented for the XPU device.

The aten ops implementation is being done in #1464 which was merged just couple days back and did not yet propagated into toch-xpu-ops commit pin update.

I believe that if we can't make update of torch-xpu-ops commit pin on pytorch side and bring in missing aten ops within say ~24 hours, then pytorch/pytorch@08831f3 must be reverted and merged again only after proper commit pin update. @albanD, @malfet

$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [3532dd4f1efc04326f894e8794c82cc836aa5c0e] [DTensor] StridedShard support uneven sharding (#150490)
git bisect good 3532dd4f1efc04326f894e8794c82cc836aa5c0e
# status: waiting for bad commit, 1 good commit known
# bad: [a78ac409b55eb531e849009727c2d86b5a615ab8] [AOTI] Add _weight_int4pack_mm to the C shim fallback list (#151059)
git bisect bad a78ac409b55eb531e849009727c2d86b5a615ab8
# bad: [e786b3bf543c95a467c5b1fbdd12e502891a1e47] Revert "[inductor] Change minimum number of SMs to 60 to let Ada use Triton GEMM backend (#150888)"
git bisect bad e786b3bf543c95a467c5b1fbdd12e502891a1e47
# good: [7b7b9d707e772a160f3f9753b48f60bab4222d06] [CI] Add XPU compiled check in CICD (#150771)
git bisect good 7b7b9d707e772a160f3f9753b48f60bab4222d06
# bad: [f663aa4e811075ee210a1a9497d7e79e68de7ffc] [c10d][tcp_store] Fix connection reset caused by wrong socket close (#150987)
git bisect bad f663aa4e811075ee210a1a9497d7e79e68de7ffc
# good: [5471e80fb4c34235468e6e5889ff09e88eb44204]  Remove guard_size_oblivious from vector_norm decomposition. (#148809)
git bisect good 5471e80fb4c34235468e6e5889ff09e88eb44204
# good: [37812009fd123d5c4a038ce798eedd4a89eeffad] [profiler] don't disable CUPTI_LAZY_REINIT for cuda >= 12.6 (#150957)
git bisect good 37812009fd123d5c4a038ce798eedd4a89eeffad
# bad: [e7ed50f27b6a87282983d960b612ea50a31cd903] [async TP] Fix handling of case where scatter dim = 0 for 2D output tensor (#150935)
git bisect bad e7ed50f27b6a87282983d960b612ea50a31cd903
# bad: [08831f30bbe745cd9f0c07d1868583a68f613514] [Intel GPU] Allow XPU backend in Depthwise_conv2d&3d operators (#149114)
git bisect bad 08831f30bbe745cd9f0c07d1868583a68f613514
# first bad commit: [08831f30bbe745cd9f0c07d1868583a68f613514] [Intel GPU] Allow XPU backend in Depthwise_conv2d&3d operators (#149114)

[chuanqi129]

fixing by PR pytorch/pytorch#151533

[chuanqi129]

The issue PR has been reverted