Describe the issue

I previously worked with Microsoft.ML.Onnxruntime.GPU 1.16.0 CUDAExecutionProvider, and everything ran fine when using a Conv layer with kernel_shape=2.
However, after upgrading to 1.22.0, the output of my segmentation model became incorrect. The model still executes without errors, but the output image shows a completely wrong pattern.

If I change the Conv layer to use kernel_shape=3 and retrain the model, the results turn out correct again. Also, If I keep kernel_shape=2 but switch to CPUExecutionProvider, the results turn out correct. This makes me suspect there’s an issue with kernel_shape=2 in the newer version of CUDAExecutionProvider.

In my use case, I need to maintain backward compatibility for existing models, so simply switching from kernel_shape=2 to 3 is not an option.

To reproduce

build a segmentation model which contains Conv:
dialations: 1,1
kernel_shape: 2,2
pads: 0,0,1,1
strides: 1,1

Urgency

No response

Platform

Windows

OS Version

22H2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.22.0

ONNX Runtime API

C#

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

12.8