Broadwell iGPU hangs running clpeak with 5.13+ kernels

[thetick1234567890]

Running clpeak causes fence timeouts / GPU hangs on my Broadwell Dell Inspiron 3558 laptop.
I am running with the latest Jammy Ubuntu, latest released compute runtime, latest Intel Media SDK, latest daily mesa and latest mainline kernel.
LibreOffice Calc, Darktable and basic OPENCL code (ie https://www.eriksmistad.no/getting-started-with-opencl-and-gpu-computing/) all work fine running OPENCL workloads on the GPU as I verified with intel-gpu-overlay.
Please let me know if you need any more details to debug.

See all the attached files before asking for more details.

clinfo.txt
clpeak.txt
glxinfo.txt
uname.txt
dmesg.txt

[eero-t]

Maybe clpeak shader kernel just runs too long. Have you tried increasing kernel GPU hang timeout, or disabling GPU hang checker completely?

(People running compute workloads often disable hang checker because compute kernels can run long enough to trigger hang checker, as those workloads are not intended for interactive use, or on desktops where hogging GPU could be an issue.)

[thetick1234567890]

Thanks for the reply.
I rebooted adding i915.enable_hangcheck=0 to grub and verified "/sys/module/i915/parameters/enable_hangcheck" shows N.
Is that the correct procedure to disable the hangcheck for OpenCL workloads ?

I still get same hang with "Fence expiration time out i915-0000:00:02.0:clpeak" but kworker (from KDE ) no longer resets rcs0.

Next I'll try booting to command line without a desktop to avoid GLX for KDE causing GPU contention with clpeak.

[eero-t]

With the hang reset gone, does clpeak produce results?

The remaining fence issue could be a kernel issue on BDW: https://gitlab.freedesktop.org/drm/intel/-/issues/3723

It would be surprising if desktop messes workload that should be using 100% GPU, and not output anything.

[thetick1234567890]

Thanks for the response.

No the Fence timeouts still occur with the three clpeak compute benchmarks ( Single-precision, Double , Integer). The other clpeak benchmarks work fine.

I'll leave this issue open, but it's really not a high priority for me as Darktable, LibreOffice Calc and my custom OPENCL/C code all work fine. I will try new kernels as they arrive and may try a few older LTS kernels as time permits.

[JablonskiMateusz]

@thetick1234567890
could you share output from lspci | grep -i vga ?

[da-phil]

I'm using ubuntu 20.04 on a Dell XPS-13-9380 i7 laptop (VGA compatible controller: Intel Corporation UHD Graphics 620 (Whiskey Lake)).
I experience similar issues (Fence expiration time out i915 output in dmesg) while running darktable which is doing the heavy-lifting processing with OpenCL. It even froze my laptop once completely.
This happened in kernel version 5.14.0-1025-oem, but when I booted a slightly older version (5.10.0-1058-oem) those kernel messages disappeared again, and I never experienced freezes anymore.

[JablonskiMateusz]

Hi @da-phil , if you see some issues on Whiskey Lake then please create a new github issue and provide all info needed to reproduce the issue, i.e. driver version, linux kernel version, clinfo output, steps to repro the issue with binary reproducer (if possible)

[thetick1234567890]

@thetick1234567890 could you share output from lspci | grep -i vga ?

00:02.0 VGA compatible controller: Intel Corporation HD Graphics 5500 (rev 09)

@JablonskiMateusz

[thetick1234567890]

Success... no hangs with kernel 5.10.0 and 5.10.105
Linux dell 5.10.0-051000-generic #202012132330 SMP Sun Dec 13 23:33:36 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Thanks @da-phil

I'll try a bunch of kernels and see if I can bisect the hang down to the first Ubuntu mainline kernel that exhibits the hang.

UPDATE:
Works - 5.10.105-0510105.202203110732 -- Latest LTS kernel that works
Works - 5.12.19-051219.202107201136 -- Latest kernel that works
Hangs - 5.13.0-051300rc1.202105092230 -- First release candidate kernel that hangs
Hangs - 5.13.0-051300.202106272333 -- First released kernel that hangs

Looks like 5.13-rc1 kernel release is the first to have this hang.

[thetick1234567890]

@JablonskiMateusz

Below is the link of the kernel drm i915 pull for the 5.13 kernel which I suspect contains the change that first started these hangs. This of course is just a pure guess as the issue could be in other related kernel component(s) involved with the iGPU.

https://lists.freedesktop.org/archives/dri-devel/2021-April/302129.html

I have done some simple kernel trial and error to narrow this hang down as shown in my previous comment. Kernel 5.13-rc1 introduced the hang. Could you please investigate and/or pass this on to Intel kernel developers as I have absolutely no idea what to do to help determine more details on the hang as the 5.13-rc1 release has huge number of changes.

https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.13-rc1/CHANGES

I would be glad to provide any more details of my hardware or software if needed. Thanks.

[eero-t]

Looks like 5.13-rc1 kernel release is the first to have this hang.

@thetick1234567890 What about 5.17 kernel, does that still have the issue?

(At least the kernel fence issue I linked was fixed before 5.17 release.)

If it's still there with the latest kernel release, please file a regression bug against kernel (and link it here): https://gitlab.freedesktop.org/drm/intel/-/issues?scope=all&state=opened&search=bdw

[thetick1234567890]

I did manage to do some testing and I have resolved my issues tested on 5.17.4 Ubuntu ppa mainline. I was also hitting hangs with only heavier loads with Darktable and some heavy loads of custom openCL I wrote. After some glancing at the kernel drm fencing code I decided to just increase i915.request_timeout_ms to a crazy large value. I noticed clpeak got much farther than before but still hit fence timeouts. So I just kept increasing it more until all the benchmarks, Darktable and my custom openCL worked . I'll have to check back what value last worked but I'm sure it's much larger than is likely needed.

Sorry for my ignorance about drm / cl workloads but this still seems like a drm issue with any 5.13+ kernel as 5.12 and older kernels don't need i915.request_timeout_ms adjusted from the default. I can see in 5.13rc1 drm code there was a significant fence refactoring which appears to me to be what triggered the issue. If I get time I'll look at 5.12 drm code and 5.13 drm fencing refactoring, but I certainly am not knowledgeable on drm fencing.

This is where I plan to start looking but it looks messy but seems relevant and matches the kernel timeline:
https://patchwork.kernel.org/project/dri-devel/cover/20210324121335.2307063-1-tvrtko.ursulin@linux.intel.com/

I am also guessing I hit the issue because this hand-me-down Broadwell laptop is a poor cl performer. Likely due to its old iGPU and is only single channel memory with one slot of memory occupied and the other memory slot being empty. I just rescued it from being thrown in the dumpster and I like to tinker with older PCs. I am just glad the Intel Compute Runtime still works with this laptop.

[eero-t]

This is where I plan to start looking but it looks messy but seems relevant and matches the kernel timeline:
https://patchwork.kernel.org/project/dri-devel/cover/20210324121335.2307063-1-tvrtko.ursulin@linux.intel.com/

Looks like you should file a kernel regression bug and refer that discussion.

Kernel hang timeout is intended for keeping interactive workloads / setups responsive. User can get really frustrated if some background GPU task reduces his/her desktop updates significantly (by running something that does not allow itself to be pre-empted to another GPU task).

On headless servers that run compute workloads, these timeouts are often completely disabled (and real hangs are handled by killing the related tasks and letting kernel do the reset then), but when they're enabled, the defaults should be such that interactivity is retained, but resets do not happen too easily.