Firmware transaction timeout when setting backlight #5397
Comments
|
It's likely that the backlight control is not the thing that killed the VPU, just the first to discover the body, unless the additional load of powering the backlight was last straw on an underpowered system.
|
These are both wrong for using the 800x480 DSI screen drivers with vc4-kms-v3d. vc4-kms-dsi-7inch already includes instantiating the edt-ft5x06 driver for the touch controller, and rpi-panel-attiny-regulator includes the backlight driver. As pelwell says, something appears to have killed the firmware. |
|
Yes - that first log line looks like the smoking gun: That means the ARM has detected an I2C interrupt condition that it wasn't expecting, and clearing it (the only sane thing to do, otherwise the ARM could end up spinning) will probably leave the firmware waiting indefinitely. |
|
But only a fraction of this power can be delivered if it is all being routed through the Pi - the topology matters. However, my questions were rendered moot by @6by9's observations. |
For some reason, if I only have the vc4-kms-dsi-7inch overlay then touch and backlight isn't working/loaded. I'm sure it's something related to this particular distribution, or how the overlays are pulled in when the image is built, but that's the part I'm trying to solve. I get this in dmesg when I only have vc4-kms-dsi-7inch in config.txt: I've been trying to decipher the overlay source file, but I'm coming up short, how can I specify overlays to do the same thing as vc4-kms-dsi-7inch does? Ultimately I will use a different display so if I can just manually do the necessary overlays then I'll have a better understanding of what components are used.
Unfortunately this distribution doesn't have those tools. I've tried in the past to manually install them, but was unsuccessful. And I'm positive that if I use a standard distribution then I won't have the problems (but that doesn't help my end goal). Is there a way to "copy" this vclog from this machine to another RPi where I can read it, or is the vclog/vcdbg doing direct firmware reads?. |
|
vclog is open source, and you can easily build it yourself: https://github.com/raspberrypi/utils/tree/master/vclog |
Just found it, great, building it now. I can copy files onto the OS and run so hopefully that will bring some clarity. Thanks for your help! |
|
I think this is what I ran into last time: Updated config.txt: |
|
It seems that I have a firmware from 1/20/2022, I don't know if that is good, bad or expected: |
|
Unfortunately that firmware is too old to support vclog - it needs to contain a patch from Jul 13 2022. |
|
There is a static build of vcdbg that might work for you: https://drive.google.com/file/d/1HS9E5vnxxNqrizB4mEYrnFoQQ1axSRKm/view?usp=sharing |
Thanks, I can run that, but still no luck :( |
Stupid question, but how can I upgrade the firmware? Is that part of the distribution image, or can/should I go through rpi-update or something similar (realizing I don't have that tool in this distribution). |
To just update the firmware, skipping the kernel: However, I don't think it would tell us much. If there were any DT-related errors you would be able to read them from /proc/device-tree/chosen/user-warnings. |
|
What is this distribution? 32 or 64bit? Firmware can be found in https://github.com/raspberrypi/firmware/tree/master/boot. *.elf and *.dat. The ATTiny on the official panel controls the backlight, DSI->DPI bridge chip, and touch, bridge, and panel reset lines. There is a question as to whether we are now debugging this 3rd party device rather than the kernel drivers. The drivers are intended for our display, so clones play second fiddle. |
|
The distribution is 64-bit. I tried to upgrade the firmware using rpi-boot (no errors), but after reboot the hash/version is the same in dmesg. Is it possible the boot process downgrades the firmware? Or a better question may be, is the firmware loaded during boot (vs stored on the chip)? I understand that this is a clone display, however it has worked perfectly fine before (with the dsi-7inch command), I believe there's something wrong with the distribution (but it's a commercial distribution, https://www.balena.io/os#download-os, if there's anything wrong then I'll work with them to get it corrected, but first step is to determine where the issue is, which you're absolutely helping me with, and I greatly appreciate it). I actually have an official display, I can try that as well to eliminate that potential issue. But I'm still curious if I can manually enter the dtoverlay commands that dsi-7inch include? |
|
Let's focus on understanding the symptoms, as @6by9 is asking you to do. |
|
This file doesn't exist (the |
That would be the correct value. If the driver has failed to probe, you could try Error -5 is EIO. You've provided only a tiny snippet of the log around that. Did you also have a |
|
As far as |
|
I tried rmmod/modprobe and I'm getting the same error. I don't see the i2c transfer error in dmesg. Here's the full pastebin: https://pastebin.com/xUdbTvtj |
|
For what it's worth, I tried to load |
Yes, there would be, if that's the case, and the i2cdetect output suggests that it is. Are you using a dt-blob.bin on the CM4? If so, can you upload it somewhere so I can examine it? Alternatively, if you know for a fact the values of DISPLAY_I2C_PORT, DISPLAY_SCL and DISPLAY_SDA, you can just post them here. Or even a datasheet or software resources for the display. |
|
You can confirm which pins rpi-ft5406 is using You may also find you can break the touchscreen by running |
|
Yes, I'm using a dt-blob, which I built from source files for the Waveshare PoE board, that I used as a template for my own baseboard (https://www.waveshare.com/wiki/Compute_Module_4_PoE_Board). I then just changed the pins to match my design (or if they were already correct, I can't remember now). I have the source file somewhere on a different computer, but attached here is the compiled file if that helps? Looking at my schematic, the I2C bus for the display is on pins 35 & 36 on the CM4 module (labeled ID_SD and ID_SC). If I remember correctly, before I switched to Bullseye, the bus was on a different bus id, but I'm pretty sure I'm using the same dt-blob (I'll be the first to admit I'm still learning how all the parts work together). However, the display, touch and backlight all work fine (with using the rpi* drivers), but I want to make sure I have it using the correct overlays and obviously eliminate the i2c exception issue. |
It didn't break, but it shows the same output, detecting the devices on 0x38 and 0x45. |
|
Just noticed something, |
|
Thanks, that's all consistent with what you said about the display being on 0 & 1. In order to get edt_ft5x06 working with your display it would need to be hacked or extended to target i2c0 instead of i2c10 (or i2c_csi_dsi as it is labelled). The source for the overlay is https://github.com/raspberrypi/linux/blob/rpi-6.1.y/arch/arm/boot/dts/overlays/edt-ft5406-overlay.dts and https://github.com/raspberrypi/linux/blob/rpi-6.1.y/arch/arm/boot/dts/overlays/edt-ft5406.dtsi. Which brings us back to your original question:
Explain precisely what you want it to do and I can tell you how to do it. It might accelerate the process if you were to make a start on a new overlay, copying the bits of vc4-kms-dsi-7inch that you need and perhaps starting to change them how you want. |
Ok, I'm glad things are consistent at least. So my end goal is to build my own display board for my CM4 baseboard. Which I've learned is quite hard, so I'm looking into HDMI for the display part, but I still need backlight and touch control. So in my process of doing that I'm working on a prototype display board that will have I2C with FT5406 and a PWM I2C controller for backlight. I was thinking if I can configure the software to use the separate components (overlays) then I can test out those two parts (besides the HDMI part) and verify that design is correct, while in parallel work on the HDMI->parallel part. And the end goal is to have it run on the BalenaOS standard/public distribution with just some boot file updates/dtblob/config.txt changes (so it'll be easier to upgrade in the future). |
|
I totally missed that you'd routed it to i2c-0 instead of 10 as you hadn't made any comment about it when I'd asked for For a simple swap, you can use You can't really split vc4-kms-dsi-7inch apart as the different devices link to one another, eg you need to tell the display where the appropriate backlight controller is, and DSI peripherals have to sit under the DSI host controller node that therefore has to be enabled. There is an underlying expectation that if you're doing a custom carrier, then you are likely to need to create your own device tree and that may as well include all the relevant configuration rather than relying on the standard overlays. Alternatively you could have one overlay that applied all the relevant changes for your base board. |
Yes. |
|
I upgraded the firmware to this version (by manually copying the elf and dat files to the boot partition since this OS has a read-only root system rpi-update didn't work): And now the I2C bus seems to be on 10 (I didn't change anything else) and the I also still get this error (but the driver is loaded and I can control the backlight, until I get the interrupt error): Noteworthy is that my other i2c bus continue to work, it's just the display i2c bus that gets locked up (but that may be expected). So now I have the correct drivers, but I'm still getting the random unexpected interrupt. |
|
What I don't understand though is that I2C bus 0 and 10 seems to be the same: |
|
Here's the dmesg for when I get the error. Sometimes I can run for minutes, other times it happens after a few seconds: |
|
What is the content of your config.txt now? |
|
config.txt: |
|
Is there any way to determine where this interrupt is coming from? I'm also curious on why there is an interrupt in the i2c driver, since i2c is polled from the master, or maybe it's unrelated (or I'm lacking understanding). |
|
The interrupt can only be because both the firmware and kernel are being asked to drive I2C0. I find this a bit suspicious. What does |
|
Note that this is how BalenaOS does it, by mouting most things readonly, I assume it helps if there's a power outage, etc (but it's not like I can control it, it's just the way they have it, it's basically using Yocto to generate the distribution). Here's the FDT file: https://transfer.sh/ym78ka/fdt-hakan Note when I upgraded the firmware I didn't change the kernel8.img as I assumed that wasn't part of the firmware (and because of the read-only filesystem I wasn't able to run rpi-update so I had to manually replace files in the boot partition). Maybe that was wrong? |
|
The fdt file shows that the |
|
That's odd, I don't see |
|
I've just noticed in #5397 (comment) bcm2708_fb shouldn't be running either if vc4-kms-v3d is enabled. You should get a simple_fb initially instead with soc/fb being disabled ( |
I also noticed this now: I don't see any reference to kms in dmesg or lsmod, but I don't know if that's expected. |
|
The failure to enable the KMS driver is prompting the firmware to automatically apply the touchscreen overlay. Strangely, I can't find any reference to an |
|
Just so I didn't mess up the firmware upgrade I was able to get rpi-update to work by running this command (I had to manually copy in readelf and its library since it wasn't in the distribution, that's why the paths are updated): So now it's running this firmware (f5c4fc199c8d8423cb427e509563737d1ac21f3c was But it's still using |
|
The |
|
Ok, so two steps forward and one back. I copied the overlays from 5.15.92 firmware and now I don't have any references to 2708 and no user-warnings. One step back because I don't have backlight control any more (same error: rpi_touchscreen_attiny 10-0045: Failed to read REG_ID reg: -5), but I assume it worked before because the old drivers got loaded instead. I tried to copy kernel8.img as well, and that bricked by system so I'm going to put that back and see if I still get interrupt errors. However I need to figure out how to get backlight control back (my app expects the sys/class node to exist). |
|
Can you run i2cdump on addresses 0x45 and 0x38 of i2c-10? |
|
Ah, so maybe two steps forward and 1.5 back... now the display i2c is back on bus id 0. I can try the swap, but is the correct fix to update my dt-blob? I'm looking at the examples and they all seem to have 0 in there. What is controlling the i2c bus ids? For reference, my carrier board is modeled after the official RPi IO Board, same pins for the display/i2c, and I'm using the dtblob from the main repo (I just removed the non-CM4 stuff). I'm not saying I'm confident all that is correct, but it's not a conscious decision to put the display i2c bus on id 0. |
|
FWIW I tried the cm-swap overlay and it didn't seem to change anything. |
|
So what does dt-blob.bin only affects the firmware, not the kernel. If using vc4-kms-v3d and vc4-kms-dsi-7inch, then it is largely irrelevant other than the first around 6 seconds of boot (the rainbow screen and early console). |
|
For the avoidance of doubt, that's |
|
Of course, my bad, the cm-swap-i2c0 wasn't in the overlays folder. That's fixed now, and the drivers for touch and backlight are loaded. I'm still back 1 step (I lost track), because now the backlight is turned off, and I don't seem to have control of it (I now have /sys/class/backlight/10-0045, but nothing happens when I write to it, and the backlight is off). The rpi_panel_attiny_regulator overlay is loaded though. |
|
To try to eliminate potential issues I set up a different CM4, same carrier board, but with the official 7" RPi display. Now I can control the backlight, it seems to load the touch controller, but after the initial boot sequence (where I see log lines) it flashes/fades to grey and then the display is blank. It seems to be similar to this issue: #4686, however I have tried to add the |
|
To further troubleshoot this I have now set up both a standard Pi 4 and a CM4 with IO Board (so not using my custom carrier). I've loaded the latest 64-bit Raspberry Pi OS Lite on both. I have tested the official 7" DSI display, and my Waveshare 4.3" clone. I've updated everything with |
|
#4946 |
|
Thanks @6by9, it sounds like I need to re-design my board as DSI0 isn't useful (with KMS) yet. Of course I could use legacy/fkms, but I prefer to build upon something that will be the best supported in the long run. I'll close this issue as I think it's been determine the root cause thanks to yours and @pelwell's help. I'm now a little wiser and learned a lot so I appreciate that. |
Describe the bug
Intermittent errors when setting backlight brightness. It works some times, but after I get this error then it stops working (and the system becomes unstable). To make things more complicated, this is a CM4 on a custom carrier board, so I can't rule out that it's not a hardware problem (which I doubt, this board has worked before without issues). I'll post this issue to see if there's any indication what the root cause may be.
Steps to reproduce the behaviour
Set backlight brightness
Device (s)
Raspberry Pi CM4
System
uname -a
Linux a487250 5.15.34-v8 #1 SMP PREEMPT Tue Apr 19 19:21:26 UTC 2022 aarch64 aarch64 aarch64 GNU/LinuxLogs
[10147.406933] i2c-bcm2835 fe205000.i2c: Got unexpected interrupt (from firmware?)
[10158.470804] ------------[ cut here ]------------
[10158.470826] Firmware transaction timeout
[10158.470877] WARNING: CPU: 1 PID: 10222 at drivers/firmware/raspberrypi.c:67 rpi_firmware_property_list+0x138/0x20c
[10158.470913] Modules linked in: ip6t_REJECT nf_reject_ipv6 ip6table_filter ipt_REJECT nf_reject_ipv4 ip6_tables xt_MASQUERADE nf_conntrack_netlink nfnetlink br_netfilter rfkill xt_owner i2c_dev joydev rpi_panel_attiny_regulator edt_ft5x06 v3d gpu_sched raspberrypi_ts bcm2835_v4l2(C) tc358762 bcm2835_isp(C) i2c_mux_pinctrl bcm2835_codec(C) gpio_keys i2c_mux videobuf2_vmalloc bcm2835_mmal_vchiq(C) v4l2_mem2mem rtc_pcf85063 videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 dwc2 raspberrypi_hwmon i2c_brcmstb roles videobuf2_common vc_sm_cma(C) videodev snd_bcm2835(C) mc rpi_backlight backlight rpivid_mem panel_simple drm_dp_aux_bus uio_pdrv_genirq nvmem_rmem uio sch_fq_codel fuse
[10158.471194] CPU: 1 PID: 10222 Comm: Event Loop 1 Tainted: G C 5.15.34-v8 #1
[10158.471209] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
[10158.471217] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[10158.471232] pc : rpi_firmware_property_list+0x138/0x20c
[10158.471244] lr : rpi_firmware_property_list+0x138/0x20c
[10158.471256] sp : ffffffc00a5e3ae0
[10158.471262] x29: ffffffc00a5e3ae0 x28: ffffff8100358000 x27: ffffff8100a1fa40
[10158.471286] x26: ffffffd473166b70 x25: ffffffc008475008 x24: ffffff8101e68500
[10158.471308] x23: 0000000000001000 x22: 0000000000000010 x21: ffffff8100a1fa00
[10158.471329] x20: ffffffc008475000 x19: 00000000ffffff92 x18: 0000000000000000
[10158.471351] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[10158.471372] x14: 0000000000000000 x13: 74756f656d697420 x12: 6e6f69746361736e
[10158.471392] x11: 656820747563205b x10: 2d2d2d2d2d2d2d2d x9 : ffffffd4710dbbac
[10158.471413] x8 : ffffffd4730438a0 x7 : 6361736e61727420 x6 : ffffffd4731d4ab9
[10158.471434] x5 : c0000000ffffefff x4 : 0000000000000000 x3 : 0000000000000027
[10158.471455] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff8100358000
[10158.471476] Call trace:
[10158.471483] rpi_firmware_property_list+0x138/0x20c
[10158.471495] rpi_firmware_property+0x7c/0xfc
[10158.471507] rpi_backlight_update_status+0x68/0xd0 [rpi_backlight]
[10158.471528] backlight_update_status+0x38/0x60 [backlight]
[10158.471552] backlight_device_set_brightness+0x54/0x94 [backlight]
[10158.471571] brightness_store+0x74/0x94 [backlight]
[10158.471589] dev_attr_store+0x24/0x38
[10158.471606] sysfs_kf_write+0x48/0x5c
[10158.471620] kernfs_fop_write_iter+0xc4/0x17c
[10158.471631] new_sync_write+0x80/0xd8
[10158.471646] vfs_write+0x118/0x13c
[10158.471660] ksys_pwrite64+0x58/0x98
[10158.471673] __arm64_sys_pwrite64+0x28/0x34
[10158.471687] invoke_syscall+0x84/0x11c
[10158.471699] el0_svc_common.constprop.0+0xcc/0x100
[10158.471711] do_el0_svc+0x54/0x84
[10158.471721] el0_svc+0x24/0x54
[10158.471737] el0t_64_sync_handler+0xbc/0x158
[10158.471751] el0t_64_sync+0x1a0/0x1a4
[10158.471763] ---[ end trace c634a32393bac947 ]---
[10158.471865] rpi-backlight rpi_backlight: Failed to set brightness
[10159.494846] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -110
[10161.542850] hwmon hwmon1: Failed to get throttled (-110)
[10162.566843] rpi-backlight rpi_backlight: Failed to set brightness
[10165.638847] rpi-backlight rpi_backlight: Failed to set brightness
[10391.686865] INFO: task kworker/3:2:7875 blocked for more than 120 seconds.
[10391.686908] Tainted: G WC 5.15.34-v8 #1
[10391.686927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10391.686943] task:kworker/3:2 state:D stack: 0 pid: 7875 ppid: 2 flags:0x00000008
[10391.686977] Workqueue: events_freezable mmc_rescan
Additional context
config.txt:
This is using a DSI clone (Waveshare 4.3" DSI), but has worked fine before.
The text was updated successfully, but these errors were encountered: