BT(?) causes random Guru Mediation Error

[EDsteve]

@LIFsCode @OOHehir
Not sure if these reboots are connected but i think it is because of BT turning OFF or ON.

Situation 1

After a few hours of recordon_detecton i triggered the knock sensor and then it rebooted with Ruru Mediation Error. This happened only once for now.
Log files:
Using commit cf51c03
Reboot while recordon_detecton.txt
Bootlog after crash.txt

Situation 2

Start the ELOC and just wait. When BT turns off due to the BT time out it restarts. This doesn't happeb all the time but a few times already since a few weeks.

Using commit: 29325f6
Reset when BT turns off.txt

Using commit: 8cc4b99
Reset when BT turns off 2.txt

Using commit: cf51c03
Reset when BT turns off 3.txt

[OOHehir]

@EDsteve @LIFsCode This could be perhaps a memory exhaustion issue, I made some changes as discussed, it may solve this issue also.

[EDsteve]

@LIFsCode @OOHehir

I am adding more reboot logs here, which hopefully give more insight to find the problem.

1. Reboots with assert failed (Happend only once in my tests)

• How it happened: ELOC boots, i connect with app, start recordon_detectoff, device reboots after a few seconds
• Commit d2dd26c
• Log: Reboot_assert failed_during_recording.txt

2. Another Guru Reboot (Happens very seldom)

• Same as "Situation 2" in my first post. ELOC reboots when BT turns off. So that issue seems to be not solved yet.
• Commit: d2dd26c
• Log: Guro_BT-timeout_d2dd26c8f8e37bfb0c3e9861ef266469df3d2101.txt

3. One more Guru

• How it happened: After 12h of detecting with three features. I tried to connect with BT, then i get the Gurur Reboot
• Older commit: 0fbf780
• Log: Guro_Reboot_when_connecting_with_BT_after_12h.txt
• Older commit. Not sure if that issue is still happening with the latest commits

[OOHehir]

@EDsteve Thanks ED. If you can use the method to trace where in the code the crashes occur it would really help. You might remember we did this before, here's a link to the documentation.

[EDsteve]

@OOHehir
Okay. I see what is going on now. The VScode Serial monitor doesn't do backtrace to the files (which i have used lately). But the PlatformIO Monitor does.

Assert failed:

• How it happened: ELOC boots, i connect with app, start recordon_detectoff, devicd reboots after BT time out
• Commit: f4a6a8e
• Log (With backtrace): Assert failed after BT timeout.txt

[LIFsCode]

@EDsteve @OOHehir this looks like some heap or heap fragmentation problem or a stack overflow which then overwrites the subsequent heap sections.

I try to add some additional debugging info and try to reproduce it at home

[EDsteve]

@LIFsCode
Forgot to mention. That this does only happen occasionaly. Haven't really seen a pattern, but i suspect it mostly happens with the first BT timout after a reboot. Not sure if that can make sense.

[EDsteve]

@OOHehir @LIFsCode

I catched another Guro error with Backtrace. This time in a loop:

Guru Reboot Loop

• How it happens: ELOC boots up, BT times out after 60 sec -> Reboot
  Once i manually restart the ELOC with the On Off switch. The reboot loop dissapears and all is back to normal
• I had the same reboot loop once a few weeks ago. And today again. Using this commit d69080d
• Log (With backtrace): Guru Reboot Loop.txt

EDIT:
The buzzer wakes up the acelerometer every time it buzzes. So when 60 sec time out happens, the buzzer wakes up the knock sensor at the same time whcih also wakes up BT again.
Only on this specific PCB the knock sensor wakes up from the buzzer. But maybe that is the reason for the reboot loop?
Will play around with the accelerometer sensitivity to find a better value.

[OOHehir]

@EDsteve @LIFsCode I just had a quick look at this log & perhaps its a memory issue also? There seems to be lots occuring simultaneously; file rotation, ArduinoJSON memory pool, file writing & a semaphore queue being established. That's just an educated guess from first reading though..

[EDsteve]

@LIFsCode @OOHehir

Another Guru reboot.
Do more log files help you to figure out where the problem comes from? Or should i stop uploading them?

Guru Reboot after interactin with BT

• How it happens: ELOC was in recOn_detectON mode for >24 hours, and after one of my rabdom BT-test-connections it rebooted
• Commit: d69080d
• Config: stamptest1706795112761.txt
• Log (With backtrace): Guru.txt

[EDsteve]

From another device i am getting BT errors. But no reboot and still working normal. Not sure if that is connected to the reboots. But i will post it anyway in case it helps:
E (133702274) BT_OSI: config_save, err_code: 0x4

BT Error in log

• How it happened: ELOC was in recOn_detectON mode for >24 hours, and after one of my random BT-test-connections this error appeares now every time BT turns on or off. But the ELOC app can connect without problems.
• Commit: d69080d
• Here the log BT-Error.txt

[OOHehir]

@LIFsCode @OOHehir

Another Guru reboot.

Do more log files help you to figure out where the problem comes from? Or should i stop uploading them?

Guru Reboot after interactin with BT

• How it happens: ELOC was in recOn_detectON mode for >24 hours, and after one of my rabdom BT-test-connections it rebooted

• Commit: d69080d

• Config: stamptest1706795112761.txt

• Log (With backtrace): Guru.txt

This seems a bit different:

Core 0 panic'ed (Interrupt wdt timeout on CPU0)

[OOHehir]

From another device i am getting BT errors. But no reboot and still working normal. Not sure if that is connected to the reboots. But i will post it anyway in case it helps: E (133702274) BT_OSI: config_save, err_code: 0x4

BT Error in log

* How it happened: ELOC was in recOn_detectON mode for >24 hours, and after one of my random BT-test-connections this error appeares now every time BT turns on or off. But the ELOC app can connect without problems.

* Commit: [d69080d](https://github.com/LIFsCode/ELOC-3.0/commit/d69080d9ce6eb05b3a9967dc8f74919bccfcbef0)

* Here the log [BT-Error.txt](https://github.com/LIFsCode/ELOC-3.0/files/14143010/BT-Error.txt)

I don't see this error code in the documentation but perhaps a wrapper for this?

[LIFsCode]

@LIFsCode @OOHehir

Another Guru reboot. Do more log files help you to figure out where the problem comes from? Or should i stop uploading them?

Guru Reboot after interactin with BT

* How it happens: ELOC was in recOn_detectON mode for >24 hours, and after one of my rabdom BT-test-connections it rebooted

* Commit: [d69080d](https://github.com/LIFsCode/ELOC-3.0/commit/d69080d9ce6eb05b3a9967dc8f74919bccfcbef0)

* Config: [stamptest1706795112761.txt](https://github.com/LIFsCode/ELOC-3.0/files/14142906/stamptest1706795112761.txt)

* Log (With backtrace): [Guru.txt](https://github.com/LIFsCode/ELOC-3.0/files/14142820/Guru.after.SD.card.check.txt)

Ok this is a different topic:
The Core0 is stucked in an ISR (Interrupt Service Routine).

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0). 

Core  0 register dump:
PC      : 0x4000bfe7  PS      : 0x00060835  A0      : 0x80085305  A1      : 0x3ffbe5d0
A2      : 0x00000001  A3      : 0x00060823  A4      : 0xfffffff0  A5      : 0x00060023
A6      : 0x00000000  A7      : 0x3ffdf580  A8      : 0x00000008  A9      : 0x3ffbe4e0  
A10     : 0x00000004  A11     : 0x00060024  A12     : 0x00060023  A13     : 0x00060023
A14     : 0x00060023  A15     : 0x00000001  SAR     : 0x0000001d  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x400928d8  LEND    : 0x400928f4  LCOUNT  : 0xffffffff
Core  0 was running in ISR context:
EPC1    : 0x401779a3  EPC2    : 0x400869c6  EPC3    : 0x400869b1  EPC4    : 0x400869b1

The only thing I could image causing this, is the knock sensor, which has an interrupt pin to wake the ELOC.
@EDsteve are you knocking on the ELOC? I think maybe the the knock sensor is too sensitive.
The interrupt is edge triggered so I assume the pin is toggling like crazy, but I haven't seen it here, so I can't tell for sure.

@EDsteve: if that happens again, could you please measure the "Accel_Trigger" signal on the PCB

[LIFsCode]

Ok, I found something else, which might give another explanation to the WDT timeout:
https://esp32.com/viewtopic.php?t=33221

Unfortunatelly I could reproduce the reboot only a single time:
WDStuck.txt

Test: Check BT core pinning

This might be related to the above discribed issue, though I'm not sure.
I will move the BT core to Core 1 and and have to ask @EDsteve to rerun the tests. Sorry it's frustrating, but I don't have any better idea.

Test: Check effects of sd card logging

@EDsteve : Do you also have these issues when the SD card logging is disabled? Since this adds a lot of file system handling even to integrated functions (as bluetooth, etc.) this might cause troubles.

I splittet optimizing the sd card logs in another issue this will be a bit more work, not sure if I manage to do it before the next deadline. If disabling the sd card log solved these issues, than that would be at least a workaround