SAFE_MODE resets on esp32-s3

[ehagerty]

CircuitPython version

Adafruit CircuitPython 8.1.0-beta.0 on 2023-03-01; ESP32-S3-DevKitC-1-N32R8 with ESP32S3
Board ID:espressif_esp32s3_devkitc_1_n32r8
UID:866B3B736F82
boot.py output:

Code/REPL

import microcontroller
microcontroller.on_next_reset(microcontroller.RunMode.NORMAL)
...wifi DHCP lease timeout or similar...

Behavior

always forces a reboot to SAFE_MODE, not NORMAL (or otherwise)

Description

any cause for a reset, whether programatically or because in our case of normal but regular DHCP lease timeouts, resets and sends you to SAFE_MODE which for production devices is essentially a dead device. No amount of logging is capturing the initial cause of the hang at present but even if it did, looking at supervisor.c I think we'd still end up stuck in SAFE_MODE? We have several hundred customers we were about to shift from micropython on the esp32 to circuitpython on the esp32-s3 which are stalled behind finding a way to resolve this issue, so anything I can do to help please just shout. I have endeavoured to read and understand the six issues I have referenced here and was unable to determine what the suggested course of resolution is.

Additional information

Referencing similar issues fixed in the 7x branch but seem not ported to 8x:
#3988
#3884
#6502

Referencing this open and related bugs:
#6791
#7405
#7291

[tannewt]

Are you using the watchdog to reset? Have you seen safemode.py?

[ehagerty]

Hi, thanks for this. I do use the watchdog but this error occurs unexpectedly at as yet unpredictable times and drops me into safe mode before the watchdog times out. Perhaps I am not understanding your query though?  Thank you for pointing me to safemode.py I will look at whether that can help us in the meantime.  Do you need more data from me to help diagnose?On 20 Mar 2023, at 21:37, Scott Shawcroft ***@***.***> wrote: Are you using the watchdog to reset? Have you seen safemode.py? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[tannewt]

on_next_reset isn't meant to prevent resetting to safe mode. It's really meant to be used with a user initiated reset.

I don't know what the bug is here. I think your best bet is to use safemode.py to recover.

[ehagerty]

Forgive me @tannewt for not being clear enough. Ignore the watchdog or my code in this discussion as it will only cloud things - I was just trying to reply to your query. Regardless of the code on the device, we see occasional hangs, resulting in a hard reset to safe mode. Based on reading all the open issues related to the esp32-s3 overnight, perhaps what we see is related to the known S3 wifi issues?

I have been able to use the safemode.py approach for now (although I'd be keen to know how you suggest logging data from that stage given that the documentation suggests there is no way to do so but you must be using something yourself doing core dev?)

I'd be very happy to try to help if I can - I've personally been writing micropython for the past 3 years, and started my career with C 35 years ago. I apologise if I'm misunderstanding correct etiquette here. We greatly appreciate you responding to us!

[anecdata]

@ehagerty you can write to flash (or nvm) from safemode.py (example at https://gist.github.com/anecdata/fe35dc6a94069fc920edf61a64750b53)

[dhalbert]

You should be able to write to the filesystem after remounting it as well. safemode.py does not do that automatically itself in case there is some problem that might prevent that.

[microdev1]

Hi, @ehagerty. If the device is entering safe mode due to a Hard Fault crash then the best way to debug this would be to flash a debug build make BOARD=... DEBUG=1 and monitor the uart output. The board won't enter safe mode but would instead print a backtrace which can be decoded using ports/espressif/tools/decode_backtrace.py.

[ehagerty]

@anecdata thank you so much, the gist is brilliant! @dhalbert Thank you very much for your explanation, I'm very grateful for your time. @microdev1 thank you I had not yet crossed the 'build completely myself' threshold with CP as I had done with MP - I'm very grateful for your explanation to get me started!

[tannewt]

Here is the guide for how to build CircuitPython.

[bill88t]

On my Feather ESP32-S3 TFT, I can import microcontroller; microcontroller.reset() just fine. It reboots to normal mode.

[jepler]

It sounds like there may still be an underlying issue causing the board to enter safe mode (e.g., hard fault, assertion failure, etc) that can be entered as its own issue. However, the misunderstanding about what "on next reset" means has been cleared up, and safemode.py has provided an adequate way to recover, so closing this one.