ESP32-S2 linux crash on circuitpython reboot

[emard]

SMP Debian 5.7.10-1 (2020-07-26) x86_64 GNU/Linux
crashes often when using ESP32-S2 and it's starting to become annoying.
Kernel just freezes, I can't see nothing like register dump in syslog to see
in which function and source line crash happened. I can reproduce crash
only in gnome environment which has nice file manager that mouns S2 disk.
If I boot only to textual console, linux won't crash then.

Normal workflow that triggers crash every cca 10-20 retries is this:

cp demo.py /media/user/1234-5678/
screen /dev/ttyACM0

import demo
... it uses SPI and bitbang IO ...
Ctrl-D
editing demo.py and cp again and linux freezes, only reset helps.

The easiest way to provoke crash is this, it's repeatable and aways the same
syntax errors and crash to otherwise correct syntax:

import gc; gc.collect(); gc.mem_free()
63792
import gc; gc.collect(); gc.mem_free()
Traceback (most recent call last):
File "", line 1
SyntaxError: invalid syntax
import gc; gc.collect(); gc.mem_free()
63536
import gc; gc.collect(); gc.mem_free()
Traceback (most recent call last):
File "", line 1
SyntaxError: invalid syntax
import gc; gc.collect(); gc.mem_free()
63536
circuitpython crashed and sometimes follows by linux crash also

I know this is not directly to blaim circuitpython but
for circuitpython to be user-friendly, adafruit developers would be
the best in coordinating with linux developers to locate this bug and help
them fixing it sooner.

[tannewt]

@dhalbert Is this the same issue you were seeing on other ports?

I'm running Linux dalinar 5.8.3-arch1-1 #1 SMP PREEMPT Fri, 21 Aug 2020 16:54:16 +0000 x86_64 GNU/Linux and haven't had an issue.

@emard Are you using an intel based system?

[emard]

Yes it's intel based system, and circuitpython is connected
to port of USB HUB (I tried different ports, makes no difference)

Bus 001 Device 005: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 122
model name : Intel(R) Celeron(R) N4000 CPU @ 1.10GHz
stepping : 1
microcode : 0x32

[deshipu]

Not sure if that helps, but I stopped having those problems with SAMD21 after downgrading my kernel to 5.4.0-42 — not sure if you can test with a different kernel, but that could help confirming it's the same bug.

[dhalbert]

This sounds similar to #3268, which occurs on other platforms. See the links there to the linux-usb mailing list and to the Launchpad bug report.

There may be multiple crash causes. I fixed one, which is that storage.erase_filesystem() would frequently crash the Linux host. I was told that Linux would not guard against strange behavior by the USB filesystem.

People have also seen crashes on disconnect (e.g., pulling the USB plug). It's not clear whether this is filesystem-based, or due to other USB devices disappearing abruptly. I think this behavior is less excusable on the kernel's part, because anyone could pull a USB drive or device out at any time.

@deshipu I tried multiple kernels for the storage.erase_filesystem() case, way back to 4.x and forward to the latest 5.7 or so. They didn't help. But if there is another cause, you could be on to something.

I have ameliorated these crashes to some extent, but not completely eliminated them, by interposing a cheap USB2 hub.

[deshipu]

@dhalbert in my case it was a crash on disconnect or hard reset

[emard]

Is there some option to prevent USB input devices from being
enumerated at ESP32-S2 (keyboard, mouse, consumer control.
and something input generic)

They appear by default even though no user's python code
is handling them.

[deshipu]

I'm sure I used that option in my PewPew boards to disable midi and hid, but it seems that it's not there anymore...

[deshipu]

It used to be USB_DEVICES = "CDC,MSC" in mpconfigboard.mk.

[dhalbert]

See supervisor/supervisor.mk for how USB_DEVICES is set. It is now computed based on whether various modules are enabled or disabled.

USB_DEVICES = "CDC,MSC,AUDIO,HID" was the full list.

You can adjust which HID devices are present as well. The full list is:
USB_HID_DEVICES = "KEYBOARD,MOUSE,CONSUMER,GAMEPAD"

The USB devices are enumerated after boot.py runs, but before code.py or the REPL runs. It can't be done completely on the fly without doing a USB reset; you can't add devices after an enumeration without disconnecting and reconnecting. Issue #1015 talks about allowing dynamic USB descriptors, making them settable in boot.py.

[dhalbert]

EDIT to previous comment: "now", not "not"

[tannewt]

Do we believe we can fix this? Or should we make it a long term issue? Is it still an issue?

[emard]

Long term ugly bug! I pull images for WROVER-S2 on daily basis and this crash is still here. If I develop something, 1 linux crash per day is minimum :) If I just try few commands if it works, no crash.

…

On 9/30/20, Scott Shawcroft ***@***.***> wrote: Do we believe we can fix this? Or should we make it a long term issue? Is it still an issue? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #3336 (comment)

[BennyE]

I found a reliable way to crash my Pi4:

(Save your valuable work before trying this!)

• Get https://circuitpython.org/libraries py-bundle (I used https://github.com/adafruit/Adafruit_CircuitPython_Bundle/releases/download/20201107/adafruit-circuitpython-bundle-py-20201107.zip)
• Upload "adafruit_binascii.py" to your featherS2 (I chose to put it to the root directory directly)
• Go to REPL and "import adafruit_binascii"
• Try to adafruit_binascii.TAB-key or "print(dir(adafruit_binascii))"
• Note that it doesn't work, if you try now to do another time "import adafruit_binascii" you'd reliably crash your host device (I use Pi mostly for development & sensor related stuff)

I hope this helps!

[dhalbert]

I'm going to mark this as a dupe of #3268. It is a Linux issue which they don't want to fix. We can try to ameliorate it by perhaps inserting some delays before switching from one filesystem to another, but it also occurs when a filesystem disappears, it seems.