REPL Access is Lost After Awhile

[robertgallup]

On the Circuit Playground Express, after running a CircuitPython program, access to the REPL is lost. Ctrl+C stops execution and the serial connection remains, but there's no way to regain the REPL prompt. Sometimes restarting the serial console (on the PC side) unlocks the REPL, or resetting the CPX board.

[dhalbert]

Could you give more details?

What version of CircuitPython are you using?
Do you notice this in general, or with a particular program? If the latter, could you supply the program?
What kind of host computer are you using, and how are you connecting via serial (mu, screen, Tera Term, etc.)?

[FoamyGuy]

I have spoken with this user on Discord a bit as well. I can fill in some of that information.

They are using a CPX with 6.0.0-rc.0 I am not sure about the code they are using.

I did reproduce this issue once this morning though, my device was running simpletest script for a color sensor (that was not connected, was testing for someone else).

I haven't been able to nail down the exact steps to replicate it consistently though. I've switched to no code.py and a simple time.monotonic() printing loop and checked back in a few times but so far I have been able to access the REPL successfully all except for the one time this morning.

In my case:
Host PC was Win7, serial connection was via Mu.

I will try to fish up the color sensor example I had running, though I don't think there is anything "special" about it, to the best of my knowledge it would have been sitting on whatever exception is thrown when the sensor isn't found

Edit: this was the script running (or more likely crashed) on my device at the time I saw this occur: https://github.com/adafruit/Adafruit_CircuitPython_AS7341/blob/main/examples/as7341_simpletest.py

[robertgallup]

The problem exists on CircuitPython v6.0.0-rc0. Did not exist on v5.3.1. It happens with this very simple program:

While True:
    pass

It's occurred using Screen in iTerm on MacOS 10.15.7. Also occurred using the Atom language-circuitpython package (that's where I first saw it).

I'm testing on an itsybitsy M4 right now to see if it's the same.

[FoamyGuy]

I had this happen again with this code running on the CPX:

import time
while True:
    #print(time.monotonic())
    time.sleep(1)

Interestingly, it ran for a long time with the print statement un-commented before this and never got into this state during that time. I'll try to see if I can narrow down whether the printing is really making a difference or if it was just coincidence.

[robertgallup]

So, the problem arises for me on the CPX between 1h 00m and 2h 37m. Will try 1h 45m next.

The missing REPL occurs with the original program:

while True:
    pass

And, I just confirmed the behavior is the same at 2h 37m with this code:

import time
while True:
	print ("Still here!")
	time.sleep(10)

Last part of the output from the later when I pressed CTRL+C was:

Still here!
Still here!
Still here!
Still here!
Still here!
Traceback (most recent call last):
  File "main.py", line 4, in <module>
KeyboardInterrupt:

[robertgallup]

After 3 hours running, my ItsyBitsy M4 has the same problem, the REPL is unresponsive.

Code is:

import time
while True:
	print ("Still here!")
	time.sleep(10)

After CTRL+C final output is:

Still here!
Still here!
Traceback (most recent call last):
  File "main.py", line 4, in <module>
KeyboardInterrupt:

Also, on the ItsyBitsy, after CTRL+C, the dotstar blink pattern is: long-green, long-red, 4 x short-blue.

When I eject the ItsyBitsy, the dotstar returns to solid green.

[dhalbert]

Looking over recent PR's, I am wondering if this has anything to do with #3546, maybe in the overflow handling. Maybe we could use a bisect to check.

[robertgallup]

It looks like #3546 was merged 9 days ago? As a quick guess, maybe try the version just before the #3546 merge and see if it's 'good' or 'bad' and go from there? (are past builds kept anywhere? I'm happy to help pin this down, but I'm not set up to build CircuitPython)

[FoamyGuy]

@robertgallup all of the builds are archived on S3. Here is the link for CPX English US builds: https://adafruit-circuit-python.s3.amazonaws.com/index.html?prefix=bin/circuitplayground_express/en_US/

If you go to circuitpython.org/downloads and find your board then look for the "Browse S3" button it will lead you to the page for that device.

[dhalbert]

https://adafruit-circuit-python.s3.amazonaws.com/bin/circuitplayground_express/en_US/adafruit-circuitpython-circuitplayground_express-en_US-20201013-3d21eec.uf2 is the merge commit for #3546. The one just before that is https://adafruit-circuit-python.s3.amazonaws.com/bin/circuitplayground_express/en_US/adafruit-circuitpython-circuitplayground_express-en_US-20201013-a010dc3.uf2.

If you are willing to do a little testing on these, that would be great! Thanks.

[dhalbert]

The way we'd normally test this is to do a "bisect". We'd pick a known bad and a known good build. Then we pick a build halfway in between, and test that. If it's bad, we then pick another build halfway between the second bad one and the good one, and repeat. Each time we conquer and divide the possibilities in half. Git has a way of doing this automatically. Unfortunately since it takes several hours to determine good/bad, this might get pretty tedious. So I made a stab at a possible place to start.

[robertgallup]

Excellent, thanks! I was just going looking for that build. I'll see what I can find out. I'm using my own 'bisect' to find out the minimum test time (currently between 1h and 2h 30m :-) )

[kevinjwalters]

Is this a duplicate of #2686 perhaps with a change that increases the frequency it occurs at?

[robertgallup]

Perhaps. Was there ever a resolution to that issue? FWIW, my issue doesn't seem to be impacted by amount of serial output.