"could not complete raw paste" on SeeedStudio XIAO-ESP32C3

[aivarannamaa]

CircuitPython version

Adafruit CircuitPython 9.0.0-alpha.5 on 2023-11-15; Seeed Studio XIAO ESP32C3 with ESP32-C3FN4

also 8.2.8 and 8.1.0

Code/REPL

> mpremote cp data.json :

Behavior

Similar issue was discussed in the context of Thonny (#7158), but the problem is reproducible also with mpremote (version 1.21.0).

Tried CP 8.2.8 and 9.0.0a5 on macOS Sonoma, Windows 11 and Ubuntu 23.10, also tried CP 8.1.0 on Ubuntu. All these tests failed in similar way -- mpremote would start writing the file but after a while wouldn't get raw paste confirmation from the device (it times out while trying to read a single byte).

I tried to upload this file: https://raw.githubusercontent.com/thonny/thonny/397679cd8f7eb11b20ba9075c694337f5aace54b/data/circuitpython-variants-esptool.json

When I cancelled the wait I got tracebacks like this (the first line shows the mpremote command line I used):

% mpremote cp data.json :
cp data.json :
 ... copying   6% [#-------------------]^CTraceback (most recent call last):
  File "/Users/aa/python_stuff/thonny/venv312/bin/mpremote", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/main.py", line 528, in main
    handler_func(state, args)
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/commands.py", line 159, in do_filesystem
    state.transport.filesystem_command(
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 476, in filesystem_command
    op(src2, dest2, progress_callback=progress_callback)
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 420, in fs_put
    self.exec("w(" + repr(data) + ")")
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 285, in exec
    ret, ret_err = self.exec_raw(command, data_consumer=data_consumer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 271, in exec_raw
    self.exec_raw_no_follow(command)
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 249, in exec_raw_no_follow
    return self.raw_paste_write(command_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 225, in raw_paste_write
    data = self.read_until(1, b"\x04")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/mpremote/transport_serial.py", line 118, in read_until
    data = self.serial.read(min_num_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aa/python_stuff/thonny/venv312/lib/python3.12/site-packages/serial/serialposix.py", line 565, in read
    ready, _, _ = select.select([self.fd, self.pipe_abort_read_r], [], [], timeout.time_left())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Description

No response

Additional information

• mpremote can transfer the file without problems if I'm using MicroPython 1.21 (https://micropython.org/download/ESP32_GENERIC_C3) on the same device.
• Similar problem has been reported by Thonny users also for Wemos Lolin C3 Mini with ESP32-C3FH4 (CircuitPython upload file fail thonny/thonny#2930)

[aivarannamaa]

The same device (SeeedStudio XIAO-ESP32C3) is having problems also when trying to write files via the plain raw-REPL mode or via plain paste mode. In paste mode the echo starts going slightly off after a while, in raw mode the OK confirmation does not appear.

Nevertheless, I don't think the problem is specific to writing files. I changed the file writing procedure to a no-op, but made sure that the amount of serial communication stayed the same. The problem still persisted.

[bill88t]

Oh, the ESP32-Cx serial implementation is at fault here, no doubt about that.
The same flush desync/incomplete bug from what I can tell.

[aivarannamaa]

The problem persists in CP 9.0b0.

I wonder why is this issue in the "Long term" milestone? Is it too hard to be handled sooner or is it considered low priority?

[tannewt]

I wonder why is this issue in the "Long term" milestone? Is it too hard to be handled sooner or is it considered low priority?

Thanks for retesting! It's a low priority because we only have one C3 board from Adafruit. It is hard to know difficulty until after it has been debugged and fixed.

[berkut0]

I have a cheap board from the Chinese market and couple my own with mini modules.

Can confirm that 8.2.10 is installable and usable, but any 9+ versions have serious problems to work on ESP32-C3 (or just with Thonny, not completly sure)

I can describe the problem this way: by selecting different boards you can get about three states. Or it works and you can write to the console, but it hangs and reboots. Or it does not connect through Thonny. Or it is not recognised by the system at all as a normal device.

update: it is almost impossible to upload files on board with any version of CP:

adafruit-circuitpython-bundle-8.x-mpy-20231207\lib\adafruit_display_text_init_.mpy (4850 bytes)
PROBLEM IN THONNY'S BACK-END: Exception while handling 'upload' (thonny.plugins.micropython.mp_back.ProtocolError: Could not complete raw paste).
See Thonny's backend.log for more info.
You may need to press "Stop/Restart" or hard-reset your CircuitPython device and try again.

[RetiredWizard]

This is not a solution by any means but if you're keen on getting something working.... I have been able to use these board by uploading files using a serial transfer program similar to the way mpremote works. It's a little unfinished as I got it working "well enough" and left it at that. The UI is a bit weird but once you get used to it, works ok.

https://github.com/RetiredWizard/Serial-File-Manager

[SeanTheITGuy]

I have used ampy and the web file.manager without issue on c3 devices. With 8.2 through 9.1-beta.

[RetiredWizard]

Nice, I knew about ampy (https://learn.adafruit.com/micropython-basics-load-files-and-run-code/install-ampy) once but completely forgot about it.

[UnexpectedMaker]

I've also been experiencing this on my TinyC6 and had to use a modified version of the Serial-File-Manager by @RetiredWizard to be able to communicate with my C6 boards running CP when under test on my test jigs.

It would be great if we could work out what this is as it seems to be specific to the Espressif serial UART setup in their C3/C6 chips, so will likely effect the H2, C61 and C5.

[berkut0]

There is a non-zero chance that @aivarannamaa can fix something in Thonny and it will work. But it would be good to understand exactly what the problem is. There seem to be solutions that work. (But I suspect that the problem lies with Ampy itself)

I like the web file manager solution, thanks a lot for the tip, I'll give it a try

[RetiredWizard]

Since the C3 was on my mind, I pulled one out for a project I was working on yesterday. I did use ampy to get the initial settings.toml file loaded, but once it was connected to WiFi, I used the Web Workflow for everything. The File Browser to transfer files, the Internet based full code editor and associated internet based serial interface (not the CP hosted serital console) and everything went very smoothly. I was even able to use the Web Workflow feature of Circup to check and update the libraries on the device. It was an enjoyable workflow to use although I don't believe, other than Circup for the libraries the Web Workflow can be used to transfer files via a script (yet?).

Using Web Workflow is definitely another option to consider until the current serial UART isses are resolved.