Fake filesystem or a pseudo-file for status information

[deshipu]

This is a loose idea for now, but I will dump what we have so far, so it can be further discussed.
The crux is to have either a pseudo-file, or a whole fake filesystem on the CircuitPython's USB MSC disk, to make available dynamically-generated status information from the running CircuitPython.

It would replace the boot_out.txt file, but because it's not physically written to the flash and does not wear the flash storage or consume extra space on the device, it could contain a variety of additional information, such as the last exception traceback, the current IP address of the device, a URL for the web interface, even things like the contents of the NVRAM or amount of free memory.

Personally, I'm particularly interested in putting the exception traceback in it, because that would greatly ease the support and debugging, without having to teach people how to use the serial console. We could simply ask them to upload the file, and we would have all the information we need for debugging: the exact version and the exception.

There are several ways we could go about implementing it. With the pseudo-file approach, we could have a file created, that points to some non-existent or reserved blocks for its content, and then have a special case for retrieving the contents of those blocks, generating them dynamically instead of reading them from the flash. With the filesystem approach, we could have a second partition in the partition table, and a whole fake filesystem, possibly with different files containing different pieces of information. In either case, the data would be read-only and dynamically generated.

Editors like Mu or Thonny, and plugins for IDEs could also check those files for status, errors and such.

[carson-coder]

I feel like there could be some sort of safety issue with telling people to share a file that has that devices ip.

[deshipu]

It's the internal IP, inside the NAT, but in the filesystem case, there would be separate files.

Giving your CircuitPython device an externally accessible IP address is already a security nightmare.

[eightycc]

I'm fond of the fake filesystem approach for these reasons:

• Does not depend on flash filesystem integrity. A corrupted or non-existent flash filesystem would not affect its ability to retrieve info from the fake filesystem.
• No complex interaction with existing flash filesystem metadata such as the VBR, FATs, and directories.
• No modifications to oofatfs are necessary to implement the fake filesystem.
• CP code writes to a fake filesystem file are simply stores into a backing RAM buffer.

[dhalbert]

@tannewt pointed out that the host OS will cache the file contents. We need to notify the host that it has changed. This may help: https://stackoverflow.com/questions/6775951/usb-mass-storage-device-side-data-changes-not-visible-on-pc-filesystem

[bill88t]

I like the idea a lot too, but there are some things to consider.

First of all, while it would be nice to have this in ram, a well written app could just implement most if not all of this functionality in boot.py to automatically handle the no-write check, or in another script (and file) and check if a write needs to be performed manually.
Unless there is an error, the ip and stuff would probably be the same every time, so no writes.
The only constantly changing thing is status, but you could say, "if there are no errors, it's running".
boot.py handles board info pretty well since auto-connect & web-workflow have already run, so you got the ip.
If there is an error, it could be caught in a catch-all in code.py, and simply appended to a file.

Secondly, if this is implemented, it should probably live after a vm exit, since it may have captured a code.py crash.
Which means dynamic buffers outside of vm..
It cannot be static since recursive or massive apps could have waaay more than a few hundred function calls.
Having a massive alloc from the get go isn't viable for small boards.

Also, sidenote. If this works can't we do a similar hack to expose filesystem mounts (like sdcards) over usb-msc?
If we could get other filesystems to show over msc, we could use a ramdisk for all of that.

The ip hardly ever changes, and catastrophic errors are worth the flash writes.

[anecdata]

I also like the idea of collecting traceback and potentially other debug info, particularly when the USB serial console is unavailable (battery projects, etc.) I think it should be non-volatile though, rather than RAM, so it persists across safe modes, or soft or hard resets, or unexpected losses of power.

Related to #8704, #7490, and #1054.

We have nvm (flash) and alarm.sleep_memory (RAM) for relatively small amounts of data, though as bytes only. Note that alarm.sleep_memory is lost on soft reset on the espressif port, but is retained on soft reset on the raspberrypi port.

We have safemode.py that can collect some level of information after a safe mode and before boot.py, boot.py that can collect some level of information after a reset and before code.py, and supervisor.get_previous_traceback() to capture exceptions (though not persistent after a reset). And we have reasons for occurrences of safe mode, reset, and reload (also somewhat transient).

One of the things we don't have at all is information on what was executing last when there is a safe mode, reload, or reset, but no exception. It's particularly challenging for most users to store debug info from code.py since it requires the storage.remount() dance (or an SD card).

[deshipu]

@anecdata it doesn't make much sense to make it non-volatile when it's going to be overwritten on the next boot anyways. And you run into flash wearing issues.

[bill88t]

Perhaps it would be interesting to have a rotating log-styled boot_out.py, which could store like the last 4 executions of whatever, along with tracebacks.

If the boots are identical and nothing errored out, no flash writes would occur.

But that would require A LOT of code.

[carson-coder]

perhaps there could be a setting to write all log outputs to a sd card so that the on board flash doesn't get worn down and it could be easier to check on logs that way