Change main filename to run next

[tannewt]

Add the ability to set the next main filename before triggering a reload. This can be used to switch between games on a system or between memory heavy animation scripts. https://forums.adafruit.com/viewtopic.php?f=60&t=139041

[kevinjwalters]

My similar discussion: https://forums.adafruit.com/viewtopic.php?f=60&t=150016 (#737 gets mentioned in that).

[robertgallup]

Mentioned this awhile ago on Discord, but thought I'd chime in here.

I'm working on a tutorial and have a bunch of example scripts all stored on the CIRCUITPY drive. Switching between them requires manually renaming or copying files to code.py/main.py. This gets tedious so I wrote a menu script that lists the available other scripts on the CIRCUITPY drive. Selecting one runs it using Import. This works, in general. The problem is that larger scripts can overflow the heap when imported even though they work fine on their own.

There might be a more general solution, but for me, it would be great to have something like sys.exit('next.py') that would restart the vm with the named script. No need for this to disrupt any existing behavior. e.g. ctrl-D from the REPL and sys.exit() could still load code.py/main.py by default.

The way this would work for my case is that the menu would be code.py/main.py and use sys.exit('next.py') to run the chosen script. All the example scripts would act normally, you'd use ctrl-C,ctrl-D to restart the menu, no special hooks to get back to the menu. Of course, if you wanted to chain several scripts, each one could use sys.exit('next.py').

[deshipu]

I have exactly the same use case for my game consoles.

[dhalbert]

We would not enhance sys.exit() for that purpose, because we aim to be a subset or the same as the CPython version. But we could certainly add it to supervisor, such as adding an arg to supervisor.reload().

[robertgallup]

Adding it to supervisor.reload() would be perfect.

[tannewt]

Note that the hardest part of this is storing the filename because the heap gets reinitialized before running the file.

[robertgallup]

Good point. Is this same issue handled in supervisor.set_next_stack_limit()? Or, is it a different problem?

[tannewt]

Ya, it is similar but stack size is a fixed size so it's easy to just statically allocate it. We could simply have a char* array with a limit to start. If folks hit the limit often we could either increase it or make the allocation dynamic.

[robertgallup]

Ah ha. I see the issue. Picking a good limit is key (though, no matter what you pick, someone will have a use case for more :) ).

[dhalbert]

If the heap memory is not touched when the VM shuts down, then I think the following would work. It relies on being to able to use the stale data in the old heap.

1. Copy the location of the string object that is the filename into a pointer variable at a fixed location
2. Reload exception throws us back to somewhere in main.
3. VM gets shut down. Loop back to restart VM
4. Before heap is re-initialized (is it zeroed?), examine the string object pointed to. Allocate an array on the stack large enough to hold the filename, and copy the filename to there.
5. Initialize the heap.
6. Pass the filename on the stack to run_code_py().

[deshipu]

I just realized there is one more piece of the puzzle missing — after the code finishes running, it would be best if the device restarted again, with the default main.py this time, instead of dropping into the REPL. I wonder if this is too much to ask for an obscure use case, though.

[dhalbert]

@deshipu If each of the alternate programs did a supervisor.reload() with no args when it's done, that would just happen. Are you thinking that is an extra piece of code that the alternative programs wouldn't have?

[deshipu]

Yes, ideally it would be the same kind of programs that you normally run as code.py — whatever the users have written or copied from examples. Requiring a reset at the end is a possible solution, but I'm sure it will often be missed and users will wonder why the device is not displaying the menu again after running their code.

Right now, when we run the programs by importing them, this is not a problem, as the control returns to the menu program after the import. But if we switched to this new way, we would probably want some way of resetting the device afterwards. Granted, most of those programs will be infinite loops, and the users will simply press reset to get back to the menu.

[dhalbert]

[started to reply and then saw your edited reply]

It would be possible to set a flag that says to reload on exit, either via supervisor.reload() or some new function. But as I was going to mention, and you already did, most example programs already are infinite loops and have no exit mechanism, so they'd have to be modified anyway.

[deshipu]

Also, you probably don't want to restart if there was an error, only if it was a normal end of the program. I might be overthinking this.

[tannewt]

Also, you probably don't want to restart if there was an error, only if it was a normal end of the program. I might be overthinking this.

I don't think you are overcomplicating it. I think it is complicated.

Another aspect is what file should be run on reload. We don't know what file has been changed when the FS is written. My guess is that we want the currently running file reloaded (and therefore we need to preserve the filename past start up into the new heap.)

A related use to this would be doing OTA file updates over wifi. We'll want a known good code.py to allow for writing new files even when the loaded file crashes.

[robertgallup]

So, I might be misunderstanding, but are you suggesting that supervisor.reload('file_name') would permanently override code.py/main.py as the file for reload? Or, are you talking about behavior on the next reload after an error? Or, neither of those...

[tannewt]

I think we'll want a separate function rather than putting it in reload. That way the code can set if for all reload causes, not just manual reload. Something closer to set_next_stack_limit: https://circuitpython.readthedocs.io/en/5.3.x/shared-bindings/supervisor/__init__.html#supervisor.set_next_stack_limit

Then we can have multiple kwargs for different reload reasons or separate functions. I imagine you want: reload, exception, safe mode and completion as reasons.

[robertgallup]

This sounds like a good general approach. As I envision my current use case, I just need to reload with a specific file without affecting any of the defaults. Though, with more options, my use case might evolve.

[deshipu]

After some thought, I think that just setting the file name for the next run would solve 90% of my case as well — the users always can reset the device to get back to the menu, and an "exit the game" option can be provided by the library, which would then do the correct thing.

[siddacious]

My loader for the OSHWA 2020 badge would appreciate this functionality. I assume that since the proposed mechanism is simply designating which file to run, there would be no need to make sure pins are released/de-initialized?

As you suggested previously @tannewt , this will mean we (I ) will need to handle the "don't reinit already initialized stuff" case for libraries. Since the libraries will need to know the 'commanded reload' vs 'default start' state, it would/could be exposed bysupervisor or similar?

[tannewt]

@siddacious I was thinking it would be a full reload that would reset everything.

What is an example of things you don't want to reset?

[robertgallup]

For my use case, I think a full reload/reset would be optimal.

[siddacious]

@tannewt I was referring to our previous convo where you suggested thinking about how sensors libs should handle a restart, though in that context I think you were talking about waking up from a sleep that was entered for power saving purposes so I may have crossed the streams.

It sounds like this is a related but different issue as there are certainly use cases where you would want a complete reset, sensors included ie: running an app that wants different configurations for a sensor.

[robertgallup]

One thing that just occurred to me is that, if possible, it might be useful to support .mpy files in this function. Currently, .mpy isn't supported during a board restart.

[dhalbert]

@robertgallup I am not sure what you mean. Do you mean you can't have code.mpy?

If something.py and something.mpy both exist, then something.py will take precedence.

[tannewt]

When it turns out that some of these options are never used, always used with the same value, or always used together, we can then remove or combine them. My current expectation is that reload_on_success will stay, sticky_on_reload (the controversial one from the last few posts) will stay if we can’t agree on a default value for it, and the rest can go.

This is fine for a prototype here but I wouldn't want to merge it in the "maximally configurable" state because removing APIs is much harder than adding more. For the PR, please do the limited API needed for the game case.

I notice we are at 6.0.0-beta.0 now. What are the chances of getting this into 6.0? I have no particular preference, just to set expectations. There is one commit waiting at https://github.com/python-ugame/circuitpython-stage already (submodule frozen/circuitpython-stage) that should go into 6.0 in my opinion. Once this lands, there will be more.

We only need to worry about 6.0.0 if we want to remove/rename APIs. Adding APIs can be done after 6.0.0. (Removing/renaming would have to wait for 7.0.0)

I think the PR is blocked on getting an adequate but minimal API set, implemented and documented.

[cwalther]

Agreed. The “maximally configurable” state was never meant to be merged as is, it’s just a prototype for people to play with to figure out what we need. I was making progress on the “retrieve last exception” front when the supervisor heap bug distracted me, so as far as I’m concerned, we’re closing in on that.

[tannewt]

@cwalther I also realized this approach here is very similar to what we'd need for setting the USB descriptor from boot.py. #1015

[cwalther]

I’m unfamiliar with USB and don’t know what a descriptor is, so I can’t comment in detail, but one thing to note is that the more pieces of information we save using this approach, the more carefully we need to think about their interaction. If one needs to be cleared and another one is newly set and a third one needs to be preserved, and not all of this is known at the same time, in what order do we do what so we don't leave holes and waste memory? I‘m actually working on planning that through between “next code“ and “traceback” right now.

[cwalther]

I think I have a generic solution that will work for any number of items (as long as they all fit on the stack at once). Implementing and testing it will probably take me a few days though.

[deshipu]

I can imagine that the memory exceptions are going to be particularly difficult to handle. especially when they happen with a small allocation and there is little memory left to process it.

[tannewt]

@cwalther Sorry to distract on this. A USB descriptor defines what type of a USB device something is such as keyboard vs audio. It's similar to this because we'd construct it in boot.py and save it as a byte string. USB is initialized after boot.py and then lives through all subsequent code.py. So, it's a similar case of save a byte string from the VM and then move it into supervisor memory to init TinyUSB with. It shouldn't add complexity because it won't change after boot.py.

[cwalther]

OK, just so I get a rough idea to keep in the back of my mind while I implement the generic system, to make sure it will be ready for it:

• Does the supervisor allocation containing the USB descriptor have to live forever (until poweroff/hard reset), or can it be freed after TinyUSB initialization (i.e. before the first code.py run)?
• How many bytes roughly is a typical USB descriptor?
• What happens when the Python functions that set up the USB descriptor in boot.py are called in code.py? Can that happen?

[tannewt]

Does the supervisor allocation containing the USB descriptor have to live forever (until poweroff/hard reset), or can it be freed after TinyUSB initialization (i.e. before the first code.py run)?

I think it needs to live forever because the host could ask for it at any time.

How many bytes roughly is a typical USB descriptor?

If I had to guess it's 500 - 1000.

What happens when the Python functions that set up the USB descriptor in boot.py are called in code.py? Can that happen?

I would just have them raise an exception. But maybe it'd be ok if the USB is unconnected.

[cwalther]

In other news, taking a closer look at supervisor_move_memory(), I see that what it does with terminal_tilegrid_tiles is of the same sort (saving stuff from the GC heap in supervisor allocations) and should be handled by the same mechanism. It could potentially be too large to fit on the stack though. Maybe what it does with the rgbmatrix and sharpdisplay buffers too, although those don’t need their contents copied. A revised system that can deal with those cases is taking shape in my head. I may be able to replace these three ad-hoc “hybrid allocators” with a central solution.

[cwalther]

Quick status update: Iteration 3 of the generic movable allocation system, built on top of the existing supervisor heap, works (as far as tested with the terminal tilegrid, more thorough tests still to be done), but adds a fair bit of code. Now working on iteration 4 to see if redesigning the supervisor heap to intrinsically support movable allocations will achieve the same goals with less code.
I hope nobody is waiting for me, cause I’m having fun down here in this rabbit hole! 🙂

[tannewt]

Sounds good @cwalther. Don't lose sight of how much added code is too much. Running the CI will build all the boards and tell you if any are out of space.

[dglaude]

I am wondering how does this interact with deep-sleep.
Could a program on the MagTag go to sleep but prepare what program to run at next wakeup?
I am still searching for a way to save state from sleep to wakeup.

[cwalther]

In the current implementation, no, it’s all stored in RAM, which is powered down during deep sleep. Moving it to any kind of non-volatile memory (I’m not familiar with what the ESP32-S2 offers in that regard) would be a new requirement.

There will probably be other things to discuss though regarding interactions with deep sleep, as I discovered today while fixing merge conflicts.

[hierophect]

Resolved (as described originally) by #3454. Deep sleep same-file restart support is not in that PR but should be added soon.

[cwalther]

@deshipu or anyone interested in sample code: Now that the prerequisites (supervisor.set_next_code_file() and supervisor.get_previous_traceback()) have landed, here is my work in progress on a menu for the PewPew M4 that makes use of them: cwalther/game-m4-menu:supervisor (diff). I am not currently working on this, so if anyone wants to take it and run with it, feel free.

The error dialog part is still mostly mockup. Here is the program I’ve been testing it with.

It‘s currently hampered by #5057.