Add Web Workflow

[tannewt]

Let's add a way to work with CircuitPython over the (local) web!

How it should work

1. First, load http://circuitpython.local:7676 in a web browser. This uses mDNS to do a local domain lookup.
2. All CircuitPython devices on the network will reply with their IP. The first will win.
3. The browser will then request / from that IP.
4. The running webserver will see the circuitpython.local host and will return with a 302 redirect to the device's specific domain which will be cpy-000000.local where 000000 is the last three bytes of the mac address. (mDNS is also used for the cpy-000000.local lookup but only on device will reply.)
5. When the device gets a / request for its unique hostname it will return a small HTML page that loads Javascript from code.circuitpython.org or somewhere else that turns the page into a full editor.
6. The editor can use a REST API or form-compatible upload to read and write files. The serial connection will be available over an insecure "websocket".
7. Each device will also provide a /peers endpoint that will cause it to do a service discovery mDNS request that finds all other CircuitPython devices on the network. The web editor can then provide links to other devices directly.

Getting on WiFi

This method doesn't address how a device gets on the network to begin with. The simplest way is by editing WiFi config settings over USB mass storage. For devices without USB, a webserial based webpage can be provided to write credentials. See how wippersnapper does it.

TODO

• Check in mDNS support.
• Add supervisor.disable_web_workflow() to prevent WiFi from running outside of user code. Depends on CIRCUITPY_WIFI_SSID.
• Add auto connect to configured WiFi. wifi.radio.connect() could be modified to save credentials.
• Stop resetting WiFi completely between VM reloads. We want to keep the serial connection alive.
• Create internal HTTP server to:
  • Redirect circuitpython.local requests to its unique hostname.
  • Reject requests with an Origin that isn't in an allowlist so regular insecure pages cannot access it.
  • Serve web editor stub at /
  • Serve /peers with other device info.
  • Provide REST API for reading and writing files.
• Connect the serial input/output to a socket. Likely 23 to match telnet. Should be usable through an insecure websocket as well.

[ATMakersBill]

Given that these devices can (in some cases must) have their SSID/KEY set and perhaps an IP, I think it would be good to add a way to set the hostname. In the secrets.py file, an optional parameter named mDNS name or hostname or the like would be super helpful.

If that isn't found, then the circuitpython.local would be the default. But this allows a longer-term solution when there are > 1 CP device on the network.

Also, why are we using port 7676? That makes the assumption that Windows Firewall won't care (for outbound requests, that's probably safe), but is there a driving requirement for this?

There are multiple ways to manage getting the name on the network. mDNS is one (UDP multicast over port 5353), but most DHCP servers will also accept hostnames set in the DHCP client request (I believe it's option 81). My router at home allows you to name servers and "pin" the name/IP lease so that the MAC will always get the same values. If we are supporting DHCP, sending the hostname would be helpful (and having it in the secrets.py file would make it easy).

[tannewt]

Given that these devices can (in some cases must) have their SSID/KEY set and perhaps an IP, I think it would be good to add a way to set the hostname. In the secrets.py file, an optional parameter named mDNS name or hostname or the like would be super helpful.

This is possible with wifi.radio.hostname already. The mDNS hostname currently defaults to cpy-###### but we should probably have them align.

#6175 allows for changing the unique name through mdns.Server(wifi.radio).hostname.

If that isn't found, then the circuitpython.local would be the default. But this allows a longer-term solution when there are > 1 CP device on the network.

The proposal above does handle >1 cp devices on the network because they respond to circuitpython.local and cpy-######.local. The latter would be unique.

Also, why are we using port 7676? That makes the assumption that Windows Firewall won't care (for outbound requests, that's probably safe), but is there a driving requirement for this?

We were just brainstorming a port number on Discord. I'm open to suggestions. We wanted to avoid 80 because the user may want to use it for their own code.

There are multiple ways to manage getting the name on the network. mDNS is one (UDP multicast over port 5353), but most DHCP servers will also accept hostnames set in the DHCP client request (I believe it's option 81). My router at home allows you to name servers and "pin" the name/IP lease so that the MAC will always get the same values. If we are supporting DHCP, sending the hostname would be helpful (and having it in the secrets.py file would make it easy).

I believe wifi.radio.hostname is already used for DHCP. It'd be good to align this with mDNS as I said above. I do think that having a selection screen will make it much easier to pick the device you want rather than needing to understand the hostname itself.

[anecdata]

wifi.radio.hostname does make it to the router via DHCP.

The ability to "pin" or reserve DHCP MAC-to-IPv4 address mappings is a convenience offered by most routers, I'm not aware of any standards that cover it.

As I read this, hostname and mDNS hostname may get aligned, but since wifi and mdns would be separate imports, it should hopefully support use cases where a wifi.radio.hostname is desired but not an mDNS hostname.

[prplz]

If you're making wifi persistent, how about a telnet server for the repl too?

[anecdata]

Stop resetting WiFi completely between VM reloads. We want to keep the serial connection alive.

This is desirable for web workflow. However, users are currently in full control of starting and stopping a Station and connecting to an AP, starting and stopping an AP, starting and stopping wifi, etc. There should be a mechanism where user code still retains full control of whether / when / how the device interacts with a wifi network, during code execution and across reloads and resets. In other words, it should still be possible for a CircuitPython device to reload or reset into a mode where it is not interacting with wifi, but code can later connect or do other wifi actions (presumably including mDNS or web workflow).

[tannewt]

If you're making wifi persistent, how about a telnet server for the repl too?

Yup! I mention something similar in #6 but forgot to add it to the TODO list.

This is desirable for web workflow. However, users are currently in full control of starting and stopping a Station and connecting to an AP, starting and stopping an AP, starting and stopping wifi, etc.

Agreed! I'd probably add a supervisor.disable_web_workflow() just like supervisor.disable_ble_workflow(). That'd move it back to the current API I think.

[FoamyGuy]

This is such a neat idea! Looking over the tasks you've outlined with an eye toward finding some part that I might be capable of helping with, I'm curious if the internal HTTP server that you mentioned is something that will get written with python code? or it's internal to the core and thus written in C?

[KTibow]

Hype! This probably won't be useful to anyone, but here's my OTA solution I built for my project. It just fetches from GitHub.

[tannewt]

This is such a neat idea! Looking over the tasks you've outlined with an eye toward finding some part that I might be capable of helping with, I'm curious if the internal HTTP server that you mentioned is something that will get written with python code? or it's internal to the core and thus written in C?

Generally we need C code for things that run outside the VM. My plan was to look into the HTTP Server that the IDF has and probably use it. It might be pretty big though. We could probably make a smaller one that is purpose built for what we need.

Hype! This probably won't be useful to anyone, but here's my OTA solution I built for my project. It just fetches from GitHub.

Thanks for the link! It's good to collect ideas.

[tannewt]

A custom built server could also build on top of common_hal and be useful for any port that supports it.

[tannewt]

Turns out Android Chrome doesn't do mDNS yet. Here is their feature request: https://crbug.com/405925

[tannewt]

Android 12+ now supports MDNS .local in Chrome. 🎉 https://www.androidpolice.com/android-mdns-local-hostname/

[Neradoc]

Hey, how about Webdav ?
I don't know much about it, except that it uses http. It would be nice to have some minimal implementation of it, if that doesn't require too many additions. That would allow accessing the files with a file manager application like Cyberduck, and use editors like VSCode, PyCharm, etc. maybe even mount as a drive on the computer (and run circup, why not).

[tannewt]

Hey, how about Webdav ?
I don't know much about it, except that it uses http. It would be nice to have some minimal implementation of it, if that doesn't require too many additions. That would allow accessing the files with a file manager application like Cyberduck, and use editors like VSCode, PyCharm, etc. maybe even mount as a drive on the computer (and run circup, why not).

Please open a separate issue for it. I think it'd be good to have a list of clients we want to support. That way we can figure out what subset of webdav they use. I skipped it for now because I was wary of how much space it'd take to implement.

[tannewt]

@FoamyGuy finished this by adding the edit page. Considering this done! Thanks all!

[michael-land]

Coming from #4050. Is there any guide/example for OTA update?

[RetiredWizard]

https://docs.circuitpython.org/en/latest/docs/workflows.html

[wangeris]

is it possible to enable Web Workflow on devices that are connected to the network via Ethernet instead of Wi-Fi?

[jepler]

No, that is not possible at this time. If there's not an open Issue about "web workflow over ethernet", it would be appropriate to open one. That said, it's not likely to be prioritized by Adafruit to work on.

[kub3let]

How can I override the root page and specify my own routes ?

Would like to host my own website / api but port 80 is already bound by CircuitPython's Web API

[FoamyGuy]

@kub3let you could try changing CIRCUITPY_WEB_API_PORT in settings.toml to be a value other than 80 (guide page: https://learn.adafruit.com/getting-started-with-web-workflow-using-the-code-editor/device-setup#creating-a-settings-dot-toml-file-3125968)

Or you could use different names for your WIFI credentials inside settings.toml. If the core system can't find the credentials under those names then it won't start the web workflow server I think.

[kub3let]

@FoamyGuy thank you, to be clear I want to keep the Workflow Web API in order to use the /code functionality.

I will change the port for now via settings.toml, seems like the best way.