Errors & discrepancies - serial menu vs. web page form. Simplify to improve usability.

[doug-foster]

Subject of the issue

Organization, content, and location for EVK settings differ between serial menus and the web config page. Some field definitions have errors & discrepancies; some similar fields have non-matching values.

Your workbench

• EVK
• release_candidate branch - 2024-04-05@14-43-51 - RTK_Everywhere_rc

Steps to reproduce

Compare the serial menu and the web menu for any given state of the EVK

Expected behavior

A user should have one source of truth for how to configure the device. If presented with multiple sources - e.g. different menus with different content, organization, and values - the user can be confused on how to proceed.

Actual behavior

See attached zip file for snapshots of errors & comparisons between the two menu systems.

Suggested change

I suggest refactoring the configuration process. This would be a significant change, but hopefully a new web form could reuse code from the current web page.

1. Use only one method - a web form on a web page - for configuration.
2. If a serial menu is needed, keep functionality to a minimum. Do not duplicate the web-based config method.
3. Group & organize configuration settings by user perspective, not a programming viewpoint.
   See below and attachment for a possible solution.
4. Web-based configuration can be contained within a single HTML/CSS/JS page. Only a browser is required to run the JS; an http server is not needed.
5. Generate/read all setting values as JSON formatted text rather than line-by-line "item=value."
6. Transfer JSON data between the browser and device using either a) copy/paste via a serial port, or b) streaming over TCP.

Discussion

At Cisco I learned customers want to quickly grasp what device settings are available, what they do, and how to change them. This initial exploration can either greatly impress or discourage a new user.

As a new EVK user, I was surprised at how different the serial and web menus were. I found the serial menu quite confusing. I did find the web page (once I could get to it after WiFi & Eth config were fixed) much easier to understand. Tooltip hints were a big help.

Every user has access to a web browser. Web-based configuration is more functional, powerful, and intuitive than a serial terminal interface. One configuration method would be easier to maintain & support.

Suggested organization

The snapshot below references fields on the web config page (some serial menu fields may be missing). View the attached txt file "suggested reorganization.zip" in VS Code; the tabbed indents allow you to collapse/expand sections similar to how a web page with form fields & accordions might work. Not complete, but it presents an idea for a different way to organization the settings.

suggested reorganization.zip

menu_errors-discrepancies-observations.zip

[avinabmalla]

There is also the command line interface that is used to get/set the setting values directly. The serial menu can be replaced entirely by the command line interface.

An Android/iOS app is already in development to use the command line settings. We can create a desktop app to do the configuration over the USB serial port that uses the command line interface, which completely eliminates the need to have the serial menu. Other developers can also use the command line interface to automate receiver configuration.

As a surveyor, there's only two modes the receiver can be in that I would be concerned about: Rover and Base. There should be no confusion to the user whether they are setting up as a rover or as a base station. Having a rover/base switch in the web interface, and then only enabling the relevant settings would be a good idea. For example, we would normally not need NTRIP server functionality when operating as rover, and NTRIP client when operating as a base station.

[doug-foster]

Ah - Avinab! Thanks for the work on SW Maps!

I quickly tried the cli from the serial menu, but didn’t see doc anywhere on commands or syntax. Wasn’t intuitive (to me). Do you know where the cli doc is?

The cli parser in Cisco IOS was awesome for configuring network devices. Self documenting too. No need for serial menus. Config scripts of cli commands worked well.

Yes! Are you writing the mobile app?

Maybe Electron for the desktop web app? The comm link could be serial or TCP stream.

How would the device and the config app be kept in sync when new settings are added to the firmware?

Agreed - configuring needs to be simpler and more modal. I know, easier to say than do.

[avinabmalla]

The CLI is currently undocumented, but the code for the CLI is in the menuCommands.ino script.

The CLI commands are quite simple:

GET|<settingName> returns the setting value.
SET|<settingName>|<value> sets the value.
APPLY saves the settings and restarts the firmware.
CMD returns OK to indicate command line is active.
EXIT closes the command line interface and should return to normal serial port/Bluetooth behavior.
LIST lists all available settings and their data type.

The LIST command will be used to keep the app in sync with the firmware, with some commonly used settings having it's own interface that's fixed on the app.

You should be able to enter the command line interface by sending a sequence of 10 dashes ---------- but this does not always work. Other way is by entering + in the serial menu.

[doug-foster]

Thanks Avinab! I'll give it a try.

I also discovered https://wicg.github.io/serial/ - browser JS to serial. Only for Chrome/Edge/Opera desktop so far.

Examples & more at https://github.com/tigoe/html-for-conndev.

[PaulZC]

Hi Doug (& Avinab),

Just echoing my comments from #227:

We are at the point where we need to add / update the Config WiFi HTML and JS for the Torch. It is a very different beast to the EVK / Facet. The UM980 messages are completely different to those for the ZED-F9P. If I'm not careful, I'll end up with two large blocks of HTML and JS, one for the ZED and a second for the UM980. It'll be messy to maintain. And will be further complicated when we add the mosaic X5 into the mix... It would indeed be so much better if the firmware could construct the webpage - as JSON maybe - zips it - and throws it to a very simple JS plus HTML page - which unzips it and does the necessary. I'll have a good chat with @nseidle about this.

Cheers!
Paul

[avinabmalla]

Hi @PaulZC

Right now I really think we need to add a layer of abstraction over the GNSS receivers and their SparkFun libraries (which are really well made btw!).

All GNSS receivers, from any manufacturer, has the same common settings that need to be configured by the user. Things like elevation mask, CN0 mask, data output formats/rates and base station coordinates for example. Having this common subset of functions in an abstract GNSSReceiver class, and then creating concrete implementations for each receiver you are using would be really helpful. The concrete implementations then utilize the relevant SparkFun GNSS library to communicate with the receiver module.

Any other receiver specific things must be handled by the receiver specific classes. It's not even necessary to expose the user to them, since all your receivers provide a direct interface to the GNSS receiver module over USB. Or they could be made accessible over the command line interface, with each GNSSReceiver implementation processing messages only relevant to it.

It would be necessary to standardize and document each setting value, and what it does, before we go on to develop the new Wi-Fi config, and a standard command line protocol for all SparkFun receivers.

[PaulZC]

Many thanks @avinabmalla : we have started to add common methods for the different GNSS modules. These are in GNSS.ino. They are not a Class, just a set of common methods, but it is a step in the right direction! They concentrate on getting information from the module: gnssGetLatitude, gnssGetSatellitesInView. Where it gets interesting is in methods like gnssFixedBaseStart, that's where the base coordinates are sent to the module. gnssConfigureRover and gnssConfigureBase define which messages are output by the module, for logging or Bluetooth, based on the settings.

@doug-foster : many thanks for the very useful errors and reorganization!

I think what we need right now is a spreadsheet where we can capture all of the settings and what they do, and define what we want to appear on the Config page and where. I'll start putting together a Google Sheet. I'll ask @nseidle for your email addresses so I can share it with you. You can find one of my emails on my profile page if you need it. Once the spreadsheet is complete, we can copy it across into the docs so users can see the settings too.

More later!

[doug-foster]

I like the new "click to lower priority" approach!

Consider this use case though. What if I want to change from the default settings to:

• L-Band as # 1
• IP (PointPerfect/MQTT) as # 2

Which would be more intuitive for the user?

• A) click "L-Band" 3 times to elevate it to # 1, then click "IP (PointPerfect/MQTT)" 1 time to elevate it to # 2 ... or
• B) click "TCP (NTRIP)" 1 time to lower it, click "IP (PointPerfect/MQTT)" 1 time to lower it, click "BlueTooth" 1 time to lower it (this gets "L-Band" into the # 1 position), then click "BlueTooth" 1 more time (this gets "IP (PointPerfect/MQTT)" into the # 2 position.

With method A, the user clicks the specific item they want to prioritize, with method B the user clicks an item to get it out of the way of the item they want to prioritize.

Perhaps a nit, but I think method A might be more intuitive. Thoughts?

[doug-foster]

The web menu & settings/serial menu are still having sync problems in https://github.com/sparkfun/SparkFun_RTK_Everywhere_Firmware_Binaries/blob/main/RTK_Everywhere_Firmware_RC-Apr_24_2024.bin.

1. Reset EVK to factory defaults
2. Enter "Cfg WiFi" from the front panel
3. connect, bring up the web menu
4. add a WiFi SSID & password
5. save settings
6. reload web form or check serial menu
7. SSID = password, password = ''

I did not try running my script to build from source, but only ran the binary in the repo.

[PaulZC]

Hi doug,

I think I fixed this yesterday afternoon. Please pull the latest release_candidate and give it another go...

Please be aware that there are more web menu changes coming - as I test it on Torch.

Thanks!
Paul

[doug-foster]

Ah - I see. Thx.

However FYI, I saw another anomaly. In "Cfg Eth" mode, the menu buttons under "Corrections Configuration" get confused.

1. enter "Cfg Eth" using the MODE button
2. open the web menu
3. click on the "L-Band" button - the label changes to "External Radio" - there are now (2) "External Radio" buttons. Huh??

I didn't see that before when in "cfg WiFi" mode and making the earlier comments on this issue .

I just went back & checked. Now I am seeing the same behavior in "cfg WiFi" mode - same anomaly.

I have no idea what I did different. I reset to factory defaults, still seeing the same anomaly.

Hmm could this be the problem? Why is there no settings file?

Menu: System
----- Mode Switch -----
Mode: Setup
B) Switch to Base mode
N) Switch to NTP Server mode
R) Switch to Rover mode
W) Switch to WiFi Config mode
----- Settings -----
b) Set Bluetooth Mode: Dual
c) Shutdown if not charging: Disabled
d) Debug software
e) Echo User Input: On
f) Display microSD Files
h) Debug hardware
n) Debug network
o) Configure RTK operation
p) Configure periodic print messages
r) Reset all settings to default
u) Toggle printed measurement scale: meters
z) Set time zone offset: 00:00:00
~) Setup button: Enabled
x) Exit
f
Files found:
2024-04-25.0 B.SFE_EVK_240425_131637.ubx
2024-04-25.0 B.SFE_EVK_240425_131921.ubx
2024-04-25.0 B.SFE_EVK_240425_150002.ubx
2024-04-25.0 B.SFE_EVK_240425_150241.ubx
2024-04-25.0 B.SFE_EVK_240425_150707.ubx
2024-04-25.0 B.SFE_EVK_240425_151115.ubx
2024-04-25.0 B.SFE_EVK_240425_160303.ubx
2024-04-25.0 B.SFE_EVK_240425_161654.ubx
2024-04-25.0 B.SFE_EVK_240425_162241.ubx
2024-04-25.0 B.SFE_EVK_240425_162657.ubx

System Status
2024-04-25 16:36:22.065
I2C: Offline - Something is causing bus problems
GNSS: Offline
Display: Online
microSD: Online
NEO-D9S L-Band: Offline -
No Keys / Corrections Received Failed - NEO-D9S firmware:
Bluetooth Low Energy (8F06): GNSS Offline
WiFi MAC Address: E0:5A:1B:D8:8F:04
Ethernet cable: connected
Ethernet MAC Address: E0:5A:1B:D8:8F:07
Ethernet IP Address: 192.168.2.90
System Uptime: 0 00:06:11.674 (Resets: 0)
Filtered by parser: 0 NMEA / 0 RTCM / 0 UBX

[PaulZC]

Have I mentioned how much I hate JavaScript...?!

correctionsSourcePriorities[x] += 1;

If correctionsSourcePriorities[x] is "5", the result is ..... drum roll please ..... "51"! Not 6.....

Fixing...

[PaulZC]

PS. We really like your idea of making the buttons increase the priority. Much more intuitive... Almost done!

[PaulZC]

OK. The web page corrections prioritisation click-to-increase is working nicely. The changes are in the release_candidate branch but haven't yet made it into a RC .bin. The SSID / Password issue is fixed too. Also in release_candidate.