Logging issue : a lot of files and logs drops

[Stefal]

Subject of the issue

We are using some RTK Express to log data when we walk or drive.
Instead of one file for each trip, we have a lot of files, drops inside the files, and too much float or single fix.

Your workbench

• What version of RTK firmware are you running? Tested with 3.1, 3.2, 3.3
• 4 differents RTK Express
• 3 differents antenna
• various SD card
• 2 to 5 Hz
• messages enabled: NMEA + nav-pvt + nav-hpposllh + nav-sat + nav-gpstime + nav-utctime

Steps to reproduce

• Power on, connect to a ntrip caster with lefebure ntrip client. Bluetooth link between the smartphone and the RTK Express.

Expected behavior

• Only one file, and much more RTK fix

Actual behavior

• 2 hours log give up to 10 files with a lot of cuts.
  some example:
  
  
  

[Stefal]

new test with the firmware v3.0:
The test was short, but I didn't have multiple ubx files. However there are some holes in the log

[pobsteta]

Idem on RTK Facet with firmware 3.2

[nseidle]

@Stefal - We found a fixed a particularly bad bug on May 18th that was causing issues with corrupt log file names and unexpected resets. This bug was introduce on Feb 28 so v3.1 should have been good, while v3.2 and v3.3 have this bug.

Can you share some of the sections of the log where the RTK Fix is lost? Can you tell if the break is due to a system reset or does the system simply lose RTK Fix and then re-gain it?

[Eric-FR]

Same with a RTK Express with 3.2 firmware. Tracks are divided in numerous different files. Sometime, I can observe device reboot. I lost several tracks: while RTK was active and used for Mapillary, I can't find any file on the SD card.

Foot walk with RTK Express recording NMEA+UBX at 1 Hz.

[Stefal]

Can you share some of the sections of the log where the RTK Fix is lost? Can you tell if the break is due to a system reset or does the system simply lose RTK Fix and then re-gain it?

I will wait for a new release with the various fixes you have mentionned, and redo some tests. I think that parts of my problems are related to the 5hz gnss frequency that the bluetooth stream can't handle. I forgot that log to file and bluetooth settings are the same, contrary to my old logger in which the stream are independant (1 or 2hz for bluetooth, and 5hz or more with many ubx messages for log to SD card).

[Stefal]

Test with the 3.4RC (RTK Firmware: v3.4-May 25 2023) :

• About 90mn log
• 5hz, with all nmea message rate set to 5 or more
• nav-pvt, nav-hpposllh, nav-sat, nav-utctime, nav-gpstime
• Connected to my own gnss base, with Lefebure Ntrip Client.

Result:

• Multiple files (9) and $GNTXT,01,01,00,ESP_RST_SW*46 messages
• Some drops (about 5 seconds without data)

I will share the logs with you in private (email)

[nseidle]

I've replicated the issue. An overnight run produced ~30 logs. The unit is closing old logs and opening new ones because no new data is being received for more than 5 seconds:

I haven't yet figured out why the ESP32 thinks the ZED is not outputting data. TBD!

[nseidle]

Serial bytes are all being read correctly even when the file size is not increasing.

This would seem to indicate the validation parser is not passing data on to the SD logging routine. I would expect the parser to throw a few sentences out across multiple hours. The fact it's throwing out multiple seconds worth of data is wrong.

[nseidle]

tldr; The release candidate should have some major improvements. Please load v3.5RC when possible and report back.

As usual, thanks for reporting this and sorry for the data corruptions!

To replicate this issue:

• Remove any SD card
• Reset device to defaults
• Enable CRC printing - option '20' on debug menu
• Connect to device over BT
• Connect NTRIP and push corrections to RTK Device over BT
• View output. You will see bad checksums that look like this:

Above: This is how the system works. The ESP32 takes in NMEA from the ZED transmits that NMEA to the phone over BT. Everything works fine, there are no NMEA or UBX sentence corruptions (ie, no data is dropped).

Above: If we start transmitting RTCM/NTRIP corrections from the phone, back to the ESP32, and back to the ZED, we begin to see bytes dropped (from the ZED-ESP NMEA communication).

Above: Interestingly, if we suppress the transmission of serial from the ESP32 to the ZED (commenting out lines in the Task.ino btReadTask code), there is no data corruption. It's therefore presumed that the BT stack is not interfering with the ESP32 UART2 (it's not an interrupt issue).

Above: Additionally, I verified the ZED was correctly outputting serial at all times. It is. The ESP32 is dropping characters out of the UART.

Things we’ve tried:

• Added a non-blocking TX ring buffer to esp_hal_uart.c
• Increased the UART interrupt to level 3 (max that it can be) in esp_hal_uart.c
• Decreased the UART baud to 57600. This should slow data rate to 5760kBytes/s. The buffer is 127 bytes so we have ~22ms.
• Decreasing task overhead (ie, removing Bluetooth escape character scanning, removing Bluetooth writing, remove SD card, etc)
• Decreasing serialGNSSRxFullThreshold (this should allow gnssReadTask to harvest the data faster)
• Moved the UART interrupts and tasks to core 0 (from core 1).
• Moved the Bluetooth tasks to core 0 (from core 1).
• Increasing the priority of the gnssReadTask, decreasing the priority of the btReadTask
• Increasing BT SPP TX/RX buffers sizes (including SPP_TX_MAX)
• Increase uartReceiveBufferSize (this should allow the gnssReadTask more time to service the buffer)

Nothing above decreased the number of corrupted sentences coming from the ZED when the ESP32 is writing data.

What is especially telling is the reduction of the UART speed. If we are slowing down the UART speed, all interrupts and tasks have a lot more time to operate before data is lost, yet we still lose data.

I believe there is a very low level issue with the esp32-idf that causes the UART FIFO to misbehave when we are simultaneously reading and writing to it. I am not able to figure it out at this time. However, there is a simple fix:

Above: We can push the RTCM over I2C rather than UART, thus avoiding a serial2.write() while we are doing a serial2.read().

Originally I was seeing corruptions every ~10s or so. With the fix in place, I have not seen a corrupted message at 10Hz, with 10 messages turned on, in 15 minutes.

There is still work to do:

• This fix increases the I2C traffic by about 530 bytes per second (~6% of max throughput of the bus at 100kHz). This is not a lot considering all the other I2C traffic (display and general ZED messages) but it is an increase.
• Control what data is sent back over Bluetooth. If we can avoid sending UBX to the phone, that would be a good thing.
• We still close log/open new log if the log file size is not increasing. We may want to change this in the future.

[Stefal]

Thanks Nathan,

This RC is uploaded on my express. I will do a quick test tomorrow evening.

[Stefal]

Tests results with the 3.5RC, during about 1,5 hour :

At least 90% of what you see in the screenshot should have been green (RTK Fix). (5hz and 4Hz, NMEA + NAV-PVT and NAV-HPPOSLLH)

Next tests : default settings with "automotive" dynamic model.
At least 90% of what you see should have been green. Blue is dgps, orange is 3D and yellow is for Float.

The conclusion : it's a failure.

Good points :

• No multiple files
• No errors detected with ubx integrity checker.

It looks like the F9P doesn't always received the rtcm data stream.

[pyrog]

Could you swap RTCM and UBX/NMEA ?

• UBX/NMEA on I2C
• RTCM on UART1

[Eric-FR]

Following your investigations, I made new tests.

RTK Express
Firmware 3.0

Dynamic : automotive
UBX+NMEA at 4 Hz

Rover static on balcony
Android smartphone connected to caster with wifi+FTTH.
Start up
1 hour recording
Shut down
4 files
I first checked integrity with UBX_Integrity_Checker (related to #345):

-No connection : Integrity OK.
-BT connection with SW_Maps, NMEA reading, no RCTM sent: Integrity OK.
-BT connection with Lefebure, NMEA + RCTM : 15 resyncs
-BT connection with SW_Maps, NMEA + RCTM : 24 resyncs

For the last two files, I also checked the age of differentials:

It is still unstable, like in: #359