Running out of heap (and thus crashing) during filesystem update will corrupt the filesystem #7950
Comments
|
Closing as this is expected behavior. It overwrites the FS directly since in may cases the FS is way larger than the standard OTA space. Use the |
|
@earlephilhower The ESP continues to function with 2000 bytes of free memory, until the update starts and is almost completed. I agree that this is the expected behaviour, but a check if there is enough free heap memory to even process the update is a must in my opinion. |
|
I don't think so - you can always check the heap space before doing the update in your code, but according to your log it seems like you "eat the memory" when the update was in progress, so it's practically impossible to avoid this crash... |
|
@Pablo2048 Nope, the crash also occured on a system where I did not eat memory and the memory is only been eaten once. Running the Updater asynchronously creates the scenario that heap usage could be changed during the update. This is something that the library should account for as it can potentially brick (end user) devices |
|
So ok, lets say that we have the following scenario - the update is running, half of the filesystem is already overwritten (on the fly) and now the heap disappeared - we just not have enough memory. What do you think has to be done? The filesystem is already corrupted (half overwriten). The only way I can see is by using @earlephilhower suggestion - ATOMIC_FS_UPDATE - when download can not be finished, filesystem cannot be overwriten. Unfortunately the maximum size of the filesystem is the free OTA area, so if you have 2M/2M you are out of luck. |
|
Ehm, and the crash you have posted seems to be in the AsyncWebServer - he is unable to allocate some strings when he tries to add some headers... |
|
It's impossible to guarantee no memory overflow while doing OTA. There's a lot going on in the system with Wifi, external IP connections (as shown in your crash), etc. So that is why if you really care you use the ATOMIC option. Also, you can use |
Yeah I got you, sorry, misunderstood your last post.
Is the atomic behaviour guaranteed when using decompression, so that nothing will be changed if the ESP crashes when decompressing? |
|
ATOMIC uses the same stuff as the program OTA and runs right at bootup before any app code is run. So it's guaranteed to be atomic except in the case of power failure. That's one niggle that's left, and it applies to FS and app. |
|
To clarify: |
Yeah I got you, sorry, misunderstood your last post.
Is the atomic behaviour guaranteed when using decompression, so that nothing will be changed if the ESP crashes when decompressing?
Would it be an idea to add something to LittleFS/Updater that checks if the update was interrupted due to a crash or something? |
This can be done in your application code - I'm strongly against using the RTC area internally by some library without some sort of central management. For example I'm using portion of RTC area for counting unexpected crashes and forcing emergency mode when the application crashes too often to avoid bricking the device. So yes - you can use the RTC memory to mark unsuccessful update, but it wont survive power failure (EEPROM would be better?). |
Basic Infos
Platform
Settings in IDE
Problem Description
When the ESP is (almost) out of heap memory, and crashes during an update, the filesystem or files get corrupted, sometimes rendering the internal webserver useless.
How to reproduce
Upload the standard filesystem + sketch via USB.
Open the serial monitor, drag the filesystem image onto the upload button.
Before you click Upload to fs, make sure to visit
ESP_IP/fill_memorya couple of times until the free heap memory reaches around 7000 bytes.Click the upload to fs button, you might want to refresh the
ESP_IP/fill_memorypage one or two times during the upload, but it should also work without that.Watch the ESP crash due to the free heap size available and after a reboot, you'll see that the LittleFS is corrupted.
Please note that the filesystem reports that index.html is 3994 bytes both before and after the update, so the filesystems file table is still intact.
Due to the OTA partition of the ESP, I don't think that this will happen during a Flash upgrade, but please feel free to test.
Sketch
Debug Messages
The original index.html file:
This is the resulting html file due to the corruption:
The text was updated successfully, but these errors were encountered: