Add run-length encoding for compressing OTA SPIFFS?

[sheffieldnikki]

Basic Infos

Hardware

Hardware: ESP-12F
Core Version: 2.4.0

Description

Would it be possible to add a very simple run-length encoding scheme to compress SPIFFS file as they are transferred to the ESP8266 ? Because the SPIFFS are generally limited to a few options (e.g. 1MB or 3MB) users are very likely to have unused space in their SPIFFS image, which is easily compressible runs of 0xFF.

Alternatively, transfer pages of SPIFFS data that have data in them (uncompressed), but embed a code to indicate to the ESP8266 that "all following pages are to be written as empty" ?

In my tests it takes 51 secs to transfer a 1MB SPIFFS image OTA to an ESP8266 over a strong WiFi signal, or about 20KB/sec.

(My workaround for now is defining a small SPIFFS area (e.g. 64KB) in my boards.txt that will upload in just a few seconds.)

[earlephilhower]

I mentioned #5635 fixing this before, but I didn't realize you were talking OTA. #5635 speeds up only serial uploads (and does it a LOT in my experience on SPIFFS), but has nothing to do w/OTA. The stub compression would need to be added to the Arduino OTA infrastructure, including 8266 driver code and host-based espota.py.

[davisonja]

If there's that much free space how about a transfer that only sent used bytes and where to put them?

[earlephilhower]

Simple RLE, only looking at # of consecutive 0xff or 0x00. I like it!

[earlephilhower]

#6820 solved this with gzunzip support in eboot, I believe. Closing.