From 9f5ceaf1ebb2a5f0e282394a727827417b931a0f Mon Sep 17 00:00:00 2001 From: "pedro.minatel" Date: Mon, 16 May 2022 16:33:08 +0100 Subject: [PATCH 1/6] Added partition table guide --- docs/source/tutorials/partition_table.rst | 137 ++++++++++++++++++++++ 1 file changed, 137 insertions(+) create mode 100644 docs/source/tutorials/partition_table.rst diff --git a/docs/source/tutorials/partition_table.rst b/docs/source/tutorials/partition_table.rst new file mode 100644 index 00000000000..fb04877504b --- /dev/null +++ b/docs/source/tutorials/partition_table.rst @@ -0,0 +1,137 @@ +############### +Partition Table +############### + +Introduction +------------ + +Partition table is used to define the flash memory organization and the different kind of data will be stored on each partition. + +You can use one of the predefined partition table schemes available or create your own. + +The partition table is created using a CSV file format with the following format: + +.. code-block:: + + # ESP-IDF Partition Table + # Name, Type, SubType, Offset, Size, Flags + +Where: + +1. **Name** + + Is the partition name and must be a unique name. This name is not relevant for the system and the size must be at maximum of 16-chars. + +2. **Type** + + This is the type of the partition. This value can be ``data`` or ``app``. + + * ``app`` (0x00) type is used to define the partition that will store the application. + + * ``data`` (0x01) type can be used to define the partition that stores general data, not the application. + +3. **SubType** + + The SubType defines the usage of the ``app`` and ``data`` partitions. + + **data** + + ``ota`` (0x00) + + The ota subtype is used to store the OTA information. This partition is used only when the OTA is used to select the initialization partition, otherwise no need to add it to your custom partition table. + The size of this partition should be a fixed size of 8kB (0x2000 bytes). + + ``nvs`` (0x02) + + The nvs partition subtype is used to define the partition to store general data, like the WiFi data, device PHY calibration data and any other data to be stored on the non-volatile memory. + This kind of partition is suitable for small custom configuration data, cloud certificates, etc. Another usage for the NVS is to store sensitive data, since the NVS supports encryption. + It is highly recommended to add at least one nvs partition, labeled with the name nvs, in your custom partition tables with size of at least 12kB (0x3000 bytes). If needed, you can increase the size of the nvs partition. + The recommended size for this partition is from 12kb to 64kb. Although larger NVS partitions can be defined, we recommend using FAT or SPIFFS filesystem for storage of larger amounts of data. + + ``coredump`` (0x03) + + The coredump partition subtype is used to store the core dump on the flash. The core dump is used to analyze critical errors like crash and panic. + This function must be enabled in the project configuration menu and set the data destination to flash. + The recommended size for this partition is 64kB (0x10000). + + ``nvs_keys`` (0x04) + + The nvs_keys partition subtype is used to store the keys when the NVS encryption is used. + The size for this partition is 4kB (0x1000). + + ``fat`` (0x81) + + The fat partition subtype defines the FAT filesystem usage, and it is suitable for larger data and if this data is often updated and changed. The FAT FS can be used with wear leveling feature to increase the erase/modification cycles per memory sector and encryption for sensitive data storage, like cloud certificates or any other data that may be protected. + To use FAT FS with wear leveling see the example. + + ``spiffs`` (0x82) + + The spiffs partition subtype defines the SPI flash filesystem usage, and it is also suitable for larger files and it also performs the wear leveling and file system consistency check. + The SPIFFS do not support flash encryption. + + **app** + + ``factory`` (0x00) + + The factory partition subtype is the default application. The bootloader will set this partition as the default application initialization if no OTA partition is found, or the OTA partitions are empty. + If the OTA partition is used, the ota_0 can be used as the default application and the factory can be removed from the partition table to save memory space. + + ``ota_0`` to ``ota_15`` (0x10 – 0x19) + + The ota_x partition subtype is used for the Over-the air update. The OTA feature requires at least two ota_x partition (usually ota_0 and ota_1) and it also requires the ota partition to keep the OTA information data. + Up to 16 OTA partitions can be defined but only two are needed for basic OTA feature. + + ``test`` (0x20) + + The test partition subtype is used for factory test procedures. + +4. **Offset** + + The offset defines the partition start address. The offset is defined by the sum of the offset and the size of the earlier partition. + +.. note:: + Offset must be multiple of 4kB (0x1000) and for app partitions it must be aligned by 64kB (0x10000). + If left blank, the offset will be automatically calculated based on the end of the previous partition, including any necessary alignment. + +5. **Size** + + Size and defines the amount of memory to be allocated on the partition. The size can be formatted as decimal, hex numbers (0x prefix), or using unit prefix K (kilo) or M (mega) i.e: 4096 = 4K = 0x1000. + +6. **Flags** + + The last column in the CSV file is the flags and it is currently used to define if the partition will be encrypted by the flash encryption feature. + + +For example, the most common partition is the ``default_8MB.csv`` (see `tools/partitions `_ folder for some examples): + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, 0x9000, 0x5000, + otadata, data, ota, 0xe000, 0x2000, + app0, app, ota_0, 0x10000, 0x330000, + app1, app, ota_1, 0x340000,0x330000, + spiffs, data, spiffs, 0x670000,0x190000, + +Using a Custom Partition Scheme +------------------------------- + +To create your own partition table, you can create the ``partitions.csv`` file in the same folder you created your sketch. The build system will automatically pick the partition table file and use it instead of the predefined ones. + +Here is an example you can use for a custom partition table: + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, , 64K, + otadata, data, ota, , 8K, + app0, app, ota_0, , 2M, + app1, app, ota_1, , 2M, + spiffs, data, spiffs, , 8M, + +This partition will use about 12MB of the 16MB flash. The offset will be automatically calculated and the units are in K and M. + +Reference +--------- + +This documentation was based on the `How to use custom partition tables on ESP32 `_ article. From 6198fd7b58b587c08c9b86f1d54683df3011eefb Mon Sep 17 00:00:00 2001 From: "pedro.minatel" Date: Mon, 16 May 2022 17:11:13 +0100 Subject: [PATCH 2/6] Using a custom partition file --- docs/source/tutorials/partition_table.rst | 28 ++++++++++++----------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/docs/source/tutorials/partition_table.rst b/docs/source/tutorials/partition_table.rst index fb04877504b..307a9505ec0 100644 --- a/docs/source/tutorials/partition_table.rst +++ b/docs/source/tutorials/partition_table.rst @@ -26,9 +26,9 @@ Where: This is the type of the partition. This value can be ``data`` or ``app``. - * ``app`` (0x00) type is used to define the partition that will store the application. + * ``app`` type is used to define the partition that will store the application. - * ``data`` (0x01) type can be used to define the partition that stores general data, not the application. + * ``data`` type can be used to define the partition that stores general data, not the application. 3. **SubType** @@ -36,52 +36,52 @@ Where: **data** - ``ota`` (0x00) + ``ota`` The ota subtype is used to store the OTA information. This partition is used only when the OTA is used to select the initialization partition, otherwise no need to add it to your custom partition table. The size of this partition should be a fixed size of 8kB (0x2000 bytes). - ``nvs`` (0x02) + ``nvs`` The nvs partition subtype is used to define the partition to store general data, like the WiFi data, device PHY calibration data and any other data to be stored on the non-volatile memory. This kind of partition is suitable for small custom configuration data, cloud certificates, etc. Another usage for the NVS is to store sensitive data, since the NVS supports encryption. It is highly recommended to add at least one nvs partition, labeled with the name nvs, in your custom partition tables with size of at least 12kB (0x3000 bytes). If needed, you can increase the size of the nvs partition. The recommended size for this partition is from 12kb to 64kb. Although larger NVS partitions can be defined, we recommend using FAT or SPIFFS filesystem for storage of larger amounts of data. - ``coredump`` (0x03) + ``coredump`` The coredump partition subtype is used to store the core dump on the flash. The core dump is used to analyze critical errors like crash and panic. This function must be enabled in the project configuration menu and set the data destination to flash. The recommended size for this partition is 64kB (0x10000). - ``nvs_keys`` (0x04) + ``nvs_keys`` The nvs_keys partition subtype is used to store the keys when the NVS encryption is used. The size for this partition is 4kB (0x1000). - ``fat`` (0x81) + ``fat`` The fat partition subtype defines the FAT filesystem usage, and it is suitable for larger data and if this data is often updated and changed. The FAT FS can be used with wear leveling feature to increase the erase/modification cycles per memory sector and encryption for sensitive data storage, like cloud certificates or any other data that may be protected. To use FAT FS with wear leveling see the example. - ``spiffs`` (0x82) + ``spiffs`` The spiffs partition subtype defines the SPI flash filesystem usage, and it is also suitable for larger files and it also performs the wear leveling and file system consistency check. The SPIFFS do not support flash encryption. **app** - ``factory`` (0x00) + ``factory`` The factory partition subtype is the default application. The bootloader will set this partition as the default application initialization if no OTA partition is found, or the OTA partitions are empty. If the OTA partition is used, the ota_0 can be used as the default application and the factory can be removed from the partition table to save memory space. - ``ota_0`` to ``ota_15`` (0x10 – 0x19) + ``ota_0`` to ``ota_15`` The ota_x partition subtype is used for the Over-the air update. The OTA feature requires at least two ota_x partition (usually ota_0 and ota_1) and it also requires the ota partition to keep the OTA information data. Up to 16 OTA partitions can be defined but only two are needed for basic OTA feature. - ``test`` (0x20) + ``test`` The test partition subtype is used for factory test procedures. @@ -91,7 +91,7 @@ Where: .. note:: Offset must be multiple of 4kB (0x1000) and for app partitions it must be aligned by 64kB (0x10000). - If left blank, the offset will be automatically calculated based on the end of the previous partition, including any necessary alignment. + If left blank, the offset will be automatically calculated based on the end of the previous partition, including any necessary alignment, however, the offset for the first partition must be always set as **0x9000**. 5. **Size** @@ -123,7 +123,7 @@ Here is an example you can use for a custom partition table: .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, , 64K, + nvs, data, nvs, 0x9000, 20K, otadata, data, ota, , 8K, app0, app, ota_0, , 2M, app1, app, ota_1, , 2M, @@ -131,6 +131,8 @@ Here is an example you can use for a custom partition table: This partition will use about 12MB of the 16MB flash. The offset will be automatically calculated and the units are in K and M. +A alternative is to create the new partition table as a new file in the `tools/partitions `_ folder and edit the `boards.txt `_ file to add your custom partition table. + Reference --------- From 6e5a91540587973984cb952e2b685ae0b168b1e8 Mon Sep 17 00:00:00 2001 From: "pedro.minatel" Date: Tue, 17 May 2022 10:14:32 +0100 Subject: [PATCH 3/6] Added some more examples for partitions --- docs/source/tutorials/partition_table.rst | 51 ++++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-) diff --git a/docs/source/tutorials/partition_table.rst b/docs/source/tutorials/partition_table.rst index 307a9505ec0..1f6076ec6e3 100644 --- a/docs/source/tutorials/partition_table.rst +++ b/docs/source/tutorials/partition_table.rst @@ -95,7 +95,7 @@ Where: 5. **Size** - Size and defines the amount of memory to be allocated on the partition. The size can be formatted as decimal, hex numbers (0x prefix), or using unit prefix K (kilo) or M (mega) i.e: 4096 = 4K = 0x1000. + Size defines the amount of memory to be allocated on the partition. The size can be formatted as decimal, hex numbers (0x prefix), or using unit prefix K (kilo) or M (mega) i.e: 4096 = 4K = 0x1000. 6. **Flags** @@ -133,6 +133,55 @@ This partition will use about 12MB of the 16MB flash. The offset will be automat A alternative is to create the new partition table as a new file in the `tools/partitions `_ folder and edit the `boards.txt `_ file to add your custom partition table. +Examples +-------- + +**2MB no OTA** + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, 0x9000, 20K, + factory, app, factory, , 1992K, + +**4MB no OTA** + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, 0x9000, 20K, + factory, app, factory, , 4000K, + +**4MB with OTA** + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, 0x9000, 20K, + otadata, data, ota, , 8K, + app0, app, ota_0, , 1900K, + app1, app, ota_1, , 1900K, + +**8MB no OTA with Storage** + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, 0x9000, 20K, + factory, app, factory, , 2M, + spiffs, data, spiffs, , 5M, + +**8MB with OTA and Storage** + +.. code-block:: + + # Name, Type, SubType, Offset, Size, Flags + nvs, data, nvs, 0x9000, 20K, + otadata, data, ota, , 8K, + app0, app, ota_0, , 2M, + app1, app, ota_1, , 2M, + spiffs, data, spiffs, , 3M, + Reference --------- From 6e686921fdd46838472a351422f583b729cb5532 Mon Sep 17 00:00:00 2001 From: "pedro.minatel" Date: Wed, 18 May 2022 15:03:42 +0100 Subject: [PATCH 4/6] Fixed the app partition offset --- docs/source/tutorials/partition_table.rst | 34 +++++++++++------------ 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/source/tutorials/partition_table.rst b/docs/source/tutorials/partition_table.rst index 1f6076ec6e3..799dbb07123 100644 --- a/docs/source/tutorials/partition_table.rst +++ b/docs/source/tutorials/partition_table.rst @@ -91,7 +91,7 @@ Where: .. note:: Offset must be multiple of 4kB (0x1000) and for app partitions it must be aligned by 64kB (0x10000). - If left blank, the offset will be automatically calculated based on the end of the previous partition, including any necessary alignment, however, the offset for the first partition must be always set as **0x9000**. + If left blank, the offset will be automatically calculated based on the end of the previous partition, including any necessary alignment, however, the offset for the first partition must be always set as **0x9000** and for the first application partition **0x10000**. 5. **Size** @@ -123,13 +123,13 @@ Here is an example you can use for a custom partition table: .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, 0x9000, 20K, - otadata, data, ota, , 8K, - app0, app, ota_0, , 2M, + nvs, data, nvs, 36K, 20K, + otadata, data, ota, 56K, 8K, + app0, app, ota_0, 64K, 2M, app1, app, ota_1, , 2M, spiffs, data, spiffs, , 8M, -This partition will use about 12MB of the 16MB flash. The offset will be automatically calculated and the units are in K and M. +This partition will use about 12MB of the 16MB flash. The offset will be automatically calculated after the first application partition and the units are in K and M. A alternative is to create the new partition table as a new file in the `tools/partitions `_ folder and edit the `boards.txt `_ file to add your custom partition table. @@ -141,25 +141,25 @@ Examples .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, 0x9000, 20K, - factory, app, factory, , 1992K, + nvs, data, nvs, 36K, 20K, + factory, app, factory, 64K, 1900K, **4MB no OTA** .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, 0x9000, 20K, - factory, app, factory, , 4000K, + nvs, data, nvs, 36K, 20K, + factory, app, factory, 64K, 4000K, **4MB with OTA** .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, 0x9000, 20K, - otadata, data, ota, , 8K, - app0, app, ota_0, , 1900K, + nvs, data, nvs, 36K, 20K, + otadata, data, ota, 56K, 8K, + app0, app, ota_0, 64K, 1900K, app1, app, ota_1, , 1900K, **8MB no OTA with Storage** @@ -167,8 +167,8 @@ Examples .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, 0x9000, 20K, - factory, app, factory, , 2M, + nvs, data, nvs, 36K, 20K, + factory, app, factory, 64K, 2M, spiffs, data, spiffs, , 5M, **8MB with OTA and Storage** @@ -176,9 +176,9 @@ Examples .. code-block:: # Name, Type, SubType, Offset, Size, Flags - nvs, data, nvs, 0x9000, 20K, - otadata, data, ota, , 8K, - app0, app, ota_0, , 2M, + nvs, data, nvs, 36K, 20K, + otadata, data, ota, 56K, 8K, + app0, app, ota_0, 64K, 2M, app1, app, ota_1, , 2M, spiffs, data, spiffs, , 3M, From 914e228953419c0acff6c2fee7d74369ccb324b3 Mon Sep 17 00:00:00 2001 From: "pedro.minatel" Date: Fri, 20 May 2022 09:21:26 +0100 Subject: [PATCH 5/6] Added bare_minimum_2MB partition table file --- tools/partitions/bare_minimum_2MB.csv | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 tools/partitions/bare_minimum_2MB.csv diff --git a/tools/partitions/bare_minimum_2MB.csv b/tools/partitions/bare_minimum_2MB.csv new file mode 100644 index 00000000000..e688a47cfa2 --- /dev/null +++ b/tools/partitions/bare_minimum_2MB.csv @@ -0,0 +1,3 @@ +# Name, Type, SubType, Offset, Size, Flags +nvs, data, nvs, 36K, 20K, +factory, app, factory, 64K, 1900K, From 2e0e4c1f0560e36739336da08dcedcd9c1950702 Mon Sep 17 00:00:00 2001 From: "pedro.minatel" Date: Mon, 30 May 2022 10:34:50 +0100 Subject: [PATCH 6/6] PR review changes --- docs/source/tutorials/partition_table.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/tutorials/partition_table.rst b/docs/source/tutorials/partition_table.rst index 799dbb07123..f073a8d0c27 100644 --- a/docs/source/tutorials/partition_table.rst +++ b/docs/source/tutorials/partition_table.rst @@ -7,9 +7,9 @@ Introduction Partition table is used to define the flash memory organization and the different kind of data will be stored on each partition. -You can use one of the predefined partition table schemes available or create your own. +You can use one of the available partition table scheme or create your own. You can see all the different schemes on the `tools/partitions `_ folder or by the Arduino IDE tools menu `Tools -> Partition Scheme`. -The partition table is created using a CSV file format with the following format: +The partition table is created by a .CSV (Comma-separeted Values) file with the following structure: .. code-block:: @@ -20,7 +20,7 @@ Where: 1. **Name** - Is the partition name and must be a unique name. This name is not relevant for the system and the size must be at maximum of 16-chars. + Is the partition name and must be a unique name. This name is not relevant for the system and the size must be at maximum of 16-chars (no special chars). 2. **Type** @@ -102,7 +102,7 @@ Where: The last column in the CSV file is the flags and it is currently used to define if the partition will be encrypted by the flash encryption feature. -For example, the most common partition is the ``default_8MB.csv`` (see `tools/partitions `_ folder for some examples): +For example, **the most common partition** is the ``default_8MB.csv`` (see `tools/partitions `_ folder for some examples): .. code-block:: @@ -116,7 +116,7 @@ For example, the most common partition is the ``default_8MB.csv`` (see `tools/pa Using a Custom Partition Scheme ------------------------------- -To create your own partition table, you can create the ``partitions.csv`` file in the same folder you created your sketch. The build system will automatically pick the partition table file and use it instead of the predefined ones. +To create your own partition table, you can create the ``partitions.csv`` file **in the same folder you created your sketch**. The build system will automatically pick the partition table file and use it instead of the predefined ones. Here is an example you can use for a custom partition table: