Use FAT16 instead of FAT12, for greater compatibility with Android and other OS's

[olivier-boesch]

It would be to make the circuitpython filesystem recognizable by android systems. By doing so, it becomes possible to easily edit code with editors on mobile devices.

Wouldn't it be nice?

As far I know, the usb filesystem used by circuitpython is fat12. Is it was in fat16, it could be recognized by android devices.

[dhalbert]

Whether FAT12, FAT16, or FAT32 is used depends on the size of the filesystem. A FAT12 filesystem is defined to be < 4087 clusters, a FAT16 filesystem is 4087 to 65526 clusters, and FAT32 is a range above that. The typical 2MB filesystem we supply can therefore only be FAT12, by definition.

One issue has been that FAT12 is not well supported, as you see. On Windows, there's a bug (which is supposedly being fixed) that delays writing to a FAT12 filesystem for tens of seconds. See #111.

I've thought about presenting a FAT16 filesystem that would appear to be larger than the actual capacity, but with a large number of bad blocks. The File Allocation Table has a mechanism for marking bad blocks. I did some experiments on this, but haven't made it a high priority. It's not so simple.

[olivier-boesch]

Thanks for your explanation. I missed the first one.
So should it be kept opened? As a reminder? Or not.

[RokeJulianLockhart]

@olivier-boesch, https://github.com/izuchukwu/USBBridge#what states

A lot of Android devices support USB OTG, which lets you read USB drives. But most, like Google's Pixel and Nexus devices, don't support FAT12 and FAT16

so are you certain that this would provide compatibility with Android?

[dhalbert]

I haven't looked at this in detail for several years, but my impression was that FAT16 would work. I have some very small SD cards and could try this with an Android tablet. But there is no work being done on this at the moment.

[bill88t]

An actual 8mb sdcard, that has 6.8mb usable is reported to have fat12 content on a fat16 partition type.

My big boi n16r8 borb, reported to have fat16 on a fat12 partition type:

My s3tft, reported to have fat12 on fat12:

Both boards refuse to show up on android, even after I fsck them.

[RokeJulianLockhart]

Turns out that FAT16 does indeed work on my Fairphone 4. I just took a MicroSD card from my Nintendo DSi XL (used in that with a MicroSD to SD convertor, in case you're confused) and it worked without needing to be formatted.

[dhalbert]

Both boards refuse to show up on android, even after I fsck them.

This may be because the FAT16 CIRCUITPY presented by the larger flash chip still only has one FAT (File Allocation Table). Most FAT16 filesystems have two.

[HelenFoster]

A FAT16 USB flash drive is working with a Pixel 2 with Android 10.

[bill88t]

This may be because the FAT16 CIRCUITPY presented by the larger flash chip still only has one FAT (File Allocation Table). Most FAT16 filesystems have two.

Well, Android is very picky. It would stand to reason it expects it.

lib/oofatfs/ff.c:5424 -> const UINT n_fats = 1; /* Number of FATs for FAT/FAT32 volume (1 or 2) */

It's an optional feature for fat16, and an unsupported feature of fat12 as far as googling gets me.

Just changing this, neither board is read by android.
I have no way checking if the change is actually applied on a storage.erase_filesystem().

[eightycc]

I've thought about presenting a FAT16 filesystem that would appear to be larger than the actual capacity, but with a large number of bad blocks. The File Allocation Table has a mechanism for marking bad blocks. I did some experiments on this, but haven't made it a high priority. It's not so simple.

As @dhalbert noted earlier, whether an OS mounts a FAT volume as FAT12, FAT16, or FAT32 is determined by its total number of clusters. An OS will calculate the total number of clusters from information present in the VBR. The thresholds vary slightly depending on the OS, but are always close to: FAT12 < 4085 < FAT16 < 65,525 < FAT32.

The smallest volume that can present as FAT16 is 2MB where the cluster size is 512 bytes. For devices formatted with larger cluster sizes, the minimum size of the volume doubles as the cluster size doubles. For the commonly used cluster size of 2048 bytes, the minimum volume size is 8MB. Microsoft sets a minimum size for FAT16 at 8MB. Because MacOS Sonoma exhibits problems with auto-mounted FAT filesystems smaller than or equal to 8MB (see #8449), a good choice for minimum volume size is 8MB + physical volume size.

It's possible to increase the volume size in the VBR to any value we like, and then fence off the non-existent clusters by marking them bad in the FAT. Ballooning the volume size like this has some problems that need to be solved:

1. Since the OS "thinks" that all of the volume's clusters are real, it's possible for it to attempt to read or write non-existent clusters. For example, a dd of the volume will ignore the bad markings in the FAT and plow ahead into non-existent clusters. OS caching and read-ahead might also be a problem.

2. Ballooning the volume also balloons the FAT. The FAT is a table of 12, 16, or 32 bit entries indexed by the cluster number. Since the FAT grows by the number of clusters in the device balloon, its increased size can consume a considerable amount of real volume space.
   For example, an M0 Trinket has 64KB (128 x 512B sectors) of filesystem flash, so our ballooned volume size will be 8MB + 64KB or 16,512 x 512B clusters. The FAT in this case has grown from 1 x 512B sector to 65 x 512B sectors and now consumes more than half the available flash.

[eightycc]

@bill88t notes

Both boards refuse to show up on android, even after I fsck them.

Android may be looking at the fake MBR that CircuitPython presents as the first sector of the flash device, seeing that it's FAT12, and giving up before attempting to mount the filesystem. See flash_read_blocks()

[eightycc]

Since the OS "thinks" that all of the volume's clusters are real, it's possible for it to attempt to read or write non-existent clusters. For example, a dd of the volume will ignore the bad markings in the FAT and plow ahead into non-existent clusters. OS caching and read-ahead might also be a problem.

The solution to this problem is to make flash_read_blocks() and flash_write_blocks() aware of ballooning. Then, on a read from the ballooned area, return a block of zeroes. On a write to the ballooned area, ignore the write.

Accomplishing this requires a persistent indication of the presence of ballooning and its geometry so that the flash read and write routines can take the correct actions. The FAT filesystem stores its VBR (Volume Boot Record) as its first sector. The VBR describes the geometry of the underlying device as well as descriptive parameters for the filesystem itself. For historical reasons, a large potion of the VBR is reserved for a bootstrap loader and is unused. FAT ballooning utilizes a few of these unused bytes to store a signature along with the actual size of the device before ballooning.

When the filesystem is created, an additional parameter indicates to f_mkfs in oofatfs/fs.c the extent of desired ballooning, if any. f_mkfs creates the ballooned filesystem by adjusting the volume size and filling the corresponding ballooned area in the filesystem's FAT with bad block indications. f_mkfs also writes the ballooning information to the filesystem's VBR.

f_mount in oofatfs/fs.c reads the ballooning information from the VBR, validates its signature, and if valid writes it into struct mp_vfs_blockdev_t. Since struct mp_vfs_blockdev_t is a part of struct fs_user_mount_t, presence of ballooning and its geometry is readily available inside CircuitPython.

Finally, flash_read_blocks() and flash_write_blocks() use this ballooning information to take special action when non-existent blocks are read or written.

Additional modifications are made so that stat returns correct information for ballooned filesystems.

The fake MBR presented by flash_read_blocks() is updated to report the actual filesystem type and ballooned extent.

[dhalbert]

On a write to the ballooned area, ignore the write.

There should only be such a write if the OS were ignoring the bad cluster bits, such as the dd example, right? I would hope that OS reformatting of the volume (which we don't recommend) would not clear those bits.

[eightycc]

OS reformatting with mkfs or similar will kill ballooning completely and revert to FAT12 with all its original problems. Certainly not recommended. On the other hand, dd of the entire volume to a host OS file and back will work correctly.

If OS reformatting is an issue, it's possible to intercept a host OS write to the VBR and then force the bad flags back into the FAT as the host writes FAT blocks. Maybe that's a hack too far...

[eightycc]

See https://support.microsoft.com/en-us/topic/default-cluster-size-for-ntfs-fat-and-exfat-9772e6f1-e31a-00d7-e18f-73169155af95 for Microsoft's take on valid FAT filesystem geometry.

[dhalbert]

Also see https://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/fatgen103.doc for quite a complete writeup from Microsoft.

[eightycc]

Ran some experiments and found that Android does support FAT12, it just doesn't like partition type x01 (FAT12). By changing the partition type to x0e (FAT16 LBA), Android successfully mounts a FAT12 filesystem. Tested by modifying the fake MBR returned by CircuitPython.

[eightycc]

Linux does not care for the partition type/filesystem mismatch, so simply changing the partition type is not a viable fix. We do now know the reason for Android refusing to mount the CP filesystem, but will need ballooning or something similar for a fully compatible fix.