diff --git a/docs/library/esp32.rst b/docs/library/esp32.rst index a593965ae2..68379624e6 100644 --- a/docs/library/esp32.rst +++ b/docs/library/esp32.rst @@ -56,10 +56,14 @@ This class gives access to the partitions in the device's flash memory. Returns a 6-tuple ``(type, subtype, addr, size, label, encrypted)``. .. method:: Partition.readblocks(block_num, buf) +.. method:: Partition.readblocks(block_num, buf, offset) .. method:: Partition.writeblocks(block_num, buf) +.. method:: Partition.writeblocks(block_num, buf, offset) .. method:: Partition.ioctl(cmd, arg) - These methods implement the block protocol defined by :class:`uos.AbstractBlockDev`. + These methods implement the simple and :ref:`extended + ` block protocol defined by + :class:`uos.AbstractBlockDev`. .. method:: Partition.set_boot() diff --git a/docs/library/uos.rst b/docs/library/uos.rst index d8d8b7a972..84d341ac21 100644 --- a/docs/library/uos.rst +++ b/docs/library/uos.rst @@ -178,6 +178,35 @@ represented by VFS classes. Build a FAT filesystem on *block_dev*. +.. class:: VfsLfs1(block_dev) + + Create a filesystem object that uses the `littlefs v1 filesystem format`_. + Storage of the littlefs filesystem is provided by *block_dev*, which must + support the :ref:`extended interface `. + Objects created by this constructor can be mounted using :func:`mount`. + + See :ref:`filesystem` for more information. + + .. staticmethod:: mkfs(block_dev) + + Build a Lfs1 filesystem on *block_dev*. + +.. class:: VfsLfs2(block_dev) + + Create a filesystem object that uses the `littlefs v2 filesystem format`_. + Storage of the littlefs filesystem is provided by *block_dev*, which must + support the :ref:`extended interface `. + Objects created by this constructor can be mounted using :func:`mount`. + + See :ref:`filesystem` for more information. + + .. staticmethod:: mkfs(block_dev) + + Build a Lfs2 filesystem on *block_dev*. + +.. _littlefs v1 filesystem format: https://github.com/ARMmbed/littlefs/tree/v1 +.. _littlefs v2 filesystem format: https://github.com/ARMmbed/littlefs + Block devices ------------- @@ -187,9 +216,15 @@ implementation of this class will usually allow access to the memory-like functionality a piece of hardware (like flash memory). A block device can be used by a particular filesystem driver to store the data for its filesystem. +.. _block-device-interface: + +Simple and extended interface +............................. + There are two compatible signatures for the ``readblocks`` and ``writeblocks`` methods (see below), in order to support a variety of use cases. A given block -device may implement one form or the other, or both at the same time. +device may implement one form or the other, or both at the same time. The second +form (with the offset parameter) is referred to as the "extended interface". .. class:: AbstractBlockDev(...) @@ -247,64 +282,5 @@ device may implement one form or the other, or both at the same time. (*arg* is unused) - 6 -- erase a block, *arg* is the block number to erase -By way of example, the following class will implement a block device that stores -its data in RAM using a ``bytearray``:: - - class RAMBlockDev: - def __init__(self, block_size, num_blocks): - self.block_size = block_size - self.data = bytearray(block_size * num_blocks) - - def readblocks(self, block_num, buf): - for i in range(len(buf)): - buf[i] = self.data[block_num * self.block_size + i] - - def writeblocks(self, block_num, buf): - for i in range(len(buf)): - self.data[block_num * self.block_size + i] = buf[i] - - def ioctl(self, op, arg): - if op == 4: # get number of blocks - return len(self.data) // self.block_size - if op == 5: # get block size - return self.block_size - -It can be used as follows:: - - import uos - - bdev = RAMBlockDev(512, 50) - uos.VfsFat.mkfs(bdev) - vfs = uos.VfsFat(bdev) - uos.mount(vfs, '/ramdisk') - -An example of a block device that supports both signatures and behaviours of -the :meth:`readblocks` and :meth:`writeblocks` methods is:: - - class RAMBlockDev: - def __init__(self, block_size, num_blocks): - self.block_size = block_size - self.data = bytearray(block_size * num_blocks) - - def readblocks(self, block, buf, offset=0): - addr = block_num * self.block_size + offset - for i in range(len(buf)): - buf[i] = self.data[addr + i] - - def writeblocks(self, block_num, buf, offset=None): - if offset is None: - # do erase, then write - for i in range(len(buf) // self.block_size): - self.ioctl(6, block_num + i) - offset = 0 - addr = block_num * self.block_size + offset - for i in range(len(buf)): - self.data[addr + i] = buf[i] - - def ioctl(self, op, arg): - if op == 4: # block count - return len(self.data) // self.block_size - if op == 5: # block size - return self.block_size - if op == 6: # block erase - return 0 +See :ref:`filesystem` for example implementations of block devices using both +protocols. diff --git a/docs/reference/filesystem.rst b/docs/reference/filesystem.rst new file mode 100644 index 0000000000..71d34e7954 --- /dev/null +++ b/docs/reference/filesystem.rst @@ -0,0 +1,278 @@ +.. _filesystem: + +Working with filesystems +======================== + +.. contents:: + +This tutorial describes how MicroPython provides an on-device filesystem, +allowing standard Python file I/O methods to be used with persistent storage. + +MicroPython automatically creates a default configuration and auto-detects the +primary filesystem, so this tutorial will be mostly useful if you want to modify +the partitioning, filesystem type, or use custom block devices. + +The filesystem is typically backed by internal flash memory on the device, but +can also use external flash, RAM, or a custom block device. + +On some ports (e.g. STM32), the filesystem may also be available over USB MSC to +a host PC. :ref:`pyboard_py` also provides a way for the host PC to access to +the filesystem on all ports. + +Note: This is mainly for use on bare-metal ports like STM32 and ESP32. On ports +with an operating system (e.g. the Unix port) the filesystem is provided by the +host OS. + +VFS +--- + +MicroPython implements a Unix-like Virtual File System (VFS) layer. All mounted +filesystems are combined into a single virtual filesystem, starting at the root +``/``. Filesystems are mounted into directories in this structure, and at +startup the working directory is changed to where the primary filesystem is +mounted. + +On STM32 / Pyboard, the internal flash is mounted at ``/flash``, and optionally +the SDCard at ``/sd``. On ESP8266/ESP32, the primary filesystem is mounted at +``/``. + +Block devices +------------- + +A block device is an instance of a class that implements the +:class:`uos.AbstractBlockDev` protocol. + +Built-in block devices +~~~~~~~~~~~~~~~~~~~~~~ + +Ports provide built-in block devices to access their primary flash. + +On power-on, MicroPython will attempt to detect the filesystem on the default +flash and configure and mount it automatically. If no filesystem is found, +MicroPython will attempt to create a FAT filesystem spanning the entire flash. +Ports can also provide a mechanism to "factory reset" the primary flash, usually +by some combination of button presses at power on. + +STM32 / Pyboard +............... + +The :ref:`pyb.Flash ` class provides access to the internal flash. On some +boards which have larger external flash (e.g. Pyboard D), it will use that +instead. The ``start`` kwarg should always be specified, i.e. +``pyb.Flash(start=0)``. + +Note: For backwards compatibility, when constructed with no arguments (i.e. +``pyb.Flash()``), it only implements the simple block interface and reflects the +virtual device presented to USB MSC (i.e. it includes a virtual partition table +at the start). + +ESP8266 +....... + +The internal flash is exposed as a block device object which is created in the +``flashbdev`` module on start up. This object is by default added as a global +variable so it can usually be accessed simply as ``bdev``. This implements the +extended interface. + +ESP32 +..... + +The :class:`esp32.Partition` class implements a block device for partitions +defined for the board. Like ESP8266, there is a global variable ``bdev`` which +points to the default partition. This implements the extended interface. + +Custom block devices +~~~~~~~~~~~~~~~~~~~~ + +The following class implements a simple block device that stores its data in +RAM using a ``bytearray``:: + + class RAMBlockDev: + def __init__(self, block_size, num_blocks): + self.block_size = block_size + self.data = bytearray(block_size * num_blocks) + + def readblocks(self, block_num, buf): + for i in range(len(buf)): + buf[i] = self.data[block_num * self.block_size + i] + + def writeblocks(self, block_num, buf): + for i in range(len(buf)): + self.data[block_num * self.block_size + i] = buf[i] + + def ioctl(self, op, arg): + if op == 4: # get number of blocks + return len(self.data) // self.block_size + if op == 5: # get block size + return self.block_size + +It can be used as follows:: + + import os + + bdev = RAMBlockDev(512, 50) + os.VfsFat.mkfs(bdev) + os.mount(bdev, '/ramdisk') + +An example of a block device that supports both the simple and extended +interface (i.e. both signatures and behaviours of the +:meth:`uos.AbstractBlockDev.readblocks` and +:meth:`uos.AbstractBlockDev.writeblocks` methods) is:: + + class RAMBlockDev: + def __init__(self, block_size, num_blocks): + self.block_size = block_size + self.data = bytearray(block_size * num_blocks) + + def readblocks(self, block, buf, offset=0): + addr = block_num * self.block_size + offset + for i in range(len(buf)): + buf[i] = self.data[addr + i] + + def writeblocks(self, block_num, buf, offset=None): + if offset is None: + # do erase, then write + for i in range(len(buf) // self.block_size): + self.ioctl(6, block_num + i) + offset = 0 + addr = block_num * self.block_size + offset + for i in range(len(buf)): + self.data[addr + i] = buf[i] + + def ioctl(self, op, arg): + if op == 4: # block count + return len(self.data) // self.block_size + if op == 5: # block size + return self.block_size + if op == 6: # block erase + return 0 + +As it supports the extended interface, it can be used with :class:`littlefs +`:: + + import os + + bdev = RAMBlockDev(512, 50) + os.VfsLfs2.mkfs(bdev) + os.mount(bdev, '/ramdisk') + +Filesystems +----------- + +MicroPython ports can provide implementations of :class:`FAT `, +:class:`littlefs v1 ` and :class:`littlefs v2 `. + +The following table shows which filesystems are included in the firmware by +default for given port/board combinations, however they can be optionally +enabled in a custom firmware build. + +==================== ===== =========== =========== +Board FAT littlefs v1 littlefs v2 +==================== ===== =========== =========== +pyboard 1.0, 1.1, D Yes No Yes +Other STM32 Yes No No +ESP8266 Yes No No +ESP32 Yes No Yes +==================== ===== =========== =========== + +FAT +~~~ + +The main advantage of the FAT filesystem is that it can be accessed over USB MSC +on supported boards (e.g. STM32) without any additional drivers required on the +host PC. + +However, FAT is not tolerant to power failure during writes and this can lead to +filesystem corruption. For applications that do not require USB MSC, it is +recommended to use littlefs instead. + +To format the entire flash using FAT:: + + # ESP8266 and ESP32 + import os + os.umount('/') + os.VfsFat.mkfs(bdev) + os.mount(bdev, '/') + + # STM32 + import os, pyb + os.umount('/flash') + os.VfsFat.mkfs(pyb.Flash(start=0)) + os.mount(pyb.Flash(start=0), '/flash') + os.chdir('/flash') + +Littlefs +~~~~~~~~ + +Littlefs_ is a filesystem designed for flash-based devices, and is much more +resistant to filesystem corruption. + +Note: It can be still be accessed over USB MSC using the `littlefs FUSE +driver`_. Note that you must use the ``-b=4096`` option to override the block +size. + +.. _littlefs FUSE driver: https://github.com/ARMmbed/littlefs-fuse/tree/master/littlefs + +.. _Littlefs: https://github.com/ARMmbed/littlefs + +To format the entire flash using littlefs v2:: + + # ESP8266 and ESP32 + import os + os.umount('/') + os.VfsLfs2.mkfs(bdev) + os.mount(bdev, '/') + + # STM32 + import os, pyb + os.umount('/flash') + os.VfsLfs2.mkfs(pyb.Flash(start=0)) + os.mount(pyb.Flash(start=0), '/flash') + os.chdir('/flash') + +Hybrid (STM32) +~~~~~~~~~~~~~~ + +By using the ``start`` and ``len`` kwargs to :class:`pyb.Flash`, you can create +block devices spanning a subset of the flash device. + +For example, to configure the first 256kiB as FAT (and available over USB MSC), +and the remainder as littlefs:: + + import os, pyb + os.umount('/flash') + p1 = pyb.Flash(start=0, len=256*1024) + p2 = pyb.Flash(start=256*1024) + os.VfsFat.mkfs(p1) + os.VfsLfs2.mkfs(p2) + os.mount(p1, '/flash') + os.mount(p2, '/data') + os.chdir('/flash') + +This might be useful to make your Python files, configuration and other +rarely-modified content available over USB MSC, but allowing for frequently +changing application data to reside on littlefs with better resilience to power +failure, etc. + +The partition at offset ``0`` will be mounted automatically (and the filesystem +type automatically detected), but you can add:: + + import os, pyb + p2 = pyb.Flash(start=256*1024) + os.mount(p2, '/data') + +to ``boot.py`` to mount the data partition. + +Hybrid (ESP32) +~~~~~~~~~~~~~~ + +On ESP32, if you build custom firmware, you can modify ``partitions.csv`` to +define an arbitrary partition layout. + +At boot, the partition named "vfs" will be mounted at ``/`` by default, but any +additional partitions can be mounted in your ``boot.py`` using:: + + import esp32, os + p = esp32.Partition.find(esp32.Partition.TYPE_DATA, label='foo') + os.mount(p, '/foo') + diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 4dd52b9c83..1eaaa85c85 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -26,4 +26,5 @@ implementation and the best practices to use them. constrained.rst packages.rst asm_thumb2_index.rst + filesystem.rst pyboard.py.rst