Protocols are nice, but there is no way for C code to verify whether
a type's "protocol" structure actually implements some particular
protocol. As a result, you can pass an object that implements the
"vfs" protocol to one that expects the "stream" protocol, and the
opposite of awesomeness ensues.
This patch adds an OPTIONAL (but enabled by default) protocol identifier
as the first member of any protocol structure. This identifier is
simply a unique QSTR chosen by the protocol designer and used by each
protocol implementer. When checking for protocol support, instead of
just checking whether the object's type has a non-NULL protocol field,
use `mp_proto_get` which implements the protocol check when possible.
The existing protocols are now named:
protocol_framebuf
protocol_i2c
protocol_pin
protocol_stream
protocol_spi
protocol_vfs
(most of these are unused in CP and are just inherited from MP; vfs and
stream are definitely used though)
I did not find any crashing examples, but here's one to give a flavor of what
is improved, using `micropython_coverage`. Before the change,
the vfs "ioctl" protocol is invoked, and the result is not intelligible
as json (but it could have resulted in a hard fault, potentially):
>>> import uos, ujson
>>> u = uos.VfsPosix('/tmp')
>>> ujson.load(u)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: syntax error in JSON
After the change, the vfs object is correctly detected as not supporting
the stream protocol:
>>> ujson.load(p)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: stream operation not supported
Following other C-level protocols, this VFS protocol is added to help
abstract away implementation details of the underlying VFS in an efficient
way. As a starting point, the import_stat function is put into this
protocol so that the VFS sub-system does not need to know about every VFS
implementation in order to do an efficient stat for importing files.
In the future it might be worth adding other functions to this protocol.
Introduction of ports subdirectory where all ports are moved to
The main change in this release is the introduction of a "ports/"
subdirectory at the top-level of the repository, and all of the ports are
moved here. In the process the "stmhal" port is renamed to "stm32" to
better reflect the MCU that it targets. In addition, the STM32 CMSIS and
HAL sources are moved to a new submodule called "stm32lib".
The bytecode has changed in this release, compared to the previous release,
and as a consequence the .mpy version number has increased to version 3.
This means that scripts compiled with the previous mpy-cross must be
recompiled to work with this new version.
There have also been various enhancements and optimisations, such as:
check for valid UTF-8 when creating str objects, support for reverse
special binary operations like __radd__, full domain checking in the math
module, support for floor-division and modulo in the viper emitter,
and addition of stack overflow checking when executing a regex.
The stm32 port sees improved support for F7 MCUs, addition of a new board
B_L475E_IOT01A based on the STM32L475, and support for the Wiznet W5500
chipset along with improved socket behaviour.
A detailed list of changes follows.
py core:
- objstr: startswith, endswith: check arg to be a string
- nlrx86,x64: replace #define of defined() with portable macro usage
- objtype: handle NotImplemented return from binary special methods
- objtype: mp_obj_class_lookup: improve debug logging
- map: remove unused new/free functions
- make m_malloc_fail() have void return type, since it doesn't return
- modstruct: in struct.pack, stop converting if there are no args left
- modstruct: check and prevent buffer-read overflow in struct unpacking
- modstruct: check and prevent buffer-write overflow in struct packing
- nlrthumb: get working again on standard Thumb arch (ie not Thumb2)
- objfloat: fix binary ops with incompatible objects
- obj: fix comparison of float/complex NaN with itself
- objtype: implement fallback for instance inplace special methods
- objtuple: properly implement comparison with incompatible types
- objstr: add check for valid UTF-8 when making a str from bytes
- objlist: properly implement comparison with incompatible types
- runtime0.h: move relational ops to the beginning of mp_binary_op_t
- runtime0.h: move MP_BINARY_OP_DIVMOD to the end of mp_binary_op_t
- objtype: make sure mp_binary_op_method_name has full size again
- runtime0.h: put inplace arith ops in front of normal operations
- builtinhelp: simplify code slightly by extracting object type
- runtime: implement dispatch for "reverse op" special methods
- nlrx86: fix building for Android/x86
- builtinhelp: change signature of help text var from pointer to array
- runtime.h: change empty mp_warning macro so var-args are non empty
- modbuiltins: implement abs() by dispatching to MP_UNARY_OP_ABS
- {objfloat,objcomplex}: optimise MP_UNARY_OP_ABS by reusing variables
- mpconfig.h: add note that using computed gotos in VM is not C99
- objstr: strip: don't strip "\0" by default
- objexcept: prevent infinite recursion when allocating exceptions
- stream: remove unnecessary checks for NULL return from vstr_add_len
- vstr: raise a RuntimeError if fixed vstr buffer overflows
- vm: use lowercase letter at start of exception message
- persistentcode: define mp_raw_code_save_file() for any unix target
- add config option to print warnings/errors to stderr
- objfloat: support raising a negative number to a fractional power
- objset: simplify set and frozenset by separating their locals dicts
- objset: check that RHS of a binary op is a set/frozenset
- objset: include the failed key in a KeyError raised from set.remove
- objtype: change type of enum-to-qstr table to uint16_t to save space
- objstr: make empty bytes object have a null-terminating byte
- mpprint: only check for null string printing when NDEBUG not defined
- objtype: clean up unary- and binary-op enum-to-qstr mapping tables
- persistentcode: bump .mpy version number to version 3
- bc: update opcode_format_table to match the bytecode
- modmath: add full checks for math domain errors
- modmath: convert log2 macro into a function
- formatfloat: don't print the negative sign of a NaN value
- formatfloat: use standard isinf, isnan funcs instead of custom ones
- modbuiltins: use existing utf8_get_char helper in builtin ord func
- emitnative: implement floor-division and modulo for viper emitter
- objtype: use CPython compatible method name for sizeof
- objtype: fit qstrs for special methods in byte type
- objtype: define all special methods if requested
- objtype: introduce MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS
extmod:
- modubinascii: only include uzlib/tinf.h when it's really needed
- modussl_mbedtls: allow to compile with MBEDTLS_DEBUG_C disabled
- machine_pinbase: put PinBase singleton in ROM
- re1.5: upgrade to v0.8.2, adds hook for stack overflow checking
- modure: add stack overflow checking when executing a regex
- uos_dupterm: update uos.dupterm() and helper funcs to have index
- uos_dupterm: swallow any errors from dupterm closing the stream
- vfs: replace VLA in proxy func with small, static sized array
- modussl: add finaliser support for ussl objects
- modussl_mbedtls: allow to compile with unix coverage build
lib:
- add new submodule, stm32lib containing STM32 CMSIS and HAL source
- embed/abort_: use mp_raise_msg helper function
- libm: fix tanhf so that it correctly handles +/- infinity args
- libm: remove implementation of log2f, use MP_NEED_LOG2 instead
- axtls: update, support for SSL_EAGAIN return code
- berkeley-db-1.xx: update, allow to override MINCACHE, DEFPSIZE
drivers:
- memory/spiflash: change from hard-coded soft SPI to generic SPI
- display/ssd1306.py: improve performance of graphics methods
- nrf24l01: make nRF24L01 test script more portable
- display/ssd1306: implement SSD1306_I2C poweron method
- display/ssd1306: make poweron() work the same with SSD1306_SPI
- wiznet5k: improve the performance of socket ops with threading
- wiznet5k: get low-level W5500 driver working
tools:
- upip: upgrade to 1.2.2
- pyboard: use repr() when quoting data in error messages
- pyboard: update docstring for additional device support
tests:
- object_new: better messages, check user __new__() method
- class_new: add checks for __init__ being called and other improvements
- class_new: add another testcase for __new__/__init__ interaction
- class_inplace_op: test for inplace op fallback to normal one
- run-bench-tests: update locations of executables, now in ports/
- class_reverse_op: test for reverse arith ops special methods
- run-tests: skip class_inplace_op for minimal profile
- run-tests: fix copy-paste mistake in var name
- cpydiff: add cases for locals() discrepancies
- extmod: add test for ure regexes leading to infinite recursion
- extmod: add test for '-' in character class in regex
- run-tests: close device under test using "finally"
- net_inet: update tls test to work with CPython and incl new site
unix port:
- rename modsocket.c to modusocket.c
- modusocket: remove #if MICROPY_SOCKET_EXTRA code blocks
- enable MICROPY_PY_REVERSE_SPECIAL_METHODS
stm32 port:
- modmachine: make machine.bootloader() work when MPU is enabled
- modmachine: improve support for sleep/deepsleep on F7 MCUs
- compute PLL freq table during build instead of at run time
- modmachine: for F7 MCU, save power by reducing internal volt reg
- boards/pllvalues.py: make script work with both Python 2 and 3
- Makefile: use lib/stm32lib instead of local cmsis and hal files
- remove cmsis and hal files, they are now a submodule
- Makefile: automatically fetch stm32lib submodule if needed
- update to new STM Cube HAL library
- fix clock initialisation of L4 MCUs
- rename stmhal port directory to stm32
- remove unused usbd_msc.c file
- boards: change remaining stm32f4xx_hal_conf.h to unix line ending
- boards: change linker scripts to use "K" instead of hex byte size
- boards: fix I2C1 pin mapping on NUCLEO_F401RE/F411RE boards
- i2c: when scanning for I2C devices only do 1 probe per address
- modnwwiznet5k: release the GIL on blocking network operations
- boards: add new board B_L475E_IOT01A based on STM32L475
- make-stmconst.py: make sure mpz const data lives in ROM
- timer: make pyb.Timer() instances persistent
- mpconfigport.h: add configuration for max periphs on L4 series
- usbdev: make the USBD callback struct const so it can go in ROM
- usbdev: change static function variable to non-static
- usbdev: put all CDC state in a struct
- usbdev: put all HID state in a struct
- usbdev: simplify CDC tx/rx buffer passing
- usbdev: simplify HID tx/rx buffer passing
- usbdev/core: add state parameter to all callback functions
- usbdev: put all state for the USB device driver in a struct
- usbdev: simplify pointers to MSC state and block dev operations
- usbdev: merge all global USB device state into a single struct
- usbdev: make device descriptor callbacks take a state pointer
- usbdev: move all the USB device descriptor state into its struct
- timer: enable ARPE so that timer freq can be changed smoothly
- modnwwiznet5k: get the IP address of an established socket
- boards: fix typos in stm32f767_af.csv table
- usbd_cdc_interface: don't reset CDC output buf on initialisation
- modnwwiznet5k: implement WIZNET5K.isconnected() method
- modusocket: make getaddrinfo() work when passed an IP address
- modusocket: return OSError(-2) if getaddrinfo fails
- mpconfigport.h: add MICROPY_THREAD_YIELD() macro
- modnwwiznet5k: add support for W5500 Ethernet chip
- modnwwiznet5k: increase SPI bus speed to 42MHz
- modnwwiznet5k: implement stream ioctl for the Wiznet driver
- mphalport: improve efficiency of mp_hal_stdout_tx_strn_cooked
- make uos.dupterm() conform to specs by using extmod version
cc3200 port:
- enable micropython.kbd_intr() method
- use standard implementation of keyboard interrupt
esp8266 port:
- rename axtls_helpers.c to posix_helpers.c
- posix_helpers: set ENOMEM on memory alloc failure
- set DEFPSIZE=1024, MINCACHE=3 for "btree" module
- esp_mphal: send data in chunks to mp_uos_dupterm_tx_strn
- modnetwork: add "bssid" keyword arg to WLAN.connect() method
- modules/webrepl_setup: add info about allowed password length
zephyr port:
- Makefile: revamp "test" target after ports were moved to ports/
- use CONFIG_NET_APP_SETTINGS to setup initial network addresses
- switch to interrupt-driven pull-style console
pic16bit port:
- add definition of SEEK_SET to unistd.h
docs:
- pyboard/tutorial: add "timeout=0" to UART in pass-through example
- more xrefs to "MicroPython port" in glossary
- library/network: fix ref to "socket" module (should be "usocket")
- machine.Signal: improve style/grammar and add usage example
- library: add description of "index" parameter to uos.dupterm()
- library/micropython: fix typo in RST formatting
- library/framebuf.rst: generalise constructor to all colour formats
- btree: describe page caching policy of the underlying implementation
- esp8266/tutorial: update neopixel with example of using 4 bbp
- library/network: clarify usage of "bssid" arg in connect() method
- pyboard/quickref: add info for Switch, RTC, CAN, Accel classes
- pyboard/tutorial: update now that yellow LED also supports PWM
- esp8266/quickref: add quickref info for RTC class
- library: add missing cross-ref links for classes in pyb module
- library/network: update docs to state that W5500 is supported
- uselect: document one-shot polling mode
- usocket: elaborate descriptions
- usocket: document inet_ntop(), inet_pton()
- library/network: add dhcp_hostname parameter
- reference/isr_rules: minor typo correction
- ussl: fix module name refs and use "MicroPython port" term
- esp8266/general: add section on TLS limitations
- usocket: document that settimeout() isn't supported by all ports
- ure: add "|" (alternative) to the list of supported operators
- reference/isr_rules.rst: add tutorial on use of micropython.schedule()
travis:
- use --upgrade when pip is installing cpp-coveralls
- update build command now that stm32 Wiznet config has changed
examples:
- hwconfig_console: add .on()/.off() methods
all:
- convert mp_uint_t to mp_unary_op_t/mp_binary_op_t where appropriate
- convert remaining "mp_uint_t n_args" to "size_t n_args"
- make new ports/ sub-directory and move all ports there
- update Makefiles and others to build with new ports/ dir layout
- remove inclusion of internal py header files
- use NULL instead of "" when calling mp_raise exception helpers
README:
- update "Dependencies" section
- add explicit section on contributing
- add gcc and arm-none-eabi-newlib to list of required components
.gitattributes:
- remove obsolete entries for stmhal/hal, stmhal/cmsis
- add entries for files that will move to ports/ dir
VLAs can be expensive on stack usage due to stack alignment requirements,
and also the fact that extra local variables are needed to track the
dynamic size of the stack. So using fixed-size arrays when possible can
help to reduce code size and stack usage.
In this particular case, the maximum value of n_args in the VLA is 2 and so
it's more efficient to just allocate this array with a fixed size. This
reduces code size by around 30 bytes on Thumb2 and Xtensa archs. It also
reduces total stack usage of the function: on Thumb2 the usage with VLA is
between 40 and 48 bytes, which is reduced to 32; on Xtensa, VLA usage is
between 64 and 80 bytes, reduced to 32; on x86-64 it's at least 88 bytes
reduced to 80.
Header files that are considered internal to the py core and should not
normally be included directly are:
py/nlr.h - internal nlr configuration and declarations
py/bc0.h - contains bytecode macro definitions
py/runtime0.h - contains basic runtime enums
Instead, the top-level header files to include are one of:
py/obj.h - includes runtime0.h and defines everything to use the
mp_obj_t type
py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
and defines everything to use the general runtime support functions
Additional, specific headers (eg py/objlist.h) can be included if needed.
This patch allows mounting of VFS objects right at the root directory, eg
os.mount(vfs, '/'). It still allows VFS's to be mounted at a path within
the root, eg os.mount(vfs, '/flash'), and such mount points will override
any paths within a VFS that is mounted at the root.
For example, if the current directory is the root dir then this patch
allows one to do uos.listdir('mnt'), where 'mnt' is a valid mount point.
Previous to this patch such a thing would not work, on needed to do
uos.listdir('/mnt') instead.
If the mounted object doesn't have a "mount" method then assume it's a
block device and try to detect the filesystem. Since we currently only
support FAT filesystems, the behaviour is to just try and create a VfsFat
object automatically, using the given block device.
This provides mp_vfs_XXX functions (eg mount, open, listdir) which are
agnostic to the underlying filesystem type, and just require an object with
the relevant filesystem-like methods (eg .mount, .open, .listidr) which can
then be mounted.
These mp_vfs_XXX functions would typically be used by a port to implement
the "uos" module, and mp_vfs_open would be the builtin open function.
This feature is controlled by MICROPY_VFS, disabled by default.
Jeff Epler
Scott Shawcroft
Dan Halbert
Damien George
Tom Collins
Scott Shawcroft