F-strings, new machine.I2S class, ESP32-C3 support and LEGO_HUB_NO6 board
This release of MicroPython adds support for f-strings (PEP-498), with a
few limitations compared to normal Python. F-strings are essentially
syntactic sugar for "".format() and make formatting strings a lot more
convenient. Other improvements to the core runtime include pretty printing
OSError when it has two arguments (an errno code and a string), scheduling
of KeyboardInterrupt on the main thread, and support for a single argument
to the optimised form of StopIteration.
In the machine module a new I2S class has been added, with support for
esp32 and stm32 ports. This provides a consistent API for transmit and
receive of audio data in blocking, non-blocking and asyncio-based
operation. Also, the json module has support for the "separators" argument
in the dump and dumps functions, and framebuf now includes a way to blit
between frame buffers of different formats using a palette. A new,
portable machine.bitstream function is also added which can output a stream
of bits with configurable timing, and is used as the basis for driving
WS2812 LEDs in a common way across ports.
There has been some restructuring of the repository directory layout, with
all third-party code now in the lib/ directory. And a new top-level
directory shared/ has been added with first-party code that was previously
in lib/ moved there.
The docs have seen further improvement with enhancements and additions to
the rp2 parts, as well as a new quick reference for the zephyr port.
The terms master/slave have been replaced with controller/peripheral,
mainly relating to I2C and SPI usage. And u-module references have been
replaced with just the module name without the u-prefix to help clear up
the intended usage of modules in MicroPython.
For the esp8266 and esp32 ports, hidden networks are now included in WLAN
scan results. On the esp32 the RMT class is enhanced with idle_level and
write_pulses modes. There is initial support for ESP32-C3 chips with
GENERIC_C3 and GENERIC_C3_USB boards.
The javascript port has had its Makefile and garbage collector
implementation reworked so it compiles and runs with latest the Emscripten
using asyncify.
The mimxrt port sees the addition of hardware I2C and SPI support, as well
as some additional methods to the machine module. There is also support
for Hyperflash chips.
The nrf port now has full VFS storage support, enables source-line on
traceback, and has .mpy features consistent with other ports.
For the rp2 port there is now more configurability for boards, and more
boards added.
The stm32 port has a new LEGO_HUB_NO6 board definition with detailed
information how to get this LEGO Hub running stock MicroPython. There is
also now support to change the CPU frequency on STM32WB MCUs. And USBD_xxx
descriptor options have been renamed to MICROPY_HW_USB_xxx.
Thanks to everyone who contributed to this release: Amir Gonnen, Andrew
Scheller, Bryan Tong Minh, Chris Wilson, Damien George, Daniel Mizyrycki,
David Lechner, David P, Fernando, finefoot, Frank Pilhofer, Glenn Ruben
Bakke, iabdalkader, Jeff Epler, Jim Mussared, Jonathan Hogg, Josh Klar,
Josh Lloyd, Julia Hathaway, Krzysztof Adamski, Matúš Olekšák, Michael
Weiss, Michel Bouwmans, Mike Causer, Mike Teachman, Ned Konz, NitiKaur,
oclyke, Patrick Van Oosterwijck, Peter Hinch, Peter Züger, Philipp
Ebensberger, robert-hh, Roberto Colistete Jr, Sashkoiv, Seon Rozenblum,
Tobias Thyrrestrup, Tom McDermott, Will Sowerbutts, Yonatan Goldschmidt.
What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.
Main components
===============
all:
- fix signed shifts and NULL access errors from -fsanitize=undefined
- update to point to files in new shared/ directory
py core:
- mpstate: make exceptions thread-local
- mpstate: schedule KeyboardInterrupt on main thread
- mperrno: add MP_ECANCELED error code
- makeqstrdefs.py: don't include .h files explicitly in preprocessing
- mark unused arguments from bytecode decoding macros
- objexcept: pretty print OSError also when it has 2 arguments
- makeversionhdr: add --tags arg to git describe
- vm: simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode
- objexcept: make mp_obj_exception_get_value support subclassed excs
- support single argument to optimised MP_OBJ_STOP_ITERATION
- introduce and use mp_raise_type_arg helper
- modsys: optimise sys.exit for code size by using exception helpers
- objexcept: make mp_obj_new_exception_arg1 inline
- obj: fix formatting of comment for mp_obj_is_integer
- emitnative: reuse need_reg_all func in need_stack_settled
- emitnative: ensure stack settling is safe mid-branch
- runtime: fix bool unary op for subclasses of native types
- builtinimport: fix condition for including do_execute_raw_code()
- mkrules: automatically build mpy-cross if it doesn't exist
- implement partial PEP-498 (f-string) support
- lexer: clear fstring_args vstr on lexer free
- mkrules.mk: do submodule sync in "make submodules"
extmod:
- btstack: add missing call to mp_bluetooth_hci_uart_deinit
- btstack: check that BLE is active before performing operations
- uasyncio: get addr and bind server socket before creating task
- axtls-include: add axtls_os_port.h to customise axTLS
- update for move of crypto-algorithms, re1.5, uzlib to lib
- moduselect: conditionally compile select()
- nimble: fix leak in l2cap_send if send-while-stalled
- btstack/btstack.mk: use -Wno-implicit-fallthrough, not =0
- utime: always invoke mp_hal_delay_ms when >= to 0ms
- modbluetooth: clamp MTU values to 32->UINT16_MAX
- nimble: allow modbluetooth binding to hook "sent HCI packet"
- nimble: add "memory stalling" mechanism for l2cap_send
- uasyncio: in open_connection use address info in socket creation
- modujson: add support for dump/dumps separators keyword-argument
- modlwip: fix close and clean up of UDP and raw sockets
- modbluetooth: add send_update arg to gatts_write
- add machine.bitstream
- modframebuf: enable blit between different formats via a palette
lib:
- tinyusb: update to version 0.10.1
- pico-sdk: update to version 1.2.0
- utils/stdout_helpers: make mp_hal_stdout_tx_strn_cooked efficient
- axtls: switch to repo at micropython/axtls
- axtls: update to latest axtls 2.1.5 wih additional commits
- re1.5: move re1.5 code from extmod to lib
- uzlib: move uzlib code from extmod to lib
- crypto-algorithms: move crypto-algorithms code from extmod to lib
- update README's based on contents of these dirs
drivers:
- neopixel: add common machine.bitstream-based neopixel module
- neopixel: optimize fill() for speed
- neopixel: reduce code size of driver
- cyw43: fix cyw43_deinit so it can be called many times in a row
- cyw43: make wifi join fail if interface is not active
mpy-cross:
- disable stack check when building with Emscripten
Support components
==================
docs:
- library: document new esp32.RMT features and fix wait_done
- library: warn that ustruct doesn't handle spaces in format strings
- esp8266/tutorial: change flash mode from dio to dout
- replace master/slave with controller/peripheral in I2C and SPI
- rp2: enhance quickref and change image to Pico pinout
- rp2: update general section to give a brief technical overview
- library/utime.rst: clarify behaviour and precision of sleep ms/us
- library/uasyncio.rst: document stream readexactly() method
- library/machine.I2S.rst: fix use of sd pin in examples
- zephyr: add quick reference for the Zephyr port
- library/zephyr: add libraries specific to the Zephyr port
- templates: add unix and zephyr quickref links to top-index
- rename ufoo.rst to foo.rst
- replace ufoo with foo in all docs
- library/index.rst: clarify module naming and purpose
- library/builtins.rst: add module title
- library/network.rst: simplify socket import
- add docs for machine.bitstream and neopixel module
- library: fix usage of :term: for frozen module reference
- esp8266: use monospace for software tools
- reference: mention that slicing a memoryview causes allocation
examples: no changes specific to this component/port
tests:
- extmod: make uasyncio_heaplock test more deterministic
- cpydiff/modules_struct_whitespace_in_format: run black
- extmod/ujson: add tests for dump/dumps separators argument
- run-multitests.py: add broadcast and wait facility
- multi_bluetooth/ble_subscribe.py: add test for subscription
- extmod/vfs_fat_finaliser.py: ensure alloc at never-used GC blocks
- basics: split f-string debug printing to separate file with .exp
- pybnative: make while.py test run on boards without pyb.delay
tools:
- autobuild: add scripts to build release firmware
- remove obsolete build-stm-latest.sh script
- ci.sh: run apt-get update in ci_powerpc_setup
- makemanifest.py: allow passing flags to mpy-tool.py
- autobuild: add mimxrt port to build scripts for nightly builds
- pyboard.py: add cmd-line option to make soft reset configurable
- mpremote: swap order of PID and VID in connect-list output
- ci.sh: build unix dev variant as part of macOS CI
- ci.sh: build GENERIC_C3 board as part of esp32 CI
- autobuild: use separate IDF version to build newer esp32 SoCs
- autobuild: add FeatherS2 and TinyS2 to esp32 auto builds
- mpremote: add seek whence for mounted files
- mpremote: raise OSError on unsupported RemoteFile.seek
- autobuild: add the MIMXRT1050_EVKB board to the daily builds
- ci.sh: add mpy-cross build to nrf port
- codeformat.py: include ports/nrf/modules/nrf in code formatting
- gen-cpydiff.py: don't rename foo to ufoo in diff output
- autobuild: add auto build for Silicognition wESP32
- mpremote: fix connect-list in case VID/PID are None
- mpremote: add "devs" shortcut for "connect list"
- mpremote: remove support for pyb.USB_VCP in/out specialisation
- autobuild: don't use "-B" for make, it's already a fresh build
- pyboard.py: move --no-exclusive/--soft-reset out of mutex group
- pyboard.py: make --no-follow use same variable as --follow
- pyboard.py: add --exclusive to match --no-exclusive
- pyboard.py: make --no-soft-reset consistent with other args
- uncrustify: force 1 newline at end of file
- mpremote: bump version to 0.0.6
CI:
- workflows: add workflow to build and test javascript port
- workflows: switch from Coveralls to Codecov
- workflows: switch from lcov to gcov
- workflows: add workflow to build and test unix dev variant
The ports
=========
all ports:
- use common mp_hal_stdout_tx_strn_cooked instead of custom one
- update for move of crypto-algorithms, uzlib to lib
- rename USBD_VID/PID config macros to MICROPY_HW_USB_VID/PID
bare-arm port: no changes specific to this component/port
cc3200 port: no changes specific to this component/port
esp8266 port:
- add __len__ to NeoPixel driver to support iterating
- Makefile: add more libm files to build
- include hidden networks in WLAN.scan results
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
esp32 port:
- boards/sdkconfig.base: disable MEMPROT_FEATURE to alloc from IRAM
- add __len__ to NeoPixel driver to support iterating
- main: allow MICROPY_DIR to be overridden
- esp32_rmt: fix RMT looping in newer IDF versions
- esp32_rmt: enhance RMT with idle_level and write_pulses modes
- add new machine.I2S class for I2S protocol support
- machine_spi: calculate actual attained baudrate
- machine_hw_spi: use a 2 item SPI queue for long transfers
- machine_dac: add MICROPY_PY_MACHINE_DAC option, enable by default
- machine_i2s: add MICROPY_PY_MACHINE_I2S option, enable by default
- fix use of mp_int_t, size_t and uintptr_t
- add initial support for ESP32C3 SoCs
- boards/GENERIC_C3: add generic C3-based board
- modmachine: release the GIL in machine.idle()
- mphalport: always yield at least once in delay_ms
- machine_uart: add flow kw-arg to enable hardware flow control
- boards: add Silicognition wESP32 board configuration
- mpconfigport.h: enable reverse and inplace special methods
- include hidden networks in WLAN.scan results
- makeimg.py: get bootloader and partition offset from sdkconfig
- enable MICROPY_PY_FSTRINGS by default
- machine_hw_spi: release GIL during transfers
- machine_pin: make check for non-output pins respect chip variant
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
- boards: add GENERIC_C3_USB board with USB serial/JTAG support
javascript port:
- rework Makefile and GC so it works with latest Emscripten
- Makefile: suppress compiler errors from array bounds
- Makefile: change variable to EXPORTED_RUNTIME_METHODS
mimxrt port:
- move calc_weekday helper function to timeutils
- machine_spi: add the SPI class to the machine module
- moduos: seed the PRNG on boot using the TRNG
- boards: set vfs partition start to 1 MBbyte
- main: skip running main.py if boot.py failed
- main: extend the information returned by help()
- mimxrt_flash: remove commented-out code
- modmachine: add a few minor methods to the machine module
- machine_led: use mp_raise_msg_varg helper
- machine_i2c: add hardware-based machine.I2C to machine module
- add support for Hyperflash chips
- boards: add support for the MIMXRT1050_EVKB board
- machine_pin: implement ioctl for Pin
minimal port:
- Makefile: add support for building with user C modules
nrf port:
- modules: replace master/slave with controller/peripheral in SPI
- boards/common.ld: calculate unused flash region
- modules/nrf: add new nrf module with flash block device
- drivers: add support for using flash block device with SoftDevice
- mpconfigport.h: expose nrf module when MICROPY_PY_NRF is set
- README: update README.md to reflect internal file systems
- mpconfigport.h: tune FAT FS configuration
- Makefile: add _fs_size linker script override from make
- modules/uos: allow a board to configure MICROPY_VFS_FAT/LFS1/LFS2
- mpconfigport.h: enable MICROPY_PY_IO_FILEIO when an FS is enabled
- qstrdefsport.h: add entries for in-built FS mount points
- main: add auto mount and auto format hook for internal flash FS
- boards: enable needed features for FAT/LFS1/LFS2
- facilitate use of freeze manifest
- boards: set FROZEN_MANIFEST blank when SD present on nrf51 targets
- modules/scripts: add file system formatting script
- Makefile: set default manifest file for all targets
- mphalport: add dummy function for mp_hal_time_ns()
- boards: enable MICROPY_VFS_LFS2 for all target boards
- modules/uos: add ilistdir to uos module
- modules/nrf: add function to enable/disable DCDC
- enable source line on tracebacks
- set .mpy features consistent with documentation and other ports
pic16bit port: no changes specific to this component/port
powerpc port: no changes specific to this component/port
qemu-arm port: no changes specific to this component/port
rp2 port:
- use 0=Monday datetime convention in RTC
- machine_rtc: in RTC.datetime, compute weekday automatically
- CMakeLists.txt: suppress compiler errors for pico-sdk and tinyusb
- tusb_config.h: set CFG_TUD_CDC_EP_BUFSIZE to 256
- machine_uart: add hardware flow control support
- machine_uart: allow overriding default machine UART pins
- machine_i2c: allow boards to configure I2C pins using new macros
- machine_spi: allow boards to configure SPI pins using new macros
- machine_uart: fix poll ioctl to also check hardware FIFO
- machine_uart: fix read when FIFO has chars but ringbuf doesn't
- tusb_port: allow boards to configure USB VID and PID
- boards/ADAFRUIT_FEATHER_RP2040: configure custom VID/PID
- boards/ADAFRUIT_FEATHER_RP2040: configure I2C/SPI default pins
- boards/SPARKFUN_PROMICRO: configure UART/I2C/SPI default pins
- boards/SPARKFUN_THINGPLUS: configure I2C/SPI default pins
- boards: add Adafruit ItsyBitsy RP2040
- boards: add Adafruit QT Py RP2040
- boards: add Pimoroni Pico LiPo 4MB
- boards: add Pimoroni Pico LiPo 16MB
- boards: add Pimoroni Tiny 2040
- CMakeLists.txt: allow a board's cmake to set the manifest path
- enable MICROPY_PY_FSTRINGS by default
- Makefile: add "submodules" target, to match other ports
- rp2_flash: disable IRQs while calling flash_erase/program
- CMakeLists.txt: add option to enable double tap reset to bootrom
- mpconfigport.h: allow boards to add root pointers
samd port:
- add support for building with user C modules
stm32 port:
- softtimer: add soft_timer_reinsert() helper function
- mpbthciport: change from systick to soft-timer for BT scheduling
- provide a custom BTstack runloop that integrates with soft timer
- usb: make irq's default trigger enable all events
- boardctrl: skip running main.py if boot.py had an error
- sdio: fix undefined reference to DMA stream on H7
- dma: add DMAMUX configuration for H7 to fix dma_nohal_init
- main: call mp_deinit() at end of main
- adc: allow using ADC12 and ADC3 for H7
- adc: define the ADC instance used for internal channels
- adc: simplify and generalise how pin_adcX table is defined
- add new machine.I2S class for I2S protocol support
- boards/NUCLEO_F446RE: fix I2C1 pin assignment to match datasheet
- replace master/slave with controller/peripheral in I2C and SPI
- systick: always POLL_HOOK when delaying for milliseconds
- sdram: make SDRAM test cache aware, and optional failure with msg
- boards/NUCLEO_F446RE: enable CAN bus support
- boards: add support for SparkFun STM32 MicroMod Processor board
- uart: fix LPUART1 baudrate set/get
- uart: support low baudrates on LPUART1
- boards/STM32F429DISC: set correct UART2 pins and add UART3/6
- boards/NUCLEO_F439ZI: add board definition for NUCLEO_F439ZI
- boards/LEGO_HUB_NO6: add board definition for LEGO_HUB_NO6
- Makefile: update to only pull in used Bluetooth library
- README.md: update supported MCUs, and submodule and mboot use
- usbd_desc: rename USBD_xxx descriptor opts to MICROPY_HW_USB_xxx
- usbd_cdc_interface: rename USBD_CDC_xx opts to MICROPY_HW_USB_xx
- powerctrl: support changing frequency on WB MCUs
- boards/NUCLEO_H743ZI2: add modified version of NUCLEO_H743ZI
- mbedtls: fix compile warning about uninitialized val
- enable MICROPY_PY_FSTRINGS by default
- add implementation of machine.bitstream
- Makefile: allow GIT_SUBMODULES and LIBS to be extended
- stm32_it: support TIM17 IRQs on WB MCUs
- disable computed goto on constrained boards
- storage: make extended-block-device more configurable
- boards/LEGO_HUB_NO6: change SPI flash storage to use hardware SPI
- boards/LEGO_HUB_NO6: skip first 1MiB of SPI flash for storage
- boards/LEGO_HUB_NO6: add make commands to backup/restore firmware
teensy port: no changes specific to this component/port
unix port:
- modffi: add option to lock GC in callback, and cfun access
- Makefile: add back LIB_SRC_C to list of object files
- variants: enable help and help("modules") on standard and dev
- Makefile: disable error compression on arm-linux-gnueabi-gcc
windows port:
- Makefile: add .exe extension to executables name
- appveyor: update to VS 2017 and use Python 3.8 for build/test
zephyr port:
- machine_spi: add support for hardware SPI
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.
It is implemented in the lexer as a syntax translation to `str.format`:
f"{a}" --> "{}".format(a)
It also supports:
f"{a=}" --> "a={}".format(a)
This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.
There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
`SyntaxError: raw f-strings are not supported`.
- literal concatenation of f-strings with adjacent strings will fail
"{}" f"{a}" --> "{}{}".format(a) (str.format will incorrectly use
the braces from the non-f-string)
f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)
- PEP-498 requires the full parser to understand the interpolated
argument, however because this entirely runs in the lexer it cannot
resolve nested braces in expressions like
f"{'}'}"
- The !r, !s, and !a conversions are not supported.
Includes tests and cpydiffs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Constant expression like "2 ** 3" will now be folded, and the special form
"X = const(2 ** 3)" will now compile because the argument to the const is
now a constant.
Fixes issue #5865.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
In this part of the code there is no way to get the ** operator, so no need
to check for it.
This commit also adds tests for this, and other related, invalid const
operations.
This should reclaim *most* code space added to handle f-strings.
However, there may be some small code growth as parse_string_literal
takes a new parameter (which will always be 0, so hopefully the optimizer
eliminates it)
This implements (most of) the PEP-498 spec for f-strings, with two
exceptions:
- raw f-strings (`fr` or `rf` prefixes) raise `NotImplementedError`
- one special corner case does not function as specified in the PEP
(more on that in a moment)
This is implemented in the core as a syntax translation, brute-forcing
all f-strings to run through `String.format`. For example, the statement
`x='world'; print(f'hello {x}')` gets translated *at a syntax level*
(injected into the lexer) to `x='world'; print('hello {}'.format(x))`.
While this may lead to weird column results in tracebacks, it seemed
like the fastest, most efficient, and *likely* most RAM-friendly option,
despite being implemented under the hood with a completely separate
`vstr_t`.
Since [string concatenation of adjacent literals is implemented in the
lexer](534b7c368d),
two side effects emerge:
- All strings with at least one f-string portion are concatenated into a
single literal which *must* be run through `String.format()` wholesale,
and:
- Concatenation of a raw string with interpolation characters with an
f-string will cause `IndexError`/`KeyError`, which is both different
from CPython *and* different from the corner case mentioned in the PEP
(which gave an example of the following:)
```python
x = 10
y = 'hi'
assert ('a' 'b' f'{x}' '{c}' f'str<{y:^4}>' 'd' 'e') == 'ab10{c}str< hi >de'
```
The above-linked commit detailed a pretty solid case for leaving string
concatenation in the lexer rather than putting it in the parser, and
undoing that decision would likely be disproportionately costly on
resources for the sake of a probably-low-impact corner case. An
alternative to become complaint with this corner case of the PEP would
be to revert to string concatenation in the parser *only when an
f-string is part of concatenation*, though I've done no investigation on
the difficulty or costs of doing this.
A decent set of tests is included. I've manually tested this on the
`unix` port on Linux and on a Feather M4 Express (`atmel-samd`) and
things seem sane.
Before this, such names would instead cause an assertion error inside
qstr_from_strn.
A simple reproducer is a python source file containing the letter "a"
repeated 256 times
This string is recognised by uncrustify, to disable formatting in the
region marked by these comments. This is necessary in the qstrdef*.h files
to prevent modification of the strings within the Q(...). In other places
it is used to prevent excessive reformatting that would make the code less
readable.
To make progress towards MicroPython supporting Python 3.5, adding the
matmul operator is important because it's a really "low level" part of the
language, being a new token and modifications to the grammar.
It doesn't make sense to make it configurable because 1) it would make the
grammar and lexer complicated/messy; 2) no other operators are
configurable; 3) it's not a feature that can be "dynamically plugged in"
via an import.
And matmul can be useful as a general purpose user-defined operator, it
doesn't have to be just for numpy use.
Based on work done by Jim Mussared.
These macros could in principle be (inline) functions so it makes sense to
have them lower case, to match the other C API functions.
The remaining macros that are upper case are:
- MP_OBJ_TO_PTR, MP_OBJ_FROM_PTR
- MP_OBJ_NEW_SMALL_INT, MP_OBJ_SMALL_INT_VALUE
- MP_OBJ_NEW_QSTR, MP_OBJ_QSTR_VALUE
- MP_OBJ_FUN_MAKE_SIG
- MP_DECLARE_CONST_xxx
- MP_DEFINE_CONST_xxx
These must remain macros because they are used when defining const data (at
least, MP_OBJ_NEW_SMALL_INT is so it makes sense to have
MP_OBJ_SMALL_INT_VALUE also a macro).
For those macros that have been made lower case, compatibility macros are
provided for the old names so that users do not need to change their code
immediately.
The parser attempts to allocate two large (~512 byte) chunks up
front. If it couldn't, then it would error out. This change will
cause it to try allocating half the previous attempt until its down
to two copies. This is ok upfront because later code checks bounds
and tries to extend the allocation if needed.
Empty __VA_ARGS__ are not allowed in the C preprocessor so adjust the rule
arg offset calculation to not use them. Also, some compilers (eg MSVC)
require an extra layer of macro expansion.
This is the sixth and final patch in a series of patches to the parser that
aims to reduce code size by compressing the data corresponding to the rules
of the grammar.
Prior to this set of patches the rules were stored as rule_t structs with
rule_id, act and arg members. And then there was a big table of pointers
which allowed to lookup the address of a rule_t struct given the id of that
rule.
The changes that have been made are:
- Breaking up of the rule_t struct into individual components, with each
component in a separate array.
- Removal of the rule_id part of the struct because it's not needed.
- Put all the rule arg data in a big array.
- Change the table of pointers to rules to a table of offsets within the
array of rule arg data.
The last point is what is done in this patch here and brings about the
biggest decreases in code size, because an array of pointers is now an
array of bytes.
Code size changes for the six patches combined is:
bare-arm: -644
minimal x86: -1856
unix x64: -5408
unix nanbox: -2080
stm32: -720
esp8266: -812
cc3200: -712
For the change in parser performance: it was measured on pyboard that these
six patches combined gave an increase in script parse time of about 0.4%.
This is due to the slightly more complicated way of looking up the data for
a rule (since the 9th bit of the offset into the rule arg data table is
calculated with an if statement). This is an acceptable increase in parse
time considering that parsing is only done once per script (if compiled on
the target).
Instead of each rule being stored in ROM as a struct with rule_id, act and
arg, the act and arg parts are now in separate arrays and the rule_id part
is removed because it's not needed. This reduces code size, by roughly one
byte per grammar rule, around 150 bytes.
The rule name is only used for debugging, and this patch makes things a bit
cleaner by completely separating out the rule name from the rest of the
rule data.
The nan-boxing representation has an extra 16-bits of space to store
small-int values, and making use of it allows to create and manipulate full
32-bit positive integers (ie up to 0xffffffff) without using the heap.
The function mp_obj_new_str_of_type is a general str object constructor
used in many places in the code to create either a str or bytes object.
When creating a str it should first check if the string data already exists
as an interned qstr, and if so then return the qstr object. This patch
makes the function have such behaviour, which helps to reduce heap usage by
reusing existing interned data where possible.
The old behaviour of mp_obj_new_str_of_type (which didn't check for
existing interned data) is made available through the function
mp_obj_new_str_copy, but should only be used in very special cases.
One consequence of this patch is that the following expression is now True:
'abc' is ' abc '.split()[0]
Header files that are considered internal to the py core and should not
normally be included directly are:
py/nlr.h - internal nlr configuration and declarations
py/bc0.h - contains bytecode macro definitions
py/runtime0.h - contains basic runtime enums
Instead, the top-level header files to include are one of:
py/obj.h - includes runtime0.h and defines everything to use the
mp_obj_t type
py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
and defines everything to use the general runtime support functions
Additional, specific headers (eg py/objlist.h) can be included if needed.
Fixes for stmhal USB mass storage, lwIP bindings and VFS regressions
This release provides an important fix for the USB mass storage device in
the stmhal port by implementing the SCSI SYNCHRONIZE_CACHE command, which
is now require by some Operating Systems. There are also fixes for the
lwIP bindings to improve non-blocking sockets and error codes. The VFS has
some regressions fixed including the ability to statvfs the root.
All changes are listed below.
py core:
- modbuiltins: add core-provided version of input() function
- objstr: catch case of negative "maxsplit" arg to str.rsplit()
- persistentcode: allow to compile with complex numbers disabled
- objstr: allow to compile with obj-repr D, and unicode disabled
- modsys: allow to compile with obj-repr D and PY_ATTRTUPLE disabled
- provide mp_decode_uint_skip() to help reduce stack usage
- makeqstrdefs.py: make script run correctly with Python 2.6
- objstringio: if created from immutable object, follow copy on write policy
extmod:
- modlwip: connect: for non-blocking mode, return EINPROGRESS
- modlwip: fix error codes for duplicate calls to connect()
- modlwip: accept: fix error code for non-blocking mode
- vfs: allow to statvfs the root directory
- vfs: allow "buffering" and "encoding" args to VFS's open()
- modframebuf: fix signed/unsigned comparison pendantic warning
lib:
- libm: use isfinite instead of finitef, for C99 compatibility
- utils/interrupt_char: remove support for KBD_EXCEPTION disabled
tests:
- basics/string_rsplit: add tests for negative "maxsplit" argument
- float: convert "sys.exit()" to "raise SystemExit"
- float/builtin_float_minmax: PEP8 fixes
- basics: convert "sys.exit()" to "raise SystemExit"
- convert remaining "sys.exit()" to "raise SystemExit"
unix port:
- convert to use core-provided version of built-in import()
- Makefile: replace references to make with $(MAKE)
windows port:
- convert to use core-provided version of built-in import()
qemu-arm port:
- Makefile: adjust object-file lists to get correct dependencies
- enable micropython.mem_*() functions to allow more tests
stmhal port:
- boards: enable DAC for NUCLEO_F767ZI board
- add support for NUCLEO_F446RE board
- pass USB handler as parameter to allow more than one USB handler
- usb: use local USB handler variable in Start-of-Frame handler
- usb: make state for USB device private to top-level USB driver
- usbdev: for MSC implement SCSI SYNCHRONIZE_CACHE command
- convert from using stmhal's input() to core provided version
cc3200 port:
- convert from using stmhal's input() to core provided version
teensy port:
- convert from using stmhal's input() to core provided version
esp8266 port:
- Makefile: replace references to make with $(MAKE)
- Makefile: add clean-modules target
- convert from using stmhal's input() to core provided version
zephyr port:
- modusocket: getaddrinfo: Fix mp_obj_len() usage
- define MICROPY_PY_SYS_PLATFORM (to "zephyr")
- machine_pin: use native Zephyr types for Zephyr API calls
docs:
- machine.Pin: remove out_value() method
- machine.Pin: add on() and off() methods
- esp8266: consistently replace Pin.high/low methods with .on/off
- esp8266/quickref: polish Pin.on()/off() examples
- network: move confusingly-named cc3200 Server class to its reference
- uos: deconditionalize, remove minor port-specific details
- uos: move cc3200 port legacy VFS mounting functions to its ref doc
- machine: sort machine classes in logical order, not alphabetically
- network: first step to describe standard network class interface
examples:
- embedding: use core-provided KeyboardInterrupt object
The parser was originally written to work without raising any exceptions
and instead return an error value to the caller. But it's now required
that a call to the parser be wrapped in an nlr handler, so we may as well
make use of that fact and simplify the parser so that it doesn't need to
keep track of any memory errors that it had. The parser anyway explicitly
raises an exception at the end if there was an error.
This patch simplifies the parser by letting the underlying memory
allocation functions raise an exception if they fail to allocate any
memory. And if there is an error parsing the "<id> = const(<val>)" pattern
then that also raises an exception right away instead of trying to recover
gracefully and then raise.
Previous to this patch any non-interned str/bytes objects would create a
special parse node that held a copy of the str/bytes data. Then in the
compiler this data would be turned into a str/bytes object. This actually
lead to 2 copies of the data, one in the parse node and one in the object.
The parse node's copy of the data would be freed at the end of the compile
stage but nevertheless it meant that the peak memory usage of the
parse/compile stage was higher than it needed to be (by an amount equal to
the number of bytes in all the non-interned str/bytes objects).
This patch changes the behaviour so that str/bytes objects are created
directly in the parser and the object stored in a const-object parse node
(which already exists for bignum, float and complex const objects). This
reduces peak RAM usage of the parse/compile stage, simplifies the parser
and compiler, and reduces code size by about 170 bytes on Thumb2 archs,
and by about 300 bytes on Xtensa archs.
Jeff Epler
Jim Mussared
Scott Shawcroft
microDev
Damien George
Diego Elio Pettenò
Damien George
stijn
Josh Klar
Josh Lloyd
Dan Halbert
Scott Shawcroft
Alexander Steffen