F-strings, new machine.I2S class, ESP32-C3 support and LEGO_HUB_NO6 board
This release of MicroPython adds support for f-strings (PEP-498), with a
few limitations compared to normal Python. F-strings are essentially
syntactic sugar for "".format() and make formatting strings a lot more
convenient. Other improvements to the core runtime include pretty printing
OSError when it has two arguments (an errno code and a string), scheduling
of KeyboardInterrupt on the main thread, and support for a single argument
to the optimised form of StopIteration.
In the machine module a new I2S class has been added, with support for
esp32 and stm32 ports. This provides a consistent API for transmit and
receive of audio data in blocking, non-blocking and asyncio-based
operation. Also, the json module has support for the "separators" argument
in the dump and dumps functions, and framebuf now includes a way to blit
between frame buffers of different formats using a palette. A new,
portable machine.bitstream function is also added which can output a stream
of bits with configurable timing, and is used as the basis for driving
WS2812 LEDs in a common way across ports.
There has been some restructuring of the repository directory layout, with
all third-party code now in the lib/ directory. And a new top-level
directory shared/ has been added with first-party code that was previously
in lib/ moved there.
The docs have seen further improvement with enhancements and additions to
the rp2 parts, as well as a new quick reference for the zephyr port.
The terms master/slave have been replaced with controller/peripheral,
mainly relating to I2C and SPI usage. And u-module references have been
replaced with just the module name without the u-prefix to help clear up
the intended usage of modules in MicroPython.
For the esp8266 and esp32 ports, hidden networks are now included in WLAN
scan results. On the esp32 the RMT class is enhanced with idle_level and
write_pulses modes. There is initial support for ESP32-C3 chips with
GENERIC_C3 and GENERIC_C3_USB boards.
The javascript port has had its Makefile and garbage collector
implementation reworked so it compiles and runs with latest the Emscripten
using asyncify.
The mimxrt port sees the addition of hardware I2C and SPI support, as well
as some additional methods to the machine module. There is also support
for Hyperflash chips.
The nrf port now has full VFS storage support, enables source-line on
traceback, and has .mpy features consistent with other ports.
For the rp2 port there is now more configurability for boards, and more
boards added.
The stm32 port has a new LEGO_HUB_NO6 board definition with detailed
information how to get this LEGO Hub running stock MicroPython. There is
also now support to change the CPU frequency on STM32WB MCUs. And USBD_xxx
descriptor options have been renamed to MICROPY_HW_USB_xxx.
Thanks to everyone who contributed to this release: Amir Gonnen, Andrew
Scheller, Bryan Tong Minh, Chris Wilson, Damien George, Daniel Mizyrycki,
David Lechner, David P, Fernando, finefoot, Frank Pilhofer, Glenn Ruben
Bakke, iabdalkader, Jeff Epler, Jim Mussared, Jonathan Hogg, Josh Klar,
Josh Lloyd, Julia Hathaway, Krzysztof Adamski, Matúš Olekšák, Michael
Weiss, Michel Bouwmans, Mike Causer, Mike Teachman, Ned Konz, NitiKaur,
oclyke, Patrick Van Oosterwijck, Peter Hinch, Peter Züger, Philipp
Ebensberger, robert-hh, Roberto Colistete Jr, Sashkoiv, Seon Rozenblum,
Tobias Thyrrestrup, Tom McDermott, Will Sowerbutts, Yonatan Goldschmidt.
What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.
Main components
===============
all:
- fix signed shifts and NULL access errors from -fsanitize=undefined
- update to point to files in new shared/ directory
py core:
- mpstate: make exceptions thread-local
- mpstate: schedule KeyboardInterrupt on main thread
- mperrno: add MP_ECANCELED error code
- makeqstrdefs.py: don't include .h files explicitly in preprocessing
- mark unused arguments from bytecode decoding macros
- objexcept: pretty print OSError also when it has 2 arguments
- makeversionhdr: add --tags arg to git describe
- vm: simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode
- objexcept: make mp_obj_exception_get_value support subclassed excs
- support single argument to optimised MP_OBJ_STOP_ITERATION
- introduce and use mp_raise_type_arg helper
- modsys: optimise sys.exit for code size by using exception helpers
- objexcept: make mp_obj_new_exception_arg1 inline
- obj: fix formatting of comment for mp_obj_is_integer
- emitnative: reuse need_reg_all func in need_stack_settled
- emitnative: ensure stack settling is safe mid-branch
- runtime: fix bool unary op for subclasses of native types
- builtinimport: fix condition for including do_execute_raw_code()
- mkrules: automatically build mpy-cross if it doesn't exist
- implement partial PEP-498 (f-string) support
- lexer: clear fstring_args vstr on lexer free
- mkrules.mk: do submodule sync in "make submodules"
extmod:
- btstack: add missing call to mp_bluetooth_hci_uart_deinit
- btstack: check that BLE is active before performing operations
- uasyncio: get addr and bind server socket before creating task
- axtls-include: add axtls_os_port.h to customise axTLS
- update for move of crypto-algorithms, re1.5, uzlib to lib
- moduselect: conditionally compile select()
- nimble: fix leak in l2cap_send if send-while-stalled
- btstack/btstack.mk: use -Wno-implicit-fallthrough, not =0
- utime: always invoke mp_hal_delay_ms when >= to 0ms
- modbluetooth: clamp MTU values to 32->UINT16_MAX
- nimble: allow modbluetooth binding to hook "sent HCI packet"
- nimble: add "memory stalling" mechanism for l2cap_send
- uasyncio: in open_connection use address info in socket creation
- modujson: add support for dump/dumps separators keyword-argument
- modlwip: fix close and clean up of UDP and raw sockets
- modbluetooth: add send_update arg to gatts_write
- add machine.bitstream
- modframebuf: enable blit between different formats via a palette
lib:
- tinyusb: update to version 0.10.1
- pico-sdk: update to version 1.2.0
- utils/stdout_helpers: make mp_hal_stdout_tx_strn_cooked efficient
- axtls: switch to repo at micropython/axtls
- axtls: update to latest axtls 2.1.5 wih additional commits
- re1.5: move re1.5 code from extmod to lib
- uzlib: move uzlib code from extmod to lib
- crypto-algorithms: move crypto-algorithms code from extmod to lib
- update README's based on contents of these dirs
drivers:
- neopixel: add common machine.bitstream-based neopixel module
- neopixel: optimize fill() for speed
- neopixel: reduce code size of driver
- cyw43: fix cyw43_deinit so it can be called many times in a row
- cyw43: make wifi join fail if interface is not active
mpy-cross:
- disable stack check when building with Emscripten
Support components
==================
docs:
- library: document new esp32.RMT features and fix wait_done
- library: warn that ustruct doesn't handle spaces in format strings
- esp8266/tutorial: change flash mode from dio to dout
- replace master/slave with controller/peripheral in I2C and SPI
- rp2: enhance quickref and change image to Pico pinout
- rp2: update general section to give a brief technical overview
- library/utime.rst: clarify behaviour and precision of sleep ms/us
- library/uasyncio.rst: document stream readexactly() method
- library/machine.I2S.rst: fix use of sd pin in examples
- zephyr: add quick reference for the Zephyr port
- library/zephyr: add libraries specific to the Zephyr port
- templates: add unix and zephyr quickref links to top-index
- rename ufoo.rst to foo.rst
- replace ufoo with foo in all docs
- library/index.rst: clarify module naming and purpose
- library/builtins.rst: add module title
- library/network.rst: simplify socket import
- add docs for machine.bitstream and neopixel module
- library: fix usage of :term: for frozen module reference
- esp8266: use monospace for software tools
- reference: mention that slicing a memoryview causes allocation
examples: no changes specific to this component/port
tests:
- extmod: make uasyncio_heaplock test more deterministic
- cpydiff/modules_struct_whitespace_in_format: run black
- extmod/ujson: add tests for dump/dumps separators argument
- run-multitests.py: add broadcast and wait facility
- multi_bluetooth/ble_subscribe.py: add test for subscription
- extmod/vfs_fat_finaliser.py: ensure alloc at never-used GC blocks
- basics: split f-string debug printing to separate file with .exp
- pybnative: make while.py test run on boards without pyb.delay
tools:
- autobuild: add scripts to build release firmware
- remove obsolete build-stm-latest.sh script
- ci.sh: run apt-get update in ci_powerpc_setup
- makemanifest.py: allow passing flags to mpy-tool.py
- autobuild: add mimxrt port to build scripts for nightly builds
- pyboard.py: add cmd-line option to make soft reset configurable
- mpremote: swap order of PID and VID in connect-list output
- ci.sh: build unix dev variant as part of macOS CI
- ci.sh: build GENERIC_C3 board as part of esp32 CI
- autobuild: use separate IDF version to build newer esp32 SoCs
- autobuild: add FeatherS2 and TinyS2 to esp32 auto builds
- mpremote: add seek whence for mounted files
- mpremote: raise OSError on unsupported RemoteFile.seek
- autobuild: add the MIMXRT1050_EVKB board to the daily builds
- ci.sh: add mpy-cross build to nrf port
- codeformat.py: include ports/nrf/modules/nrf in code formatting
- gen-cpydiff.py: don't rename foo to ufoo in diff output
- autobuild: add auto build for Silicognition wESP32
- mpremote: fix connect-list in case VID/PID are None
- mpremote: add "devs" shortcut for "connect list"
- mpremote: remove support for pyb.USB_VCP in/out specialisation
- autobuild: don't use "-B" for make, it's already a fresh build
- pyboard.py: move --no-exclusive/--soft-reset out of mutex group
- pyboard.py: make --no-follow use same variable as --follow
- pyboard.py: add --exclusive to match --no-exclusive
- pyboard.py: make --no-soft-reset consistent with other args
- uncrustify: force 1 newline at end of file
- mpremote: bump version to 0.0.6
CI:
- workflows: add workflow to build and test javascript port
- workflows: switch from Coveralls to Codecov
- workflows: switch from lcov to gcov
- workflows: add workflow to build and test unix dev variant
The ports
=========
all ports:
- use common mp_hal_stdout_tx_strn_cooked instead of custom one
- update for move of crypto-algorithms, uzlib to lib
- rename USBD_VID/PID config macros to MICROPY_HW_USB_VID/PID
bare-arm port: no changes specific to this component/port
cc3200 port: no changes specific to this component/port
esp8266 port:
- add __len__ to NeoPixel driver to support iterating
- Makefile: add more libm files to build
- include hidden networks in WLAN.scan results
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
esp32 port:
- boards/sdkconfig.base: disable MEMPROT_FEATURE to alloc from IRAM
- add __len__ to NeoPixel driver to support iterating
- main: allow MICROPY_DIR to be overridden
- esp32_rmt: fix RMT looping in newer IDF versions
- esp32_rmt: enhance RMT with idle_level and write_pulses modes
- add new machine.I2S class for I2S protocol support
- machine_spi: calculate actual attained baudrate
- machine_hw_spi: use a 2 item SPI queue for long transfers
- machine_dac: add MICROPY_PY_MACHINE_DAC option, enable by default
- machine_i2s: add MICROPY_PY_MACHINE_I2S option, enable by default
- fix use of mp_int_t, size_t and uintptr_t
- add initial support for ESP32C3 SoCs
- boards/GENERIC_C3: add generic C3-based board
- modmachine: release the GIL in machine.idle()
- mphalport: always yield at least once in delay_ms
- machine_uart: add flow kw-arg to enable hardware flow control
- boards: add Silicognition wESP32 board configuration
- mpconfigport.h: enable reverse and inplace special methods
- include hidden networks in WLAN.scan results
- makeimg.py: get bootloader and partition offset from sdkconfig
- enable MICROPY_PY_FSTRINGS by default
- machine_hw_spi: release GIL during transfers
- machine_pin: make check for non-output pins respect chip variant
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
- boards: add GENERIC_C3_USB board with USB serial/JTAG support
javascript port:
- rework Makefile and GC so it works with latest Emscripten
- Makefile: suppress compiler errors from array bounds
- Makefile: change variable to EXPORTED_RUNTIME_METHODS
mimxrt port:
- move calc_weekday helper function to timeutils
- machine_spi: add the SPI class to the machine module
- moduos: seed the PRNG on boot using the TRNG
- boards: set vfs partition start to 1 MBbyte
- main: skip running main.py if boot.py failed
- main: extend the information returned by help()
- mimxrt_flash: remove commented-out code
- modmachine: add a few minor methods to the machine module
- machine_led: use mp_raise_msg_varg helper
- machine_i2c: add hardware-based machine.I2C to machine module
- add support for Hyperflash chips
- boards: add support for the MIMXRT1050_EVKB board
- machine_pin: implement ioctl for Pin
minimal port:
- Makefile: add support for building with user C modules
nrf port:
- modules: replace master/slave with controller/peripheral in SPI
- boards/common.ld: calculate unused flash region
- modules/nrf: add new nrf module with flash block device
- drivers: add support for using flash block device with SoftDevice
- mpconfigport.h: expose nrf module when MICROPY_PY_NRF is set
- README: update README.md to reflect internal file systems
- mpconfigport.h: tune FAT FS configuration
- Makefile: add _fs_size linker script override from make
- modules/uos: allow a board to configure MICROPY_VFS_FAT/LFS1/LFS2
- mpconfigport.h: enable MICROPY_PY_IO_FILEIO when an FS is enabled
- qstrdefsport.h: add entries for in-built FS mount points
- main: add auto mount and auto format hook for internal flash FS
- boards: enable needed features for FAT/LFS1/LFS2
- facilitate use of freeze manifest
- boards: set FROZEN_MANIFEST blank when SD present on nrf51 targets
- modules/scripts: add file system formatting script
- Makefile: set default manifest file for all targets
- mphalport: add dummy function for mp_hal_time_ns()
- boards: enable MICROPY_VFS_LFS2 for all target boards
- modules/uos: add ilistdir to uos module
- modules/nrf: add function to enable/disable DCDC
- enable source line on tracebacks
- set .mpy features consistent with documentation and other ports
pic16bit port: no changes specific to this component/port
powerpc port: no changes specific to this component/port
qemu-arm port: no changes specific to this component/port
rp2 port:
- use 0=Monday datetime convention in RTC
- machine_rtc: in RTC.datetime, compute weekday automatically
- CMakeLists.txt: suppress compiler errors for pico-sdk and tinyusb
- tusb_config.h: set CFG_TUD_CDC_EP_BUFSIZE to 256
- machine_uart: add hardware flow control support
- machine_uart: allow overriding default machine UART pins
- machine_i2c: allow boards to configure I2C pins using new macros
- machine_spi: allow boards to configure SPI pins using new macros
- machine_uart: fix poll ioctl to also check hardware FIFO
- machine_uart: fix read when FIFO has chars but ringbuf doesn't
- tusb_port: allow boards to configure USB VID and PID
- boards/ADAFRUIT_FEATHER_RP2040: configure custom VID/PID
- boards/ADAFRUIT_FEATHER_RP2040: configure I2C/SPI default pins
- boards/SPARKFUN_PROMICRO: configure UART/I2C/SPI default pins
- boards/SPARKFUN_THINGPLUS: configure I2C/SPI default pins
- boards: add Adafruit ItsyBitsy RP2040
- boards: add Adafruit QT Py RP2040
- boards: add Pimoroni Pico LiPo 4MB
- boards: add Pimoroni Pico LiPo 16MB
- boards: add Pimoroni Tiny 2040
- CMakeLists.txt: allow a board's cmake to set the manifest path
- enable MICROPY_PY_FSTRINGS by default
- Makefile: add "submodules" target, to match other ports
- rp2_flash: disable IRQs while calling flash_erase/program
- CMakeLists.txt: add option to enable double tap reset to bootrom
- mpconfigport.h: allow boards to add root pointers
samd port:
- add support for building with user C modules
stm32 port:
- softtimer: add soft_timer_reinsert() helper function
- mpbthciport: change from systick to soft-timer for BT scheduling
- provide a custom BTstack runloop that integrates with soft timer
- usb: make irq's default trigger enable all events
- boardctrl: skip running main.py if boot.py had an error
- sdio: fix undefined reference to DMA stream on H7
- dma: add DMAMUX configuration for H7 to fix dma_nohal_init
- main: call mp_deinit() at end of main
- adc: allow using ADC12 and ADC3 for H7
- adc: define the ADC instance used for internal channels
- adc: simplify and generalise how pin_adcX table is defined
- add new machine.I2S class for I2S protocol support
- boards/NUCLEO_F446RE: fix I2C1 pin assignment to match datasheet
- replace master/slave with controller/peripheral in I2C and SPI
- systick: always POLL_HOOK when delaying for milliseconds
- sdram: make SDRAM test cache aware, and optional failure with msg
- boards/NUCLEO_F446RE: enable CAN bus support
- boards: add support for SparkFun STM32 MicroMod Processor board
- uart: fix LPUART1 baudrate set/get
- uart: support low baudrates on LPUART1
- boards/STM32F429DISC: set correct UART2 pins and add UART3/6
- boards/NUCLEO_F439ZI: add board definition for NUCLEO_F439ZI
- boards/LEGO_HUB_NO6: add board definition for LEGO_HUB_NO6
- Makefile: update to only pull in used Bluetooth library
- README.md: update supported MCUs, and submodule and mboot use
- usbd_desc: rename USBD_xxx descriptor opts to MICROPY_HW_USB_xxx
- usbd_cdc_interface: rename USBD_CDC_xx opts to MICROPY_HW_USB_xx
- powerctrl: support changing frequency on WB MCUs
- boards/NUCLEO_H743ZI2: add modified version of NUCLEO_H743ZI
- mbedtls: fix compile warning about uninitialized val
- enable MICROPY_PY_FSTRINGS by default
- add implementation of machine.bitstream
- Makefile: allow GIT_SUBMODULES and LIBS to be extended
- stm32_it: support TIM17 IRQs on WB MCUs
- disable computed goto on constrained boards
- storage: make extended-block-device more configurable
- boards/LEGO_HUB_NO6: change SPI flash storage to use hardware SPI
- boards/LEGO_HUB_NO6: skip first 1MiB of SPI flash for storage
- boards/LEGO_HUB_NO6: add make commands to backup/restore firmware
teensy port: no changes specific to this component/port
unix port:
- modffi: add option to lock GC in callback, and cfun access
- Makefile: add back LIB_SRC_C to list of object files
- variants: enable help and help("modules") on standard and dev
- Makefile: disable error compression on arm-linux-gnueabi-gcc
windows port:
- Makefile: add .exe extension to executables name
- appveyor: update to VS 2017 and use Python 3.8 for build/test
zephyr port:
- machine_spi: add support for hardware SPI
complete pin mapping for Feather pins
stubbed out files needed for complilation. still to be modified
0 out all CPY modules in mpconfigboard.mk until we get the build running
add csv for pin generation for STM32L4R5
add F4R5 references in peripherals files
refactored out board files BECAUSE I AM AN IDIOT; add L4 series system clocks file from CubeMX
took a guess at the number of USB endpoint pairs to get the build done
guess was close, but wrong. It is 8
clean up peripheral DEFs
Fixes build error:
```
In file included from ../../py/mpstate.h:33,
from ../../py/mpstate.c:27:
../../py/misc.h: In function 'vstr_str':
../../py/misc.h:196:1: sorry, unimplemented: Thumb-1 hard-float VFP ABI
static inline char *vstr_str(vstr_t *vstr) {
^~~~~~
```
Sleuthing steps:
* verify that the feather_stm32f4_express board builds correctly
* put a `#error` at the bottom of the `mpstate.c` file.
* build for the feather and swan boards, with V=2 to capture the build command for that file.
* use a differencing tool to inspect the differences between the two invocations
* inspecting the differences, I saw a missing `-mcpu=cortex-m4` I tested by adding that to the Swan build command. The file built fine (stopping at the hard error, but no other warnings.)
A grep through the sources revealed where this flag was being set for the stm ports.
With this commit, the build gets further, but does not complete. The next exciting episode in this unfolding coding saga is just a commit away!
working build with minimal set of modules for the Blues Swan r5
chore:change header copyright name to Blues Wireless Contributors
USB operational. Fixed up clocks to be hardwired for LSE no HSE case. (Trying to combine HSE in there made the code much more complex, and I don't have a board to test it out on.)
USART working
adds support for `ENABLE_3V3` and `DISCHARGE_3V3` pins. I am surprised that pin definitions are quite low-level and don't include default direction and state, so the code currently has to initialize `ENABLE_3V3` pin as output. The LED takes over a second to discharge, so I wonder if the board startup code is not having the desired affect.
short circuit implementation of backup memory for the STM32L4
all the ports
remove company name from board name to be consistent with the Arduino board definition.
add default pins for I2C, SPI and UART, so that `board.I2C` et al. works as expected. Confirmed I2C timing.
fix board name
fix incorrect pin definition. add test to allow manual check of each output pin
analog IO
code changes for WebUSB. Doesn't appear to work, will revisit later.
ensure that `sys.platform` is available
checkin missing file
feat: make room for a larger filesystem so the sensor tutorial will fit on the device.
fix:(stm32l4r5zi.csv): merged AF0-7 and AF8-15 into single lines and removed extraneous headers mixed in with the data.
fix(parse_af_csv.py): pin index in the csv is 0 not 1, and AF index made 1 larger
chore(Swan R5): update peripherals pins from `parse_af_csv.py` output
optimize flash sector access
This patch casts the delay argument to mp_hal_delay_ms()
from mp_uint_t to uint64_t when scaling from milliseconds
to ticks to avoid 32-bit integer overflow when time.sleep()
is called with a duration of greater than 70 minutes.
Signed-off-by: Trammell Hudson <hudson@trmm.net>
Sometimes we don't want the delay in the device's startup.
I used a simple flag, but now it also occurs to me that we could
instead make the wait time a define, and set it to 0 to disable it.
This will un-break examples that got confused by the presence of
two Nordic UART Services. It also adds a version characteristic
that gives the CircuitPython build tag back.
Fixes#5252
There is a race between when we run background tasks and when we
sleep. If an interrupt happens between the two, then we may delay
executing the background task. On some ports we checked this for
TinyUSB already. On iMX RT, we didn't which caused USB issues.
This PR makes it more generic for all background tasks including
USB.
Fixes#5086 and maybe others.
* The new nonstandard '%S' format takes a pointer to compressed_string_t
and prints it
* The new mp_cprintf and mp_vcprintf take a format string that is a
compressed_string_t
By storing "count of words by length", the long `wends` table can be
replaced with a short `wlencount` table. This saves flash storage space.
Extend the range of string lengths that can be in the dictionary.
Originally it was to 2 to 9; at one point it was changed to 3 to 9.
Putting the lower bound back at 2 has a positive impact on the French
translation (a bunch of them, such as "ch", "\r\n", "%q", are used).
Increasing the maximum length gets 'mpossible', ' doit être ',
and 'CircuitPyth' at the long end. This adds a bit of processing time
to makeqstrdata. The specific 2/11 values are again empirical based on
the French translation on the adafruit_proxlight_trinkey_m0.
Call `supervisor.disable_ble_workflow()` and the BLE workflow will
be disabled until the chip is reset.
This also includes a couple fixes:
1. Terminals can now be deinit by setting the tilegrid to NULL. This
prevents using the tilegrid before display is init.
2. Fix BLE serial send amount when sending more than a single packet.
Fixes#5049
nRF CircuitPython boards will now provide the file transfer
service defined here: https://github.com/adafruit/Adafruit_CircuitPython_BLE_File_Transfer
USB capable boards will only advertise if previously bonded to a
device or if the reset button is pressed during the fast blue
flashes on start up. When pressed, the board will restart again but
the blue period will not flash.
Boards without USB will always advertise.
When previously bonded, the advertisement is private so that no
other peers can connect. If advertising publicly, the tx power is
lowered to reduce the likelihood of bonding from a distance.
This PR also fixes issues with loading identities of bonded peers
so that our address can now be resolved and we can resolve others'
addresses when scanning.
It was possible for _only_ a low allocation to be performed.
In this case, `high_head` is NULL, and the comparison
`MP_STATE_VM(first_embedded_allocation) < high_head` would fail.
Closes: #4871
This also removes the need to pin share because we don't use the
status LED while user code is running.
The status flashes fallback to the HW_STATUS LED if no RGB LED is
present. Each status has a unique blink pattern as well.
One caveat is the REPL state. In order to not pin share, we set the
RGB color once. PWM and single color will be shutoff immediately but
DotStars and NeoPixels will hold the color until the user overrides
it.
Fixes#4133
* modframebuf: _mp_framebuf_p_t is not "really" a protocol, but the
QSTR assignment caused problems when building as a dynamic module
* modure: str_index_to_ptr is not in the natmod API, disable URE match
spans when dynamic. mp_obj_len() is a bugfix, we should throw here
if the object is not string-like
* moduzlib: Correct paths to uzlib headers & sources. this relative
path (from moduzlib.c to the referenced file) works in all cases,
the other only worked from ports/PORTNAME.
* dynruntime: Handle 2-arg m_malloc, assert_native_inited, add a
micropythonish mp_arg_check_num_mp, fix mp_raise_msg to use dumb
strings, add mp_raise_arg1
* nativeglue: ad assert_native_inited
* translate: MP_ERROR_TEXT evaluates to its argument for DYNRUNTIME
* mpy-tool: A straggling magic number change
* mpy_ld: Have to renumber manually after dynruntime change
* import_mpy_native_gc.py: Update copy of features0 baked into this test
When `reload_requested` is detected, the run reason will no longer be
automatically overwritten as an AUTO_RELOAD, making SUPERVISOR_RELOAD a
detectable reload reason. Autoreload now sets the reload reason itself.
After discussing with danh, I noticed that `a/**/b` would not match `a/b`.
After correcting this and re-running "pre-commit run --all", additional
files were reindented, including the codeformat script itself.
In #4683, tannewt noticed that uncrustify was not running on some
file in common-hal.
I investigated and found that it was not being run on a bunch of paths.
Rather than make incremental changes, I rewrote list_files to work
bsaed on regular expressions; these regular expressions are created from
the same git-style glob patterns.
I spot-checked some specific filenames after this change, and all looks good:
```
$ python3 tools/codeformat.py -v --dry-run tests/basics/int_small.py ports/raspberrypi/common-hal/pulseio/PulseIn.c extmod/virtpin.c tests/thread/thread_exit1.py ports/raspberrypi/background.h extmod/re1.5/recursiveloop.c
tools/codeformat.py -v --dry-run tests/basics/int_small.py ports/raspberrypi/common-hal/pulseio/PulseIn.c extmod/virtpin.c tests/thread/thread_exit1.py ports/raspberrypi/background.h extmod/re1.5/recursiveloop.c
uncrustify -c /home/jepler/src/circuitpython/tools/uncrustify.cfg -lC --no-backup extmod/virtpin.c ports/raspberrypi/background.h ports/raspberrypi/common-hal/pulseio/PulseIn.c
black --fast --line-length=99 -v tests/thread/thread_exit1.py
```
recursiveloop and int_small are excluded, while PulseIn, virtpin,
and background are included.
Testing running from a subdirectory (not _specifically_ supported though):
```
(cd ports && python3 ../tools/codeformat.py -v --dry-run raspberrypi/common-hal/pulseio/PulseIn.c ../extmod/virtpin.c)
../tools/codeformat.py -v --dry-run raspberrypi/common-hal/pulseio/PulseIn.c ../extmod/virtpin.c
uncrustify -c /home/jepler/src/circuitpython/tools/uncrustify.cfg -lC --no-backup ../extmod/virtpin.c raspberrypi/common-hal/pulseio/PulseIn.
```
As a side-effect, a bunch more files are re-formatted now. :-P
Unify USB-related makefile var and C def as CIRCUITPY_USB.
Always define it as 0 or 1, same as all other settings.
USB_AVAILABLE was conditionally defined in supervisor.mk,
but never actually used to #ifdef USB-related code.
Loosely related to #4546
The interrupt may have a higher priority than the serial output's
(USB) interrupt and may never make room. This makes prints from
interrupts (like the BLE event calls) best effort for what can be
queued up. The rest of the output will be dropped.
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update"
will all point to the same memory.
Shaves ~1KB off the image size, and potentially allows
bigger savings if qstr attrs are initialized in qstr_init(),
and not stored in the image.
This manifested as incorrect error messages from mpy-cross, like
```
$ mpy-cross doesnotexist.py
OSError: [Errno 2] cno such file/director
```
The remaining bits in `b` must be shifted to the correct position before
entering the loop.
For most (all?) actual builds, compress_max_length_bits was 8 and the
problem went unnoticed.
This switches stage2 to C and uses Jinja to change the C code based
on flash settings from https://github.com/adafruit/nvm.toml. It
produces the fastest settings for the given set of external flashes.
Flash size is no longer hard coded so switching flashes with similar
capabilities but different sizes should *just work*.
This PR also places "ITCM" code in RAM to save the XIP cache for
code execution. Further optimization is possible. A blink code.py
still requires a number of flash fetches every blink.
Fixes#4041
Instead of counting words in make, which is slightly awful, notice that
possible_devices is local to external_flash.c, so we can declare the array
with an automatic bound, and then get the count as the element-count
(MP_ARRAY_SIZE) of the array.
Since EXTERNAL_FLASH_DEVICE_COUNT is no longer a global macro, switch
a few sites to using EXTERNAL_FLASH_DEVICES in `#if` checks instead.
This adds some additional code in mkfs which doesn't seem necessary, and
Disabling it saves 172 bytes flash.
Testing performed: Using a Feather M0 Adalogger, checked that
* an sdcard could still be mounted (using adafruit_sdcard)
* os.listdir() of "/" and "/sd" worked
* CIRCUITPY still mounted
This is a first go at it, done by naive replacing of all array
operations with corresponding operations on the list. Note that
there is a lot of unnecessary type conversions, here. Also, list_pop
has been copied, because it's decalerd STATIC in py/objlist.h
Since we want to expose the list of group's children to the user,
we should only have the original objects in it, without any other
additional data, and compute the native object as needed.
This saves about 60 bytes (Feather M4 went from 45040 -> 45100 bytes free)
66 bytes of data eliminated, but 6 bytes paid back to initialize the length
field.
The RP2040 is new microcontroller from Raspberry Pi that features
two Cortex M0s and eight PIO state machines that are good for
crunching lots of data. It has 264k RAM and a built in UF2
bootloader too.
Datasheet: https://pico.raspberrypi.org/files/rp2040_datasheet.pdf
Some ports need an extra operation to ensure that the main task is
awoken so that a queued background task will execute during an ongoing
light sleep.
This removes the need to enable supervisor ticks while I2SOut is operating.
Closes: #3952
This changes lots of files to unify `board.h` across ports. It adds
`board_deinit` when CIRCUITPY_ALARM is set. `main.c` uses it to
deinit the board before deep sleeping (even when pretending.)
Deep sleep is now a two step process for the port. First, the
port should prepare to deep sleep based on the given alarms. It
should set alarms for both deep and pretend sleep. In particular,
the pretend versions should be set immediately so that we don't
miss an alarm as we shutdown. These alarms should also wake from
`port_idle_until_interrupt` which is used when pretending to deep
sleep.
Second, when real deep sleeping, `alarm_enter_deep_sleep` is called.
The port should set any alarms it didn't during prepare based on
data it saved internally during prepare.
ESP32-S2 sleep is a bit reorganized to locate more logic with
TimeAlarm. This will help it scale to more alarm types.
Fixes#3786
This allows calls to `allocate_memory()` while the VM is running, it will then allocate from the GC heap (unless there is a suitable hole among the supervisor allocations), and when the VM exits and the GC heap is freed, the allocation will be moved to the bottom of the former GC heap and transformed into a proper supervisor allocation. Existing movable allocations will also be moved to defragment the supervisor heap and ensure that the next VM run gets as much memory as possible for the GC heap.
By itself this breaks terminalio because it violates the assumption that supervisor_display_move_memory() still has access to an undisturbed heap to copy the tilegrid from. It will work in many cases, but if you're unlucky you will get garbled terminal contents after exiting from the vm run that created the display. This will be fixed in the following commit, which is separate to simplify review.
This makes a more useful display on the portrait magtag, allowing 21
characters across instead of just 18. There are 20 full rows of text,
instead of 21. The total number of characters increases slightly from 378
to 420.
For comparison, the Commodore VIC 20 had 22 rows of 23 characters for a
total of 506 characters. :-P
* No weak link for modules. It only impacts _os and _time and is
already disabled for non-full builds.
* Turn off PA00 and PA01 because they are the crystal on the Metro
M0 Express.
* Change ejected default to false to move it to BSS. It is set on
USB connection anyway.
* Set sinc_filter to const. Doesn't help flash but keeps it out of
RAM.
This unifies the flash config to the settings used by the Boot ROM.
This makes the config unique per board which allows for changing
quad enable and status bit differences per flash device. It also
allows for timing differences due to the board layout.
This change also tweaks linker layout to leave more ram space for
the CircuitPython heap.
This requires recovering the pointer of the allocation, which could be done by adding up neighbor lengths, but the simpler way is to stop NULLing it out in the first place and instead mark an allocation as freed by the client by setting the lowest bit of the length (which is always zero in a valid length).
When allocations were freed in a different order from the reverse of how they were allocated (leaving holes), the heap would get into an inconsistent state, eventually resulting in crashes.
free_memory() relies on having allocations in order, but allocate_memory() did not guarantee that: It reused the first allocation with a NULL ptr without ensuring that it was between low_address and high_address. When it belongs to a hole in the allocated memory, such an allocation is not really free for reuse, because free_memory() still needs its length.
Instead, explicitly mark allocations available for reuse with a special (invalid) value in the length field. Only allocations that lie between low_address and high_address are marked that way.
I have a function where it should be impossible to reach the end, so I put in a safe-mode reset at the bottom:
```
int find_unused_slot(void) {
// precondition: you already verified that a slot was available
for (int i=0; i<NUM_SLOTS; i++) {
if( slot_free(i)) {
return i;
}
}
safe_mode_reset(MICROPY_FATAL_ERROR);
}
```
However, the compiler still gave a diagnostic, because safe_mode_reset was not declared NORETURN.
So I started by teaching the compiler that reset_into_safe_mode never returned. This leads at least one level deeper due to reset_cpu needing to be a NORETURN function. Each port is a little different in this area. I also marked reset_to_bootloader as NORETURN.
Additional notes:
* stm32's reset_to_bootloader was not implemented, but now does a bare reset. Most stm32s are not fitted with uf2 bootloaders anyway.
* ditto cxd56
* esp32s2 did not implement reset_cpu at all. I used esp_restart(). (not tested)
* litex did not implement reset_cpu at all. I used reboot_ctrl_write. But notably this is what reset_to_bootloader already did, so one or the other must be incorrect (not tested). reboot_ctrl_write cannot be declared NORETURN, as it returns unless the special value 0xac is written), so a new unreachable forever-loop is added.
* cxd56's reset is via a boardctl() call which can't generically be declared NORETURN, so a new unreacahble "for(;;)" forever-loop is added.
* In several places, NVIC_SystemReset is redeclared with NORETURN applied. This is accepted just fine by gcc. I chose this as preferable to editing the multiple copies of CMSIS headers where it is normally declared.
* the stub safe_mode reset simply aborts. This is used in mpy-cross.
These changes remove the caveat from supervisor.runtime.serial_connected.
It appears that _tud_cdc_connected() only tracks explicit changes to the
"DTR" bit, which leads to disconnects not being registered.
Instead:
* when line state is changed explicitly, track the dtr value in
_serial_connected
* when the USB bus is suspended, set _serial_connected to False
Testing performed (using sam e54 xplained): Run a program to show
the state of `serial_connected` on the LED:
```
import digitalio
import supervisor
import board
led = digitalio.DigitalInOut(board.LED)
while True:
led.switch_to_output(not supervisor.runtime.serial_connected)
```
Try all the following:
* open, close serial terminal program
- LED status tracks whether terminal is open
* turn on/off data lines using the switchable charge-only cable
- LED turns off when switch is in "charger" position
- LED turns back on when switch is in Data position and terminal is
opened (but doesn't turn back on just because switch position is
changed)
Massive savings. Thanks so much @ciscorn for providing the initial
code for choosing the dictionary.
This adds a bit of time to the build, both to find the dictionary
but also because (for reasons I don't fully understand), the binary
search in the compress() function no longer worked and had to be
replaced with a linear search.
I think this is because the intended invariant is that for codebook
entries that encode to the same number of bits, the entries are ordered
in ascending value. However, I mis-placed the transition from "words"
to "byte/char values" so the codebook entries for words are in word-order
rather than their code order.
Because this price is only paid at build time, I didn't care to determine
exactly where the correct fix was.
I also commented out a line to produce the "estimated total memory size"
-- at least on the unix build with TRANSLATION=ja, this led to a build
time KeyError trying to compute the codebook size for all the strings.
I think this occurs because some single unicode code point ('ァ') is
no longer present as itself in the compressed strings, due to always
being replaced by a word.
As promised, this seems to save hundreds of bytes in the German translation
on the trinket m0.
Testing performed:
- built trinket_m0 in several languages
- built and ran unix port in several languages (en, de_DE, ja) and ran
simple error-producing codes like ./micropython -c '1/0'
Two problems: The lead byte for 3-byte sequences was wrong, and one
mid-byte was not even filled in due to a missing "++"!
Apparently this was broken ever since the first "Compress as unicode,
not bytes" commit, but I believed I'd "tested" it by running on the
Pinyin translation.
This rendered at least the Korean and Japanese translations completely
illegible, affecting 5.0 and all later releases.
Compress common unicode bigrams by making code points in the range
0x80 - 0xbf (inclusive) represent them. Then, they can be greedily
encoded and the substituted code points handled by the existing Huffman
compression. Normally code points in the range 0x80-0xbf are not used
in Unicode, so we stake our own claim. Using the more arguably correct
"Private Use Area" (PUA) would mean that for scripts that only use
code points under 256 we would use more memory for the "values" table.
bigram means "two letters", and is also sometimes called a "digram".
It's nothing to do with "big RAM". For our purposes, a bigram represents
two successive unicode code points, so for instance in our build on
trinket m0 for english the most frequent are:
['t ', 'e ', 'in', 'd ', ...].
The bigrams are selected based on frequency in the corpus, but the
selection is not necessarily optimal, for these reasons I can think of:
* Suppose the corpus was just "tea" repeated 100 times. The
top bigrams would be "te", and "ea". However,
overlap, "te" could never be used. Thus, some bigrams might actually
waste space
* I _assume_ this has to be why e.g., bigram 0x86 "s " is more
frequent than bigram 0x85 " a" in English for Trinket M0, because
sequences like "can't add" would get the "t " digram and then
be unable to use the " a" digram.
* And generally, if a bigram is frequent then so are its constituents.
Say that "i" and "n" both encode to just 5 or 6 bits, then the huffman
code for "in" had better compress to 10 or fewer bits or it's a net
loss!
* I checked though! "i" is 5 bits, "n" is 6 bits (lucky guess)
but the bigram 0x83 also just 6 bits, so this one is a win of
5 bits for every "it" minus overhead. Yay, this round goes to team
compression.
* On the other hand, the least frequent bigram 0x9d " n" is 10 bits
long and its constituent code points are 4+6 bits so there's no
savings, but there is the cost of the table entry.
* and somehow 0x9f 'an' is never used at all!
With or without accounting for overlaps, there is some optimum number
of bigrams. Adding one more bigram uses at least 2 bytes (for the
entry in the bigram table; 4 bytes if code points >255 are in the
source text) and also needs a slot in the Huffman dictionary, so
adding bigrams beyond the optimim number makes compression worse again.
If it's an improvement, the fact that it's not guaranteed optimal
doesn't seem to matter too much. It just leaves a little more fruit
for the next sweep to pick up. Perhaps try adding the most frequent
bigram not yet present, until it doesn't improve compression overall.
Right now, de_DE is again the "fullest" build on trinket_m0. (It's
reclaimed that spot from the ja translation somehow) This change saves
104 bytes there, increasing free space about 6.8%. In the larger
(but not critically full) pyportal build it saves 324 bytes.
The specific number of bigrams used (32) was chosen as it is the max
number that fit within the 0x80..0xbf range. Larger tables would
require the use of 16 bit code points in the de_DE build, losing savings
overall.
(Side note: The most frequent letters in English have been said
to be: ETA OIN SHRDLU; but we have UAC EIL MOPRST in our corpus)
Otherwise, out of range writes would occur in tilegrid_set_tile, causing a safe mode reset.
```
Hardware watchpoint 6: -location *stack_alloc->ptr
Old value = 24652061
New value = 24641565
0x000444f2 in common_hal_displayio_tilegrid_set_tile (self=0x200002c8 <supervisor_terminal_text_grid>, x=1, y=1, tile_index=0 '\000')
at ../../shared-module/displayio/TileGrid.c:236
236 if (!self->partial_change) {
(gdb)
```
