This blends two "565"-format bitmaps, including byteswapped ones. All
the bitmaps have to have the same memory format.
The routine takes about 63ms on a Kaluga when operating on 320x240 bitmaps.
Of course, displaying the bitmap also takes time.
There's untested code for the L8 (8-bit greyscale) case. This can be
enabled once gifio is merged.
By having a pair of buffers, the capture hardware can fill one buffer while
Python code (including displayio, etc) operates on the other buffer. This
increases the responsiveness of camera-using code.
On the Kaluga it makes the following improvements:
* 320x240 viewfinder at 30fps instead of 15fps using directio
* 240x240 animated gif capture at 10fps instead of 7.5fps
As discussed at length on Discord, the "usual end user" code will look like
this:
camera = ...
with camera.continuous_capture(buffer1, buffer2) as capture:
for frame in capture:
# Do something with frame
However, rather than presenting a context manager, the core code consists of
three new functions to start & stop continuous capture, and to get the next
frame. The reason is twofold. First, it's simply easier to implement the
context manager object in pure Python. Second, for more advanced usage, the
context manager may be too limiting, and it's easier to iterate on the right
design in Python code. In particular, I noticed that adapting the
JPEG-capturing programs to use continuous capture mode needed a change in
program structure.
The camera app was structured as
```python
while True:
if shutter button was just pressed:
capture a jpeg frame
else:
update the viewfinder
```
However, "capture a jpeg frame" needs to (A) switch the camera settings and (B)
capture into a different, larger buffer then (C) return to the earlier
settings. This can't be done during continuous capture mode. So just
restructuring it as follows isn't going to work:
```python
with camera.continuous_capture(buffer1, buffer2) as capture:
for frame in capture:
if shutter button was just pressed:
capture a jpeg frame, without disturbing continuous capture mode
else:
update the viewfinder
```
The continuous mode is only implemented in the espressif port; others
will throw an exception if the associated methods are invoked. It's not
impossible to implement there, just not a priority, since these micros don't
have enough RAM for two framebuffer copies at any resonable sizes.
The capture code, including single-shot capture, now take mp_obj_t in the
common-hal layer, instead of a buffer & length. This was done for the
continuous capture mode because it has to identify & return to the user the
proper Python object representing the original buffer. In the Espressif port,
it was convenient to implement single capture in terms of a multi-capture,
which is why I changed the singleshot routine's signature too.
The easiest thing to implement was to use the i/j numbers, but they were not
directly related to image x/y coordinates. This may slow things down a tiny
little bit, but it looks much better.
* Increase colorspace conversion efficiency.
This not only avoids a function call, it avoids the time-consuming
switch statement in conver_pixel (replacing it with a single
conditional on the byteswap flag + accounting for BGR/RGB during
palette creation)
* Buffer all the bytes of a single frame together. By reducing
low level write calls we get a decent speed increase even though
it increases data-shuffling a bit.
Together with some other changes that enable "double buffered" camera
capture, this gets me to 8.8fps capturing QVGA (320x240) gifs and
11fps capturing 240x240 square gifs.
This involves:
* Adding a new "L8" colorspace for colorconverters
* factoring out displayio_colorconverter_convert_pixel
* Making a minimal "colorspace only" version of displayio for the
unix port (testing purposes)
* fixing an error message
I only tested writing B&W animated images, with the following script:
```python
import displayio
import gifio
with gifio.GifWriter("foo.gif", 64, 64, displayio.Colorspace.L8) as g:
for i in range(0, 256, 14):
data = bytes([i, 255-i] * 32 + [255-i, i] * 32) * 32
print("add_frame")
g.add_frame(data)
# expected to raise an error, buffer is not big enough
with gifio.GifWriter("/dev/null", 64, 64, displayio.Colorspace.L8) as g:
g.add_frame(bytes([3,3,3]))
```
This increases write rates (of gifio from #5490) from about 2.4fps to over
5fps by making more efficient use of the SD card protocol.
Because of details of oofatfs, it usually manages 64 writes in a single
CMD25, then two writes in a different area of the SD card (presumably,
filesystem metadata). I couldn't find where to increase "64" to a higher
number. This may depend on the allocation size of the filesystem.
I tried preallocating too, but oddly it significantly lowered the write
rate.
Any trailing data is committed when the file is close()d, or when
the `sync` method of the SDCard object is called.
At present, Adafruit's rotary encoders all move 1 quadrature cycle per
detent, so we originally hard-coded division-by-4. However, other
encoders exist, including ones without detents, ones with 2 detents per
cycle, and others with 4 detents per cycle.
The new `divisor` property and constructor argument allows selecting
a divisor of 1, 2, or 4; with the default of 4 giving backward
compatibility.
The property is not supported (yet?) on espressif MCUs; it throws an
error if a value other than 4 is set.
F-strings, new machine.I2S class, ESP32-C3 support and LEGO_HUB_NO6 board
This release of MicroPython adds support for f-strings (PEP-498), with a
few limitations compared to normal Python. F-strings are essentially
syntactic sugar for "".format() and make formatting strings a lot more
convenient. Other improvements to the core runtime include pretty printing
OSError when it has two arguments (an errno code and a string), scheduling
of KeyboardInterrupt on the main thread, and support for a single argument
to the optimised form of StopIteration.
In the machine module a new I2S class has been added, with support for
esp32 and stm32 ports. This provides a consistent API for transmit and
receive of audio data in blocking, non-blocking and asyncio-based
operation. Also, the json module has support for the "separators" argument
in the dump and dumps functions, and framebuf now includes a way to blit
between frame buffers of different formats using a palette. A new,
portable machine.bitstream function is also added which can output a stream
of bits with configurable timing, and is used as the basis for driving
WS2812 LEDs in a common way across ports.
There has been some restructuring of the repository directory layout, with
all third-party code now in the lib/ directory. And a new top-level
directory shared/ has been added with first-party code that was previously
in lib/ moved there.
The docs have seen further improvement with enhancements and additions to
the rp2 parts, as well as a new quick reference for the zephyr port.
The terms master/slave have been replaced with controller/peripheral,
mainly relating to I2C and SPI usage. And u-module references have been
replaced with just the module name without the u-prefix to help clear up
the intended usage of modules in MicroPython.
For the esp8266 and esp32 ports, hidden networks are now included in WLAN
scan results. On the esp32 the RMT class is enhanced with idle_level and
write_pulses modes. There is initial support for ESP32-C3 chips with
GENERIC_C3 and GENERIC_C3_USB boards.
The javascript port has had its Makefile and garbage collector
implementation reworked so it compiles and runs with latest the Emscripten
using asyncify.
The mimxrt port sees the addition of hardware I2C and SPI support, as well
as some additional methods to the machine module. There is also support
for Hyperflash chips.
The nrf port now has full VFS storage support, enables source-line on
traceback, and has .mpy features consistent with other ports.
For the rp2 port there is now more configurability for boards, and more
boards added.
The stm32 port has a new LEGO_HUB_NO6 board definition with detailed
information how to get this LEGO Hub running stock MicroPython. There is
also now support to change the CPU frequency on STM32WB MCUs. And USBD_xxx
descriptor options have been renamed to MICROPY_HW_USB_xxx.
Thanks to everyone who contributed to this release: Amir Gonnen, Andrew
Scheller, Bryan Tong Minh, Chris Wilson, Damien George, Daniel Mizyrycki,
David Lechner, David P, Fernando, finefoot, Frank Pilhofer, Glenn Ruben
Bakke, iabdalkader, Jeff Epler, Jim Mussared, Jonathan Hogg, Josh Klar,
Josh Lloyd, Julia Hathaway, Krzysztof Adamski, Matúš Olekšák, Michael
Weiss, Michel Bouwmans, Mike Causer, Mike Teachman, Ned Konz, NitiKaur,
oclyke, Patrick Van Oosterwijck, Peter Hinch, Peter Züger, Philipp
Ebensberger, robert-hh, Roberto Colistete Jr, Sashkoiv, Seon Rozenblum,
Tobias Thyrrestrup, Tom McDermott, Will Sowerbutts, Yonatan Goldschmidt.
What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.
Main components
===============
all:
- fix signed shifts and NULL access errors from -fsanitize=undefined
- update to point to files in new shared/ directory
py core:
- mpstate: make exceptions thread-local
- mpstate: schedule KeyboardInterrupt on main thread
- mperrno: add MP_ECANCELED error code
- makeqstrdefs.py: don't include .h files explicitly in preprocessing
- mark unused arguments from bytecode decoding macros
- objexcept: pretty print OSError also when it has 2 arguments
- makeversionhdr: add --tags arg to git describe
- vm: simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode
- objexcept: make mp_obj_exception_get_value support subclassed excs
- support single argument to optimised MP_OBJ_STOP_ITERATION
- introduce and use mp_raise_type_arg helper
- modsys: optimise sys.exit for code size by using exception helpers
- objexcept: make mp_obj_new_exception_arg1 inline
- obj: fix formatting of comment for mp_obj_is_integer
- emitnative: reuse need_reg_all func in need_stack_settled
- emitnative: ensure stack settling is safe mid-branch
- runtime: fix bool unary op for subclasses of native types
- builtinimport: fix condition for including do_execute_raw_code()
- mkrules: automatically build mpy-cross if it doesn't exist
- implement partial PEP-498 (f-string) support
- lexer: clear fstring_args vstr on lexer free
- mkrules.mk: do submodule sync in "make submodules"
extmod:
- btstack: add missing call to mp_bluetooth_hci_uart_deinit
- btstack: check that BLE is active before performing operations
- uasyncio: get addr and bind server socket before creating task
- axtls-include: add axtls_os_port.h to customise axTLS
- update for move of crypto-algorithms, re1.5, uzlib to lib
- moduselect: conditionally compile select()
- nimble: fix leak in l2cap_send if send-while-stalled
- btstack/btstack.mk: use -Wno-implicit-fallthrough, not =0
- utime: always invoke mp_hal_delay_ms when >= to 0ms
- modbluetooth: clamp MTU values to 32->UINT16_MAX
- nimble: allow modbluetooth binding to hook "sent HCI packet"
- nimble: add "memory stalling" mechanism for l2cap_send
- uasyncio: in open_connection use address info in socket creation
- modujson: add support for dump/dumps separators keyword-argument
- modlwip: fix close and clean up of UDP and raw sockets
- modbluetooth: add send_update arg to gatts_write
- add machine.bitstream
- modframebuf: enable blit between different formats via a palette
lib:
- tinyusb: update to version 0.10.1
- pico-sdk: update to version 1.2.0
- utils/stdout_helpers: make mp_hal_stdout_tx_strn_cooked efficient
- axtls: switch to repo at micropython/axtls
- axtls: update to latest axtls 2.1.5 wih additional commits
- re1.5: move re1.5 code from extmod to lib
- uzlib: move uzlib code from extmod to lib
- crypto-algorithms: move crypto-algorithms code from extmod to lib
- update README's based on contents of these dirs
drivers:
- neopixel: add common machine.bitstream-based neopixel module
- neopixel: optimize fill() for speed
- neopixel: reduce code size of driver
- cyw43: fix cyw43_deinit so it can be called many times in a row
- cyw43: make wifi join fail if interface is not active
mpy-cross:
- disable stack check when building with Emscripten
Support components
==================
docs:
- library: document new esp32.RMT features and fix wait_done
- library: warn that ustruct doesn't handle spaces in format strings
- esp8266/tutorial: change flash mode from dio to dout
- replace master/slave with controller/peripheral in I2C and SPI
- rp2: enhance quickref and change image to Pico pinout
- rp2: update general section to give a brief technical overview
- library/utime.rst: clarify behaviour and precision of sleep ms/us
- library/uasyncio.rst: document stream readexactly() method
- library/machine.I2S.rst: fix use of sd pin in examples
- zephyr: add quick reference for the Zephyr port
- library/zephyr: add libraries specific to the Zephyr port
- templates: add unix and zephyr quickref links to top-index
- rename ufoo.rst to foo.rst
- replace ufoo with foo in all docs
- library/index.rst: clarify module naming and purpose
- library/builtins.rst: add module title
- library/network.rst: simplify socket import
- add docs for machine.bitstream and neopixel module
- library: fix usage of :term: for frozen module reference
- esp8266: use monospace for software tools
- reference: mention that slicing a memoryview causes allocation
examples: no changes specific to this component/port
tests:
- extmod: make uasyncio_heaplock test more deterministic
- cpydiff/modules_struct_whitespace_in_format: run black
- extmod/ujson: add tests for dump/dumps separators argument
- run-multitests.py: add broadcast and wait facility
- multi_bluetooth/ble_subscribe.py: add test for subscription
- extmod/vfs_fat_finaliser.py: ensure alloc at never-used GC blocks
- basics: split f-string debug printing to separate file with .exp
- pybnative: make while.py test run on boards without pyb.delay
tools:
- autobuild: add scripts to build release firmware
- remove obsolete build-stm-latest.sh script
- ci.sh: run apt-get update in ci_powerpc_setup
- makemanifest.py: allow passing flags to mpy-tool.py
- autobuild: add mimxrt port to build scripts for nightly builds
- pyboard.py: add cmd-line option to make soft reset configurable
- mpremote: swap order of PID and VID in connect-list output
- ci.sh: build unix dev variant as part of macOS CI
- ci.sh: build GENERIC_C3 board as part of esp32 CI
- autobuild: use separate IDF version to build newer esp32 SoCs
- autobuild: add FeatherS2 and TinyS2 to esp32 auto builds
- mpremote: add seek whence for mounted files
- mpremote: raise OSError on unsupported RemoteFile.seek
- autobuild: add the MIMXRT1050_EVKB board to the daily builds
- ci.sh: add mpy-cross build to nrf port
- codeformat.py: include ports/nrf/modules/nrf in code formatting
- gen-cpydiff.py: don't rename foo to ufoo in diff output
- autobuild: add auto build for Silicognition wESP32
- mpremote: fix connect-list in case VID/PID are None
- mpremote: add "devs" shortcut for "connect list"
- mpremote: remove support for pyb.USB_VCP in/out specialisation
- autobuild: don't use "-B" for make, it's already a fresh build
- pyboard.py: move --no-exclusive/--soft-reset out of mutex group
- pyboard.py: make --no-follow use same variable as --follow
- pyboard.py: add --exclusive to match --no-exclusive
- pyboard.py: make --no-soft-reset consistent with other args
- uncrustify: force 1 newline at end of file
- mpremote: bump version to 0.0.6
CI:
- workflows: add workflow to build and test javascript port
- workflows: switch from Coveralls to Codecov
- workflows: switch from lcov to gcov
- workflows: add workflow to build and test unix dev variant
The ports
=========
all ports:
- use common mp_hal_stdout_tx_strn_cooked instead of custom one
- update for move of crypto-algorithms, uzlib to lib
- rename USBD_VID/PID config macros to MICROPY_HW_USB_VID/PID
bare-arm port: no changes specific to this component/port
cc3200 port: no changes specific to this component/port
esp8266 port:
- add __len__ to NeoPixel driver to support iterating
- Makefile: add more libm files to build
- include hidden networks in WLAN.scan results
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
esp32 port:
- boards/sdkconfig.base: disable MEMPROT_FEATURE to alloc from IRAM
- add __len__ to NeoPixel driver to support iterating
- main: allow MICROPY_DIR to be overridden
- esp32_rmt: fix RMT looping in newer IDF versions
- esp32_rmt: enhance RMT with idle_level and write_pulses modes
- add new machine.I2S class for I2S protocol support
- machine_spi: calculate actual attained baudrate
- machine_hw_spi: use a 2 item SPI queue for long transfers
- machine_dac: add MICROPY_PY_MACHINE_DAC option, enable by default
- machine_i2s: add MICROPY_PY_MACHINE_I2S option, enable by default
- fix use of mp_int_t, size_t and uintptr_t
- add initial support for ESP32C3 SoCs
- boards/GENERIC_C3: add generic C3-based board
- modmachine: release the GIL in machine.idle()
- mphalport: always yield at least once in delay_ms
- machine_uart: add flow kw-arg to enable hardware flow control
- boards: add Silicognition wESP32 board configuration
- mpconfigport.h: enable reverse and inplace special methods
- include hidden networks in WLAN.scan results
- makeimg.py: get bootloader and partition offset from sdkconfig
- enable MICROPY_PY_FSTRINGS by default
- machine_hw_spi: release GIL during transfers
- machine_pin: make check for non-output pins respect chip variant
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
- boards: add GENERIC_C3_USB board with USB serial/JTAG support
javascript port:
- rework Makefile and GC so it works with latest Emscripten
- Makefile: suppress compiler errors from array bounds
- Makefile: change variable to EXPORTED_RUNTIME_METHODS
mimxrt port:
- move calc_weekday helper function to timeutils
- machine_spi: add the SPI class to the machine module
- moduos: seed the PRNG on boot using the TRNG
- boards: set vfs partition start to 1 MBbyte
- main: skip running main.py if boot.py failed
- main: extend the information returned by help()
- mimxrt_flash: remove commented-out code
- modmachine: add a few minor methods to the machine module
- machine_led: use mp_raise_msg_varg helper
- machine_i2c: add hardware-based machine.I2C to machine module
- add support for Hyperflash chips
- boards: add support for the MIMXRT1050_EVKB board
- machine_pin: implement ioctl for Pin
minimal port:
- Makefile: add support for building with user C modules
nrf port:
- modules: replace master/slave with controller/peripheral in SPI
- boards/common.ld: calculate unused flash region
- modules/nrf: add new nrf module with flash block device
- drivers: add support for using flash block device with SoftDevice
- mpconfigport.h: expose nrf module when MICROPY_PY_NRF is set
- README: update README.md to reflect internal file systems
- mpconfigport.h: tune FAT FS configuration
- Makefile: add _fs_size linker script override from make
- modules/uos: allow a board to configure MICROPY_VFS_FAT/LFS1/LFS2
- mpconfigport.h: enable MICROPY_PY_IO_FILEIO when an FS is enabled
- qstrdefsport.h: add entries for in-built FS mount points
- main: add auto mount and auto format hook for internal flash FS
- boards: enable needed features for FAT/LFS1/LFS2
- facilitate use of freeze manifest
- boards: set FROZEN_MANIFEST blank when SD present on nrf51 targets
- modules/scripts: add file system formatting script
- Makefile: set default manifest file for all targets
- mphalport: add dummy function for mp_hal_time_ns()
- boards: enable MICROPY_VFS_LFS2 for all target boards
- modules/uos: add ilistdir to uos module
- modules/nrf: add function to enable/disable DCDC
- enable source line on tracebacks
- set .mpy features consistent with documentation and other ports
pic16bit port: no changes specific to this component/port
powerpc port: no changes specific to this component/port
qemu-arm port: no changes specific to this component/port
rp2 port:
- use 0=Monday datetime convention in RTC
- machine_rtc: in RTC.datetime, compute weekday automatically
- CMakeLists.txt: suppress compiler errors for pico-sdk and tinyusb
- tusb_config.h: set CFG_TUD_CDC_EP_BUFSIZE to 256
- machine_uart: add hardware flow control support
- machine_uart: allow overriding default machine UART pins
- machine_i2c: allow boards to configure I2C pins using new macros
- machine_spi: allow boards to configure SPI pins using new macros
- machine_uart: fix poll ioctl to also check hardware FIFO
- machine_uart: fix read when FIFO has chars but ringbuf doesn't
- tusb_port: allow boards to configure USB VID and PID
- boards/ADAFRUIT_FEATHER_RP2040: configure custom VID/PID
- boards/ADAFRUIT_FEATHER_RP2040: configure I2C/SPI default pins
- boards/SPARKFUN_PROMICRO: configure UART/I2C/SPI default pins
- boards/SPARKFUN_THINGPLUS: configure I2C/SPI default pins
- boards: add Adafruit ItsyBitsy RP2040
- boards: add Adafruit QT Py RP2040
- boards: add Pimoroni Pico LiPo 4MB
- boards: add Pimoroni Pico LiPo 16MB
- boards: add Pimoroni Tiny 2040
- CMakeLists.txt: allow a board's cmake to set the manifest path
- enable MICROPY_PY_FSTRINGS by default
- Makefile: add "submodules" target, to match other ports
- rp2_flash: disable IRQs while calling flash_erase/program
- CMakeLists.txt: add option to enable double tap reset to bootrom
- mpconfigport.h: allow boards to add root pointers
samd port:
- add support for building with user C modules
stm32 port:
- softtimer: add soft_timer_reinsert() helper function
- mpbthciport: change from systick to soft-timer for BT scheduling
- provide a custom BTstack runloop that integrates with soft timer
- usb: make irq's default trigger enable all events
- boardctrl: skip running main.py if boot.py had an error
- sdio: fix undefined reference to DMA stream on H7
- dma: add DMAMUX configuration for H7 to fix dma_nohal_init
- main: call mp_deinit() at end of main
- adc: allow using ADC12 and ADC3 for H7
- adc: define the ADC instance used for internal channels
- adc: simplify and generalise how pin_adcX table is defined
- add new machine.I2S class for I2S protocol support
- boards/NUCLEO_F446RE: fix I2C1 pin assignment to match datasheet
- replace master/slave with controller/peripheral in I2C and SPI
- systick: always POLL_HOOK when delaying for milliseconds
- sdram: make SDRAM test cache aware, and optional failure with msg
- boards/NUCLEO_F446RE: enable CAN bus support
- boards: add support for SparkFun STM32 MicroMod Processor board
- uart: fix LPUART1 baudrate set/get
- uart: support low baudrates on LPUART1
- boards/STM32F429DISC: set correct UART2 pins and add UART3/6
- boards/NUCLEO_F439ZI: add board definition for NUCLEO_F439ZI
- boards/LEGO_HUB_NO6: add board definition for LEGO_HUB_NO6
- Makefile: update to only pull in used Bluetooth library
- README.md: update supported MCUs, and submodule and mboot use
- usbd_desc: rename USBD_xxx descriptor opts to MICROPY_HW_USB_xxx
- usbd_cdc_interface: rename USBD_CDC_xx opts to MICROPY_HW_USB_xx
- powerctrl: support changing frequency on WB MCUs
- boards/NUCLEO_H743ZI2: add modified version of NUCLEO_H743ZI
- mbedtls: fix compile warning about uninitialized val
- enable MICROPY_PY_FSTRINGS by default
- add implementation of machine.bitstream
- Makefile: allow GIT_SUBMODULES and LIBS to be extended
- stm32_it: support TIM17 IRQs on WB MCUs
- disable computed goto on constrained boards
- storage: make extended-block-device more configurable
- boards/LEGO_HUB_NO6: change SPI flash storage to use hardware SPI
- boards/LEGO_HUB_NO6: skip first 1MiB of SPI flash for storage
- boards/LEGO_HUB_NO6: add make commands to backup/restore firmware
teensy port: no changes specific to this component/port
unix port:
- modffi: add option to lock GC in callback, and cfun access
- Makefile: add back LIB_SRC_C to list of object files
- variants: enable help and help("modules") on standard and dev
- Makefile: disable error compression on arm-linux-gnueabi-gcc
windows port:
- Makefile: add .exe extension to executables name
- appveyor: update to VS 2017 and use Python 3.8 for build/test
zephyr port:
- machine_spi: add support for hardware SPI
The "scale" parameter wasn't exposed in the library, and there
were some problems in how it was handled.
This also fixes some types in the pixel-drawing functions.
complete pin mapping for Feather pins
stubbed out files needed for complilation. still to be modified
0 out all CPY modules in mpconfigboard.mk until we get the build running
add csv for pin generation for STM32L4R5
add F4R5 references in peripherals files
refactored out board files BECAUSE I AM AN IDIOT; add L4 series system clocks file from CubeMX
took a guess at the number of USB endpoint pairs to get the build done
guess was close, but wrong. It is 8
clean up peripheral DEFs
Fixes build error:
```
In file included from ../../py/mpstate.h:33,
from ../../py/mpstate.c:27:
../../py/misc.h: In function 'vstr_str':
../../py/misc.h:196:1: sorry, unimplemented: Thumb-1 hard-float VFP ABI
static inline char *vstr_str(vstr_t *vstr) {
^~~~~~
```
Sleuthing steps:
* verify that the feather_stm32f4_express board builds correctly
* put a `#error` at the bottom of the `mpstate.c` file.
* build for the feather and swan boards, with V=2 to capture the build command for that file.
* use a differencing tool to inspect the differences between the two invocations
* inspecting the differences, I saw a missing `-mcpu=cortex-m4` I tested by adding that to the Swan build command. The file built fine (stopping at the hard error, but no other warnings.)
A grep through the sources revealed where this flag was being set for the stm ports.
With this commit, the build gets further, but does not complete. The next exciting episode in this unfolding coding saga is just a commit away!
working build with minimal set of modules for the Blues Swan r5
chore:change header copyright name to Blues Wireless Contributors
USB operational. Fixed up clocks to be hardwired for LSE no HSE case. (Trying to combine HSE in there made the code much more complex, and I don't have a board to test it out on.)
USART working
adds support for `ENABLE_3V3` and `DISCHARGE_3V3` pins. I am surprised that pin definitions are quite low-level and don't include default direction and state, so the code currently has to initialize `ENABLE_3V3` pin as output. The LED takes over a second to discharge, so I wonder if the board startup code is not having the desired affect.
short circuit implementation of backup memory for the STM32L4
all the ports
remove company name from board name to be consistent with the Arduino board definition.
add default pins for I2C, SPI and UART, so that `board.I2C` et al. works as expected. Confirmed I2C timing.
fix board name
fix incorrect pin definition. add test to allow manual check of each output pin
analog IO
code changes for WebUSB. Doesn't appear to work, will revisit later.
ensure that `sys.platform` is available
checkin missing file
feat: make room for a larger filesystem so the sensor tutorial will fit on the device.
fix:(stm32l4r5zi.csv): merged AF0-7 and AF8-15 into single lines and removed extraneous headers mixed in with the data.
fix(parse_af_csv.py): pin index in the csv is 0 not 1, and AF index made 1 larger
chore(Swan R5): update peripherals pins from `parse_af_csv.py` output
optimize flash sector access
* Reduce the number of supported HID reports of IDs per descriptor.
This saves ~200 bytes in the default HID objects.
* (Not enabled) Compute QSTR attrs on init. This trades 1k RAM for
flash. Flash is the default (1).
The default KEYBOARD report descriptor had a signed/unsigned error,
and also could have allowed more keycodes. So I changed it, using the
very vanilla descriptor from a very plain extremely common commercial
keyboard, modifying it only have 5 LED's instead of 3, and added a
report ID.
* The new nonstandard '%S' format takes a pointer to compressed_string_t
and prints it
* The new mp_cprintf and mp_vcprintf take a format string that is a
compressed_string_t
* fix absolute_transform dirtying early instead of after the change, missing the draw
* fix transpose and mirror. (0,0) -> location in all vector shapes now in all rotations.
now works with all vector shapes, even those with internal reference locations
that are negative. All shape locations are anchored to their 0,0 but they can
display pixels from negative coordinates if the shape's location on the screen
would have room for it.
* add heuristic to avoid drawing area unnecessarily
* fix Polygon.points
* fix transpose
* fix mirror x and y
Known broken:
Polygon with negative Y coordinates does not work right.
* Removes VectorShape from user python interactions
* Re-integrates vectorio with displayio behind draw protocol implementations
* Implements draw protocol with VectorShape
* Composes VectorShape behaviors into Rectangle, Circle and Polygon
* Fixes terrible pixel garbage being left behind
* Improves redraw performance (heuristically) by tracking dirty area separately from current area.
Known Issues:
It does not work with transposed views.
Call `supervisor.disable_ble_workflow()` and the BLE workflow will
be disabled until the chip is reset.
This also includes a couple fixes:
1. Terminals can now be deinit by setting the tilegrid to NULL. This
prevents using the tilegrid before display is init.
2. Fix BLE serial send amount when sending more than a single packet.
Fixes#5049
This is a breaking change with previous palette semantic with respect to python code that uses vectorio.
Displayio has breaking changes in cpy 7 for Group's removal of max_size parameter so this is as good a
time as any to break everything.
Currently:
To color vectorio shapes correctly you have to pass in a palette with length 2. Palette[0] must be set transparent and palette[1] must be the color you want.
New:
To color vectorio shapes correctly you pass in a palette with length >= 1. Palette[0] will be the color of the shape.
Also improves pixels per second when skipping areas that aren't covered by the shape.
Before, when an OnDiskBitmap was a paletted bitmap type, the palette
was internal to the OnDiskBitmap, and it internally performed the palette
conversion itself. When using with a tilegrid, a ColorConverter() object
always had to be passed.
Now, an OnDiskBitmap has a "pixel_shader" property. If the bitmap is
a paletted bitmap type, it is a (modifiable) Palette object. Otherwise,
it is a ColorConverter() object as before. This allows palette effects
to be applied to paletted OnDiskBitmaps.
Code that used to say:
```python
face = displayio.TileGrid(odb, pixel_shader=displayio.ColorConverter())
```
must be updated to say:
```python
face = displayio.TileGrid(odb, pixel_shader=odb.pixel_shader)
```
Compatible code for 6.x and 7.x can say
```python
face = displayio.TileGrid(odb, pixel_shader=getattr(odb, 'pixel_shader', ColorConverter())
```
We can't handle rgbmatrix's interrupts from here until the display is
reinitialized, so set the display as paused.
With this change, I can survive multiple cycles with wifi+rgbmatrix
on an esp32s2. Before, it usually failed.
This also removes the need to pin share because we don't use the
status LED while user code is running.
The status flashes fallback to the HW_STATUS LED if no RGB LED is
present. Each status has a unique blink pattern as well.
One caveat is the REPL state. In order to not pin share, we set the
RGB color once. PWM and single color will be shutoff immediately but
DotStars and NeoPixels will hold the color until the user overrides
it.
Fixes#4133
Unify USB-related makefile var and C def as CIRCUITPY_USB.
Always define it as 0 or 1, same as all other settings.
USB_AVAILABLE was conditionally defined in supervisor.mk,
but never actually used to #ifdef USB-related code.
Loosely related to #4546
It is required to call .dirty() with appropriate arguments after modifications through the buffer protocol, or the display might not be updated correctly.
This is a modest code savings, but more importantly it reduces
boilerplate in bitmap-modifying routines.
Callers need only ensure they call displayio_bitmap_set_dirty_area in
advance of the bitmap modifications they perform.
(note that this assumes that no bitmap operation can enter background
tasks. If an operation COULD enter background tasks, it MUST re-dirty
the area it touches when it exits, simply by a fresh call to
set_dirty_area with the same area as before)
.. simplifying code in the process. For instance, now fill_region
uses area routines to order and constrain its coordinates.
Happily, this change also frees a modest amount of code space.
.. and simplify the implmentation of displayio_area_union
This _slightly_ changes the behavior of displayio_area_union:
Formerly, if one of the areas was empty, its coordinates were still
used in the min/max calculations.
Now, if one of the areas is empty, the result gets the other area's coords
In particular, taking the union of the empty area with coords (0,0,0,0)
with the non-empty area (x1,y1,x2,y2) would give the area (0,0,x2,y2)
before, and (x1,y1,x2,y2) after the change.
When reading uncompressed bitmap data directly, readinto can work
much more quickly than a Python-coded loop.
On a Raspberry Pi Pico, I benchmarked a modified version of
adafruit_bitmap_font's pcf reader which uses readinto instead of
the existing code. My test font was a 72-point file created from Arial.
This decreased the time to load all the ASCII glyphs from 4.9 seconds to
just 0.44 seconds.
While this attempts to support many pixel configurations (1/2/4/8/16/24/32
bpp; swapped words and pixels) only the single combination used by
PCF fonts was tested.
This is a first go at it, done by naive replacing of all array
operations with corresponding operations on the list. Note that
there is a lot of unnecessary type conversions, here. Also, list_pop
has been copied, because it's decalerd STATIC in py/objlist.h
Since we want to expose the list of group's children to the user,
we should only have the original objects in it, without any other
additional data, and compute the native object as needed.
* Comment on the reason for scaling by 256
* Divide by 256 instead of shifting
* fix a cast; eliminate an unneeded roundf() to get a few bytes code back
On the Pico, this increases the "fill rate" of
pixels[:] = newvalues
considerably. On a strip of 240 RGB LEDs, auto_write=False, the timings
are:
|| Brightness || Before || After || Improvement ||
|| 1.0 || 117 kpix/s || 307 kpix/s || 2.62x ||
|| 0.07 || 117 kpix/s || 273 kpix/s || 2.33x ||
It's worth noting that even the "before" rate is fast compared to the
time to transmit a single neopixel, but any time we can gain back
in the whole pipeline will let marginal animations work a little better.
To set all the pixels in this way and then show() gives a pleasant bump
to the framerate, from about 108Hz to 124Hz (1.15x)
The main source of speed-up is using integer math instead of floating
point math for the calculation of the post-scaled pixel values. A slight
secondary gain is achieved by avoiding the scaling altogether when
the scale factor is 1.0.
Because the math is not exactly the same, some scaled pixel values may
change by +- 1 RGBW "step". In practice, this is unlikely to matter.
The gains are bigger on the Pico and other M0 microcontrollers than M4
microcontrollers with floating point math in the hardware.
Happily, flash size is also improved a bit on the Pico build I did,
going from
> 542552 bytes used, 506024 bytes free in flash firmware space out of 1048576 bytes (1024.0kB).
to
> 542376 bytes used, 506200 bytes free in flash firmware space out of 1048576 bytes (1024.0kB).
Also found a race condition between timer_disable and redraw, which
would happen if I debugger-paused inside common_hal_rgbmatrix_timer_disable
or put a delay or print inside it. That's what pausing inside reconstruct
fixes.
So that the "right timer" can be chosen, `timer_allocate` now gets the `self`
pointer. It's guaranteed at this point that the pin information is accurate,
so you can e.g., find a PWM unit related to the pins themselves.
This required touching each port to add the parameter even though it's
unused everywhere but raspberrypi.
* Introduce explicit serpentine: bool argument instead of using negative
numbers (thanks, ghost of @tannewt sitting on one shoulder)
* Fix several calculations of height
Testing performed (matrixportal):
* set up a serpentine 64x64 virtual display with 2 64x32 tiles
* tried all 4 rotations
* looked at output of REPL
Changed calls: PointSize(), LineWidth(), VertexTranslateX() and VertexTranslateY()
Units for all the above are now pixels, not fixed-point integers. This matches OpenGL.
Docstrings updated accordingly
The RP2040 is new microcontroller from Raspberry Pi that features
two Cortex M0s and eight PIO state machines that are good for
crunching lots of data. It has 264k RAM and a built in UF2
bootloader too.
Datasheet: https://pico.raspberrypi.org/files/rp2040_datasheet.pdf
Microsoft documentation says:
> If biCompression equals BI_RGB and the bitmap uses 8 bpp or less, the bitmap has a color table immediatelly following the BITMAPINFOHEADER structure. The color table consists of an array of RGBQUAD values. The size of the array is given by the biClrUsed member. If biClrUsed is zero, the array contains the maximum number of colors for the given bitdepth; that is, 2^biBitCount colors.
Formerly, we treated 0 colors as "no image palette" during construction,
but then during common_hal_displayio_ondiskbitmap_get_pixel indexed into
the palette anyway. This could have unpredictable results. On a pygamer,
it gave an image that was blue and black. On magtag, it gave a crash.
The transparent_color field was never initialized. I _think_ this means
its value was always set to 0, or the blackest of blacks. Instead,
initialize it to the sentinel value, newly given the name
NO_TRANSPARENT_COLOR.
This exposed a second problem: The test for whether there was an existing
transparent color was wrong (backwards). I am guessing that this was not
found due to the first bug; since the converter had a transparent color,
the correct test would have led to always getting the error "Only one
color can be transparent at a time".
Closes#3723
Hybrid allocation is now part of the infrastructure. Moving memory contents would not be necessary because displayio can recreate them, but does not hurt.
Hybrid allocation is now part of the infrastructure. Moving memory contents would not be necessary because displayio can recreate them, but does not hurt.
This allows calls to `allocate_memory()` while the VM is running, it will then allocate from the GC heap (unless there is a suitable hole among the supervisor allocations), and when the VM exits and the GC heap is freed, the allocation will be moved to the bottom of the former GC heap and transformed into a proper supervisor allocation. Existing movable allocations will also be moved to defragment the supervisor heap and ensure that the next VM run gets as much memory as possible for the GC heap.
By itself this breaks terminalio because it violates the assumption that supervisor_display_move_memory() still has access to an undisturbed heap to copy the tilegrid from. It will work in many cases, but if you're unlucky you will get garbled terminal contents after exiting from the vm run that created the display. This will be fixed in the following commit, which is separate to simplify review.
* Initialize the EPaper display on the MagTag at start.
* Tweak the display send to take a const buffer.
* Correct Luma math
* Multiply the blue component, not add.
* Add all of the components together before dividing. This
reduces the impact of truncated division.
After calling board.SPI().deinit(), calling board.SPI() again would return the unusable deinited object and there was no way of getting it back into an initialized state until the end of the session.
Fixes#3581.
Pins were marked as never_reset by common_hal_displayio_fourwire_construct() and common_hal_sharpdisplay_framebuffer_construct(), but these marks were never removed, so at the end of a session after displayio.release_displays(), {spi|i2c}_singleton would be set to NULL but the pins would not be reset. In the next session, board.SPI() and board.I2C() were unable to reconstruct the object because the pins were still in use.
For symmetry with creation of the singleton, add deinitialization before setting it to NULL in reset_board_busses(). This makes the pins resettable, so that reset_port(), moved behind it, then resets them.
At the end of a session that called displayio.release_displays() (and did not initialize a new display), a board.I2C() bus that was previously used by a display would wrongly be considered still in use. While I can’t think of any unrecoverable problem this would cause in the next session, it violates the assumption that a soft reboot resets everything not needed by displays, potentially leading to confusion.
By itself, this change does not fix the problem yet - rather, it introduces the same issue as in #3581 for SPI. This needs to be solved in the same way for I2C and SPI.
This is enabled by #3482
I was unable to determine why previously I had added sizeof(void*)
to the GC heap allocation, so I removed that code as a mistake.
@cwalther determined that for boards with 2 displays (monster m4sk),
start_terminal would be called for each one, leaking supervisor heap
entries.
Determine, by comparing addresses, whether the display being acted on
is the first display (number zero) and do (or do not) call start_terminal.
stop_terminal can safely be called multiple times, so there's no need
to guard against calling it more than once.
Slight behavioral change: The terminal size would follow the displays[0]
size, not the displays[1] size
If the display is paused, `_PM_swapbuffer_maybe` will never return.
So, when brightness is 0, refresh does nothing. This makes it necessary
to update the display when unpausing.
Closes: #3524
A call to supervisor_start_terminal remained in
common_hal_displayio_display_construct and was copied to other display
_construct functions, even though it was also being done in
displayio_display_core_construct when that was factored out.
Originally, this was harmless, except it created an extra allocation.
When investigating #3482, I found that this bug became harmful,
especially for displays that were created in Python code, because it
caused a supervisor allocation to leak.
I believe that it is safe to merge #3482 after this PR is merged.
An RGBMatrix has no bus and no bus_free method. It is always possible
to refresh the display.
This was not a problem before, but the fix I suggested (#3449) added
a call to core_bus_free when a FramebufferDisplay was being refreshed.
This was not caught during testing.
This is a band-aid fix and it brings to light a second problem in which
a SharpDisplay + FrameBuffer will not have a 'bus' object, and yet does
operate using a shared SPI bus. This kind of display will need a
"bus-free" like function to be added, or it can have problems like
#3309.
It was incorrect to NULL out the pointer to our heap allocated buffer in
`reset`, because subsequent to framebuffer_reset, but while
the heap was still active, we could call `get_bufinfo` again,
leading to a fresh allocation on the heap that is about to be destroyed.
Typical stack trace:
```
#1 0x0006c368 in sharpdisplay_framebuffer_get_bufinfo
#2 0x0006ad6e in _refresh_display
#3 0x0006b168 in framebufferio_framebufferdisplay_background
#4 0x00069d22 in displayio_background
#5 0x00045496 in supervisor_background_tasks
#6 0x000446e8 in background_callback_run_all
#7 0x00045546 in supervisor_run_background_tasks_if_tick
#8 0x0005b042 in common_hal_neopixel_write
#9 0x00044c4c in clear_temp_status
#10 0x000497de in spi_flash_flush_keep_cache
#11 0x00049a66 in supervisor_external_flash_flush
#12 0x00044b22 in supervisor_flash_flush
#13 0x0004490e in filesystem_flush
#14 0x00043e18 in cleanup_after_vm
#15 0x0004414c in run_repl
#16 0x000441ce in main
```
When this happened -- which was inconsistent -- the display would keep
some heap allocation across reset which is exactly what we need to avoid.
NULLing the pointer in reconstruct follows what RGBMatrix does, and that
code is a bit more battle-tested anyway.
If I had a motivation for structuring the SharpMemory code differently,
I can no longer recall it.
Testing performed: Ran my complicated calculator program over multiple
iterations without observing signs of heap corruption.
Closes: #3473
This already begins obscuring things, because now there are two sets of
shared-module functions for manipulating the same structure, e.g.,
common_hal_canio_remote_transmission_request_get_id and
common_hal_canio_message_get_id
Tested & working:
* Send standard packets
* Receive standard packets (1 FIFO, no filter)
Interoperation between SAM E54 Xplained running this tree and
MicroPython running on STM32F405 Feather with an external
transceiver was also tested.
Many other aspects of a full implementation are not yet present,
such as error detection and recovery.
The expectations of displayio.Display and frambufferio.FramebufferDisplay
are different when it comes to rotation.
In displayio.Display, if you call core_construct with a WxH = 64x32
and rotation=90, you get something that is 32 pixels wide and 64 pixels
tall in the LCD's coordinate system.
This is fine, as the existing definitions were written to work like this.
With framebuffer displays, however, the underlying framebuffer (such as
RGBMatrix) says "I am WxH pixels wide in my coordinate system" and the
constructor is given a rotation; when the rotation indicates a transpose
that means "exchange rows and columns, so that to the Groups displayed
on it, there is an effectively HxW pixel region for use".
Happily, we already have a set_rotation method. Thus (modulo the time
spent debugging things anyway:) the fix is simple: Always request no
rotation from core_construct, then immediately fix up the rotation
to match what was requested.
Testing performed: 32x16 RGBMatrix on Metro M4 Express (but using
the Airlift firmware, as this is the configuration the error was reported
on):
* initially construct display at 0, 90, 180, 270 degrees
* later change angle to 0, 90, 180, 270 degrees
* no garbled display
* no safe mode crashes
e.g., allocating a 192x32x6bpp matrix would be enough to trigger this
reliably on a Metro M4 Express using the "memory hogging" layout.
Allocating 64x32x6bpp could trigger it, but somewhat unreliably.
There are several things going on here:
* we make the failing call with interrupts off
* we were throwing an exception with interrupts off
* protomatter failed badly in _PM_free when it was partially-initialized
Incorporate the fix from protomatter, switch to a non-throwing malloc
variant, and ensure that interrupts get turned back on.
This decreases the quality of the MemoryError (it cannot report the size
of the failed allocation) but allows CircuitPython to survive, rather
than faulting.
