All variants now use extmod/moduos.c as their uos module implementation.
In particular this means they all have MICROPY_VFS enabled and use VfsPosix
for their filesystem.
As part of this, the available functions in uos become more consistent with
other ports:
- coverage variant gets uos.urandom
- minimal and standard variant get: unlink, chdir, getcwd, listdir
Signed-off-by: Damien George <damien@micropython.org>
Ensure the symmetry of PWM: the duty rate of X and Q channels was not 50%,
when it should have been. That is evident at high frequencies, like 15Mhz
or 37.5 MHz. At low frequencies the deviation mattered less. The A/B
channels were fine.
Also round up or down non-integer division factors. Before, always the
floor value was used.
That caused Ethernet to lock up at high data rates after ~200MByte data
average in a row. Tested now with data bursts up to 10 GByte and overall
data rates of ~8MByte/s at the Eth100 port.
Sometimes frames could not be sent immediately because the controller was
still busy with previous frames. Then, an error was returned to lwip.
This fix adds a limited number of retries for this busy state, waiting
100µs before the next attempt. Typically the transmit succeeds now at the
second attempt.
Second change: Reset the controller for a clean state after soft reset.
OCOTP_Init() has been removed from mphalport.c. The library files are
missing for the MIMXRT1015, and for just reading the OCOTP the Init is not
required.
The disk_access header was moved to a different path in Zephyr v2.6.0.
The old path was deprecated for two releases (v2.6.0 and v2.7.0) and
will no longer be supported after Zephyr v2.7.0.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
If setting the frequency to a value used already by an existing timer, this
timer will be used. But still, the duty cycle for that channel may have to
be changed.
Fixes issues #8306 and #8345.
If MicroPython threads are enabled, loops waiting for an incoming event
should release the GIL and suspend, allowing other tasks to run while they
wait.
Prior to this commit, the problem can easily be observed by running a
thread that is both busy and regularly releases the GIL (for example a loop
doing something then sleeping a few ms after each iteration). When the
main task is at the REPL, the thread is significantly stalled. If the main
task is manually made to release the GIL (for example, by calling
utime.sleep_ms(500)) the other thread can be seen immediately working at
the expected speed again.
Additionally, there are various instances in where blocking functions run
MICROPY_EVENT_POLL_HOOK in a loop while they wait for a certain event/
condition. For example the uselect methods poll objects to determine
whether data is available, but uses 100% of CPU while it does, constantly
calling MICROPY_EVENT_POLL_HOOK in the process.
The MICROPY_EVENT_POLL_HOOK macro is only ever used in waiting loops, where
(if threads are enabled) it makes sense to yield for a single tick so that
these loops do not consume all CPU cycles but instead other threads may
execute. (In fact, the thing these loops wait for may even indirectly or
directly depend on another task being able to run.)
This change moves the sleep that was inside the REPL input function to
inside the MICROPY_EVENT_POLL_HOOK macro, where the GIL is already being
released, solving both the blocking REPL issue and the 100% CPU use issue
at the same time.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
The stack (and arg) of core1 is itself a root pointer, not just the entries
in it. Without this fix the GC could reclaim the entire stack (and
argument object).
Fixes issues #7124 and #7981.
.py files are valid source files and shouldn't be ignored. This line was
from the early days when .py files in the unix directory were used for
testing.
Signed-off-by: Damien George <damien@micropython.org>
The xmlns attribute is required for older msbuild version (e.g. for
VS2015). Add it where needed, and reorder the attributes so all
files look the same.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
The unix port's main.c gets used by unix and windows ports, and with a
variety of compilers, so it's convenient to see which version is actually
being used immediately when starting micropython. This is similar to what
CPython does.
If MBOOT_BOARD_ENTRY_INIT is defined by a board then that function must now
make sure system clocks are configured, eg by calling mboot_entry_init().
Signed-off-by: Damien George <damien@micropython.org>
If a board wants to customise the clocks it can define the following:
MBOOT_CLK_PLLM
MBOOT_CLK_PLLN
MBOOT_CLK_PLLP
MBOOT_CLK_PLLQ
MBOOT_CLK_PLLR (only needed on STM32H7)
MBOOT_FLASH_LATENCY
MBOOT_CLK_AHB_DIV
MBOOT_CLK_APB1_DIV
MBOOT_CLK_APB2_DIV
MBOOT_CLK_APB3_DIV (only needed on STM32H7)
MBOOT_CLK_APB4_DIV (only needed on STM32H7)
Signed-off-by: Damien George <damien@micropython.org>
In the `after_test` section, the current directory is `ports/windows` when
tests are run, so running `run-tests.py` without changing the directory or
specifying a path causes a file not found error.
This commit fixes the problem by changing the directory before calling
`run-tests.py`.
Signed-off-by: David Lechner <david@pybricks.com>
The UART hardware flow control was not working correctly, the receive FIFO
was always fetched and RTS was never deasserted. This is not a problem
when hardware flow control is not used: normally, if the receive FIFO is
full, the UART receiver won't receive data into the FIFO anymore, but the
current implementation fetches from the FIFO and discards it instead.
The problem is that data is discarded even when RTS is enabled.
This commit fixes the issue by only taking from the FIFO if there is room
in the ring buffer to put the character.
Signed-off-by: YoungJoon Chun <yjchun@mac.com>
Prior to this fix, if the ADC atten value was not explicitly given then
adc1_config_channel_atten() would never be called.
Fixes issue #8275.
Signed-off-by: Damien George <damien@micropython.org>
The correct day-of-week is stored in the RTC (0=Monday, 6=Sunday) so there
is no need to adjust it for the return value of time.localtime().
Fixes issue #7889.
Signed-off-by: Damien George <damien@micropython.org>
The default stm32lib remains lib/stm32lib, but it can now be easily
overriden at build time by specifying STM32LIB_DIR, or STM32LIB_CMSIS_DIR
and STM32LIB_HAL_DIR.
Signed-off-by: Damien George <damien@micropython.org>
This also fixes a possible race condition when exiting initialisation mode:
reading then writing to ISR (via ISR &= ~RTC_ISR_INIT) will clear any flags
that were set by the hardware between the read and the write. The correct
way to clear just the INIT bit is to just do a single write via ISR =
~RTC_ISR_INIT, which will not clear any other flags (they must be written
to 0 to clear), and that is exactly what LL_RTC_DisableInitMode does.
Signed-off-by: Damien George <damien@micropython.org>
And don't assert on the sector number in sector_erase, so it can support
erasing arbitrary sectors.
Signed-off-by: Damien George <damien@micropython.org>
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c. Enabling MICROPY_PY_MACHINE will include this
module.
As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).
Signed-off-by: Damien George <damien@micropython.org>
This change allows the same heap allocation rules to be used when using
malloc regardless if the board has SPRAM or normal RAM.
Integrating with the esp32-camera for example requires that ESP32 SPRAM be
allocatable using the esp-idf capabilities aware allocation functions. In
the case of esp32-camera it's for the framebuffer.
Detect when CONFIG_SPIRAM_USE_MALLOC is in use and use the standard
automatic configuration of leaving 1/2 of the SPRAM available to other
FreeRTOS tasks.
For example the ESP32-C3 has 2 TX channels and 2 RX channels in total, and
in this case channel 1 must be the default for bitstream.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support for the STM32G4 series of MCUs, and a board
definition for NUCLEO_G474RE. This board has the REPL on LPUART1 which is
connected to the on-board ST-link USB-UART.
It's needed at least on F4 because this file overrides the weak function
HAL_RCC_DeInit() from hal_rcc.c.
Signed-off-by: Damien George <damien@micropython.org>
PLL3-Q is more reliable than PLL1-Q for the USB clock source when entering
mboot from various reset states (eg power on vs MCU reset). (It was found
that if the main application used PLL3-Q then sometimes the USB clock
source would stay stuck on PLL3-Q and not switch to PLL1-Q after a reset.)
Other related changes:
- SystemCoreClockUpdate() should be called on H7 because the calculation
can be involved in some cases.
- __set_PRIMASK(0) should be called because on H7 the built-in ST DFU
bootloader exits with IRQs disabled.
Signed-off-by: Damien George <damien@micropython.org>
H7 MCUs have ECC and writes do not go through to SRAM until 64-bits have
been written (on another location is written). So use 64-bit writes for
the bootloader-state variable so it is committed before the system reset.
As part of this change, the lower byte of the bootloader address in
BL_STATE must now be the magic number 0x5a5 for the state to be valid
(previously this was 0x000 which is not as robust).
Signed-off-by: Damien George <damien@micropython.org>
The original code used a independent state with regards to the interrupt.
During heavy bus error conditions the internal state could become
out-of-sync with the interrupts.
Further explanation: during the development of an application using CAN
communication, a interrupt-run-away was found in some situations. It was
found that the error interrupt triggered (Warning, Passive or Bus-Off, all
triggered it) the run-away. The only recovery was a reset.
Two problems were found:
- the error interrupt is enabled but not cleared in the interrupt routine;
- an internal variable 'State' that was used to track the message received
state (empty, new, full, overflow) that was not directly related to
interrupt that indicated the state.
In this commit these issues are fixed by adding more values for the
interrupt reason (warning, passive, bus off) and clearing the error
interrupts, and making the internal state directly dependent on the
interrupt state for received messages.
Furthermore, introducing the FIFO1 in the CAN receive stage, another issue
existed. Even if the messages are received into the FIFO1 (by selecting
message filtering for FIFO0 and FIFO1), the interrupt firing was indicating
FIFO0 Rx. The configuration of the interrupts for this is now also fixed.
The CAN peripheral has 2 interrupt lines going into the NVIC controller.
The assignment of the interrupt reasons to these 2 interrupt lines was
missing. Now the reception of FIFO1 messages triggers the second interrupt
line. Other interrupts (Rx FIFO0 and bus error) are assigned to the first
interrupt line.
Tested on a Nucleo-G474, and also checked the HAL function to work with the
H7 family.
The board.json file is intentionally excluded, until the board will be
sold. But including it into the mimxrt series make it easier to keep
the build up-to-date.
- Manufacturer, set by MICROPY_HW_USB_STR_MANUF; default "MicroPython"
- Board name, as set by MICROPY_HW_BOARD_NAME
- Unique-ID, same as returned by machine.unique_id()
- USB Vendor ID, as set by MICROPY_HW_USB_VID; default 0xf055
- USB Product ID, as set by MICROPY_HW_USB_PID; default 0x9802
Using the keyword argument cs=nnn in the constructor. The cs1
pin has to be defined in mpconfigboard.h.
Note: Only a few boards have the CS1 pin exposed to the connectors.
The Pin config setting by IOMUXC_SetPinConfig() is supplied by a
bit pattern. That pattern is specific for a MCU family. In preparation
for supporting the MIMXRT117x family, the constant bit pattern is
replaced by a function call, such that the bit pattern is created
at a single place. The code for this functions was taken from
machine_pin.c.
Note: A working port for the MIMXRT1176 exists already.
This library file has a bug, in that TransferBlocking returns before the
transfer has finished. That is a problem if a write follows immediately
a read.
If in a board's mkconfigboard.mk the following symbol is set:
MICROPY_HW_BOARD_FLASH_FILES = 1
then the files:
($BOARD)_flexspi_flash_config.h and
qspi_nor_flash_config.c and/or
qspi_hyper_flash_config.c
are expected in the board directory. Otherwise the common files from
the hal directory are used.
The keyword "af" has been deprecated for some time and "alt" should be used
instead (but "af" still works).
Signed-off-by: Damien George <damien@micropython.org>
This follows up on #5489, where we changed the esp32 core pinning to core 0
in order to work around an issue with IDF < 4.2.0. Now that IDF > 4.2.0 is
available, we allow pinning back to core 1, which eliminates some
problematic callback latency with WiFi enabled.
NimBLE is also pinned to core 1 - the same core as MicroPython - when using
IDF >=4.2.
Rework the ADC implementation to follow the improved ADC/ADCBlock API.
This adds support for calibrated voltage readings and the ADC2 block. The
ADC API is backwards compatible with what it was before this change.
Resolves#6219.
The RP2040 I2C hardware can do writes of length 1 and 2, just not of length
0. So only use software I2C for writes of length 0, to improve
performance.
Also increase the software I2C timeout for zero-length writes to
accommodate the behaviour of a wider range of I2C devices.
Fixes issue #8167.
Signed-off-by: Damien George <damien@micropython.org>
Without these methods a lot of existing "portable" scripts are broken.
This change improves portability by making rp2 machine.UART more compliant
with the documented machine UART interface.
So that a board can access other HAL_RCC functions if it needs them (this
was not possible previously by just adding hal_rcc.c to the src list for a
board because it would clash with the custom HAL_RCC_GetHCLKFreq function).
Signed-off-by: Damien George <damien@micropython.org>
This adds MBEDTLS_MD_SHA1 to the list of default hashes for TLS 1.2
handshake signatures. Although SHA-1 is weak, this option is turned on in
the default mbedtls configuration file, and allows better compatibility
with older servers. In particular it allows an stm32-mbedtls-based client
to connect to an axtls-based client (eg default unix port and esp8266).
Signed-off-by: Damien George <damien@micropython.org>
The bit-bang implementation was replaced with the RMT implementation in
599b61c086. This commit brings back that
bit-bang code, and allows it to be selected via the new static method:
esp32.RMT.bitstream_channel(None)
The bit-bang implementation may be useful if the RMT needs to be used for
something else, or if bit-banging is more stable in certain applications.
Signed-off-by: Damien George <damien@micropython.org>
MicroPython currently runs on core 0 of the esp32. Calling
rmt_driver_install will mean that the RMT interrupt handler is also
serviced on core 0. This can lead to glitches in the RMT output if
WiFi is enabled (for esp32.RMT and machine.bitstream).
This patch calls rmt_driver_install on core 1, ensuring that the RMT
interrupt handler is serviced on core 1. This prevents glitches.
Fixes issue #8161.
Signed-off-by: Damien George <damien@micropython.org>
The main Makefile builds the mpy-cross executable automatically if
it doesn't exist since 78718fffb1,
so build it first to make sure it doesn't get needlessly rebuilt.
This board has only 2MiB of flash so the build needs to be reduced in size
to fit. Commit 549448e8bb made all boards
build with -O2 by default (for performance) so this overrides that default.
Signed-off-by: Damien George <damien@micropython.org>
This is needed because these ports allocate mbedtls data on the MicroPython
heap, and SSL socket objects must be fully cleaned up when they are garbage
collected, to free this memory allocated by mbedtls. As part of this,
gc_sweep_all() will now ensure that the MP_STATE_PORT(mbedtls_memory)
linked-list is fully deallocated on soft reset.
Signed-off-by: Damien George <damien@micropython.org>
CONFIG_USB was removed in Zephyr v2.7.0 after some Kconfig rework that
made it sufficient to use CONFIG_USB_DEVICE_STACK only.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
Updates the Zephyr port to get the UART console device from devicetree
instead of Kconfig. The Kconfig option CONFIG_UART_CONSOLE_ON_DEV_NAME
was removed in Zephyr v2.7.0.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
The reboot header was moved to a different path in Zephyr v2.6.0. The
old path was deprecated for two releases (v2.6.0 and v2.7.0) and will no
longer be supported after Zephyr v2.7.0.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
As a prerequisite to upgrading to Zephyr v2.7.0, upgrade the minimum
CMake version required for the Zephyr port to 3.20.0.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
After changing the bitstream implementation to use the RMT driver in
commit 72d86158121e32bbabaeade08f449d507bf40f9a
("esp32/machine_bitstream.c: Replace with RMT-based driver."), using
multiple `Neopixel` instances shows signal duplication between the
instances (i.e. a `write()` on one instance is written to all instances).
On invocation, the rmt driver configures the GPIO matrix to route the
output signal to the respective GPIO pin. When called for a different
`NeoPixel` instance using a different pin, the new route is established,
but the old route still exists. Now, the RMT output signal is sent to both
pins.
Fix this by setting the standard GPIO output function for the current pin
after uninstalling the RMT driver.
Signed-off-by: Simon Baatz <gmbnomis@gmail.com>
