.. it needs to operate on a FILE* rather than FIL depending on
the build.
Note that this is comparing output to expected, not to cpython dotenv
package. Because run-tests.py starts the CPython interpreter with the
'-S' (skip site initialization) flag, pip-installed packages are
not available for import inside a test file. Instead, the exp
file is generated manually:
```
circuitpython/tests$ python3 circuitpython/dotenv_test.py > circuitpython/dotenv_test.py.exp
```
Unfortunately, the test fails on test e15:
```diff
FAILURE /home/jepler/src/circuitpython/tests/results/circuitpython_dotenv_test.py
--- /home/jepler/src/circuitpython/tests/results/circuitpython_dotenv_test.py.exp 2022-10-04 09:48:16.307703128 -0500
+++ /home/jepler/src/circuitpython/tests/results/circuitpython_dotenv_test.py.out 2022-10-04 09:48:16.307703128 -0500
@@ -14,7 +14,7 @@
line
e13 e13value
e14 None
-e15 e15value
+e15 None
e16 #
e17 def
e18 #has a hash
```
`b'\xaa \xaa'.count(b'\xaa')` now (correctly) returns 2 instead of 1.
Fixes issue #9404.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Allows optimisation of cases like:
import micropython
_DEBUG = micropython.const(False)
if _DEBUG:
print('Debugging info')
Previously the 'if' statement was only optimised out if the type of the
const() argument was integer.
The change is implemented in a way that makes the compiler slightly smaller
(-16 bytes on PYBV11) but compilation will also be very slightly slower.
As a bonus, if const support is enabled then the compiler can now optimise
const truthy/falsey expressions of other types, like:
while "something":
pass
... unclear if that is useful, but perhaps it could be.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This uses the frozentest.mpy that is also used by ports/minimal.
Also fixes two bugs that these new tests picked up:
- File extension matching in manifestfile.py.
- Handling of freeze_mpy results in makemanifest.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
frozentest.mpy was previously duplicated in ports/minimal and
ports/powerpc.
This needs to be re-generated on every .mpy version increase, so might as
well just have a single copy of it.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prevent handle leaks when file objects aren't closed explicitly and
fix some MICROPY_CPYTHON_COMPAT issues: this wasn't properly adhered
to because #ifdef was used so it was always on, and closing files
multiple times should be avoided unconditionally.
This matches class `__dict__`, and is similarly gated on
MICROPY_CPYTHON_COMPAT. Unlike class though, because modules's globals are
actually dict instances, the result is a mutable dictionary.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.
With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version. A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION. This allows 3
additional updates to the native ABI per bytecode revision.
The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.
This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
to a mp_getiter_iternext_custom_t instance, which then defines both
getiter and iternext.
And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.
Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a latent issue that wasn't caught by CI because there was no
configuration that had both stackless+uasyncio.
The previous check to skip with stackless builds only worked when the
bytecode emitter was used by default. Force the check to use the bytecode
emitter.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally. Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.
If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed. In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.
Fixes issues #6568 and #8506.
Rather than drawing the entire boundary to catch missing pixels, just
detect the cases where boundary pixels are skipped during node calculation
and pre-emptively draw them then.
This adds 72 bytes on PYBV11, but makes filled poly() 20% faster.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Add method for drawing polygons.
For non-filled polygons, uses the existing line-drawing code to render
arbitrary polygons using the given coords list, at the given x,y position,
in the given colour.
For filled polygons, arbitrary closed polygons are rendered using a fast
point-in-polygon algorithm to determine where the edges of the polygon lie
on each pixel row.
Tests and documentation updates are also included.
Signed-off-by: Mat Booth <mat.booth@gmail.com>
Rework the conversion of floats to decimal strings so it aligns precisely
with the conversion of strings to floats in parsenum.c. This is to avoid
rendering 1eX as 9.99999eX-1 etc. This is achieved by removing the power-
of-10 tables and using pow() to compute the exponent directly, and that's
done efficiently by first estimating the power-of-10 exponent from the
power-of-2 exponent in the floating-point representation.
Code size is reduced by roughly 100 to 200 bytes by this commit.
Signed-off-by: Dan Ellis <dan.ellis@gmail.com>
This is useful in situations where the ThreadSafeFlag is reused and needs
to be cleared of any previous, unwanted event.
For example, clear the flag at the start of an operation, trigger the
operation (eg an I2C write), then (a)wait for an external event to set the
flag (eg a pin IRQ). Further events may trigger the flag again but these
are unwanted and should be cleared before the next cycle starts.
This commit adds the bytes methods to bytearray, matching CPython. The
existing implementations of these methods for str/bytes are reused for
bytearray with minor updates to match CPython return types.
For details on the CPython behaviour see
https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations
The work to merge locals tables for str/bytes/bytearray/array was done by
@jimmo. Because of this merging of locals the change in code size for this
commit is mostly negative:
bare-arm: +0 +0.000%
minimal x86: +29 +0.018%
unix x64: -792 -0.128% standard[incl -448(data)]
unix nanbox: -436 -0.078% nanbox[incl -448(data)]
stm32: -40 -0.010% PYBV10
cc3200: -32 -0.017%
esp8266: -28 -0.004% GENERIC
esp32: -72 -0.005% GENERIC[incl -200(data)]
mimxrt: -40 -0.011% TEENSY40
renesas-ra: -40 -0.006% RA6M2_EK
nrf: -16 -0.009% pca10040
rp2: -64 -0.013% PICO
samd: +148 +0.105% ADAFRUIT_ITSYBITSY_M4_EXPRESS
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.
Signed-off-by: Damien George <damien@micropython.org>
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory. Thus
some paths had to be adjusted.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
The reallocation trigger for unpacking star args with unknown length
did not take into account the number of fixed args remaining. So it was
possible that the unpacked iterators could take up exactly the memory
allocated then nothing would be left for fixed args after the star args.
This causes a segfault crash.
This is fixed by taking into account the remaining number of fixed args
in the check to decide whether to realloc yet or not.
Signed-off-by: David Lechner <david@pybricks.com>
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
On ports with more than one filesystem, the type will be wrong, for example
if using LFS but FAT enabled, then the type will be FAT. So it's not
possible to use these classes to identify a file object type.
Furthermore, constructing an io.FileIO currently crashes on FAT, and
make_new isn't supported on LFS.
And the io.TextIOWrapper class does not match CPython at all.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit simplifies mp_obj_get_complex_maybe() by first calling
mp_obj_get_float_maybe() to handle the cases corresponding to floats.
Only if that fails does it attempt to extra a full complex number.
This reduces code size and also means that mp_obj_get_complex_maybe() now
supports user-defined classes defining __float__; in particular this allows
user-defined classes to be used as arguments to cmath-module function.
Furthermore, complex_make_new() can now be simplified to directly call
mp_obj_get_complex(), instead of mp_obj_get_complex_maybe() followed by
mp_obj_get_float(). This also improves error messages from complex with
an invalid argument, it now raises "can't convert <type> to complex" rather
than "can't convert <type> to float".
Signed-off-by: Damien George <damien@micropython.org>
Add cert_reqs and cadata keyword-args to ssl.wrap_socket() and
ssl.CERT_NONE, ssl.CERT_OPTIONAL, ssl.CERT_REQUIRED constants to allow
certificate validation.
CPython doesn't accept cadata in ssl.wrap_socket(), but it does in
SSLContext.load_verify_locations(), so we use this name to at least match
the same name in load_verify_locations().
Add docs for these new arguments, as well as docs for the existing
server_hostname argument which is important for certificate validation.
Tests are added as well.
Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
The empty tuple is usually a constant object, but named tuples must be
allocated to allow modification. Added explicit allocation to fix this.
Also added a regression test to verify creating an empty named tuple works.
Fixes issue #7870.
Signed-off-by: Lars Haulin <lars.haulin@gmail.com>
For STM32L072 and similar, very low end targets.
The other perf_bench tests run out of memory, crash, or fail on
prerequisite features.
Signed-off-by: Angus Gratton <gus@projectgus.com>
Prior to this commit, complex("j") would return 0j, and complex("nanj")
would return nan+0j. This commit makes sure "j" is tested for after
parsing the number (nan, inf or a decimal), and also supports the case of
"j" on its own.
Signed-off-by: Damien George <damien@micropython.org>
This changes the btree implementation to use the buffer protocol for
reading key/values in all methods. `str` and `bytes` objects are not the
only bytes-like objects that could be used.
Documentation and tests are also updated.
Addresses issue #8748.
Signed-off-by: David Lechner <david@pybricks.com>
This new logic tracks when an unconditional jump/raise occurs in the
emitted code stream (bytecode or native machine code) and suppresses all
subsequent code, until a label is assigned. This eliminates a lot of
cases of dead code, with relatively simple logic.
This commit combined with the previous one (that removed the existing
dead-code finding logic) has the following code size change:
bare-arm: -16 -0.028%
minimal x86: -60 -0.036%
unix x64: -368 -0.070%
unix nanbox: -80 -0.017%
stm32: -204 -0.052% PYBV10
cc3200: +0 +0.000%
esp8266: -232 -0.033% GENERIC
esp32: -224 -0.015% GENERIC[incl -40(data)]
mimxrt: -192 -0.054% TEENSY40
renesas-ra: -200 -0.032% RA6M2_EK
nrf: +28 +0.015% pca10040
rp2: -256 -0.050% PICO
samd: -12 -0.009% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Signed-off-by: Damien George <damien@micropython.org>
The sys.tracebacklimit feature has changed semantics a bit from CPython 3.7
(in the way it modifies the output), so provide a .exp file for the test so
it doesn't rely on CPython.
Signed-off-by: Damien George <damien@micropython.org>
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table. The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py. But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.
This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table. mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.
Signed-off-by: Damien George <damien@micropython.org>
This works if your network is pre-configured in boot.py as an object called
"nic". Without this, multitests expects to access the WLAN/LAN class which
isn't always correct.
Signed-off-by: Andrew Leech <andrew@alelec.net>
This fixes the cases where the task being waited on finishes just before or
just after the wait_for itself is cancelled.
Fixes issue #8717.
Signed-off-by: Damien George <damien@micropython.org>
And make it so this test can run on any target.
LED and time testing has been removed from this test, that can now be
tested using: ./run-tests.py --via-mpy --emit native.
Signed-off-by: Damien George <damien@micropython.org>
If a port is not using internal error numbers, which match both lwIP and
Linux error numbers, ENTOCONN from standard libraries errno.h equals 128,
not 107.
This enables the new `-X realtime` runtime option when running tests on
macOS. This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers. This
makes tests that depend on the passage of time more likely to succeed.
CI tests that were disabled because of this are now enabled again.
Signed-off-by: David Lechner <david@pybricks.com>
The performance benchmark tests now support `--via-mpy` and `--emit native`
on remote targets. For example:
$ ./run-perfbench.py -p --via-mpy --emit native 100 100
Signed-off-by: Damien George <damien@micropython.org>
This adds support for the `--via-mpy` and `--emit native` options when
running tests on remote targets (via pyboard.py). It's now possible to do:
$ ./run-tests.py --target pyboard --via-mpy
$ ./run-tests.py --target pyboard --via-mpy --emit native
Signed-off-by: Damien George <damien@micropython.org>
Now that constant tuples are supported in the parser, eg (1, True, "str"),
it's a small step to allow anything that is a constant to be used with the
pattern:
from micropython import const
X = const(obj)
This commit makes the required changes to allow the following types of
constants:
from micropython import const
_INT = const(123)
_FLOAT = const(1.2)
_COMPLEX = const(3.4j)
_STR = const("str")
_BYTES = const(b"bytes")
_TUPLE = const((_INT, _STR, _BYTES))
_TUPLE2 = const((None, False, True, ..., (), _TUPLE))
Prior to this, only integers could be used in const(...).
Signed-off-by: Damien George <damien@micropython.org>
Because the test modifies the (now) bytearray object, and if it's a bytes
object it's not guaranteed that it can be modified, or that this constant
object isn't used elsewhere.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, all qstrs were required to be allocated (by calling
mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one). But
this is an unnecessary restriction, which is lifted by this commit.
Lifting the restriction simplifies the compiler because it can allocate
qstrs in later passes.
This also generates better code, because in some cases (eg when a variable
is closed over) the scope of an identifier is not known until a bit later
and then the identifier no longer needs its qstr allocated in the global
table.
Code size is reduced for all ports with this commit.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
Non-real-time systems like Windows, Linux and macOS do not have reliable
timing, so increase the sleep intervals to make these tests more likely to
pass.
Signed-off-by: Damien George <damien@micropython.org>
When in a class body or at the module level don't implicitly close over
variables that have been assigned to.
Fixes issue #8603.
Signed-off-by: Damien George <damien@micropython.org>
This fixes a bug where the gather is cancelled externally and then one of
its sub-tasks (that the gather was waiting on) finishes right between the
cancellation being queued and being executed.
Signed-off-by: Damien George <damien@micropython.org>
This double-raise test could fail when task[0] raises and stops the gather
before task[1] raises, then task[1] is left to raise later on and spoil the
test.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support to the parser so that tuples which contain only
constant elements (bool, int, str, bytes, etc) are immediately converted to
a tuple object. This makes it more efficient to use tuples containing
constant data because they no longer need to be created at runtime by the
bytecode (or native code).
Furthermore, with this improvement constant tuples that are part of frozen
code are now able to be stored fully in ROM (this will be implemented in
later commits).
Code size is increased by about 400 bytes on Cortex-M4 platforms.
See related issue #722.
Signed-off-by: Damien George <damien@micropython.org>
This follows the CPython change: https://bugs.python.org/issue21455
Socket listen backlog defaults to 2 if not given, based on most bare metal
targets not having many resources for a large backlog. On UNIX it defaults
to SOMAXCONN or 128, whichever is less.
There were two issues with the existing code:
1. "1 << i" is computed as a 32-bit number so would overflow when
executed on 64-bit machines (when mp_uint_t is 64-bit). This meant that
*args beyond 32 positions would not be handled correctly.
2. star_args must fit as a positive small int so that it is encoded
correctly in the emitted code. MP_SMALL_INT_BITS is too big because it
overflows a small int by 1 bit. MP_SMALL_INT_BITS - 1 does not work
because it produces a signed small int which is then sign extended when
extracted (even by mp_obj_get_int_truncated), and this sign extension
means that any position arg after *args is also treated as a star-arg.
So the maximum bit position is MP_SMALL_INT_BITS - 2. This means that
MP_OBJ_SMALL_INT_VALUE() can be used instead of
mp_obj_get_int_truncated() to get the value of star_args.
These issues are fixed by this commit, and a test added.
Signed-off-by: Damien George <damien@micropython.org>
This fixes code coverage for the case where a *arg without __len__ is
unpacked and uses exactly the amount of memory that was allocated for
kw args. This triggers the code branch where the memory for the kw args
gets reallocated since it was used already by the *arg unpacking.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow unpacking multiple
star args in a function or method call.
This is implemented by changing the emitted bytecodes so that both
positional args and star args are stored as positional args. A bitmap is
added to indicate if an argument at a given position is a positional
argument or a star arg.
In the generated code, this new bitmap takes the place of the old star arg.
It is stored as a small int, so this means only the first N arguments can
be star args where N is the number of bits in a small int.
The runtime is modified to interpret this new bytecode format while still
trying to perform as few memory reallocations as possible.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow multiple ** unpackings
when calling a function or method.
The compiler is modified to encode the argument as a None: obj key-value
pair (similar to how regular keyword arguments are encoded as str: obj
pairs). The extra object that was pushed on the stack to hold a single **
unpacking object is no longer used and is removed.
The runtime is modified to decode this new format.
Signed-off-by: David Lechner <david@pybricks.com>
The following fixes are made:
- cancelling a gather now cancels all sub-tasks of the gather (previously
it would only cancel the first)
- if any sub-task of a gather raises an exception then the gather finishes
(previously it would only finish if the first sub-task raised)
Fixes issues #5798, #7807, #7901.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
This allows you to list and explore connected USB devices. It
only stubs out the methods to communicate to endpoints. That will
come in a follow up once TinyUSB has it. (It's in progress.)
Related to #5986
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
Some versions of Python (for instance: the mingw-w64 version which can be
installed on MSYS2) do include a pty module and claim to be posix-like
(os.name == 'posix'), yet the select.select call used in run-tests.py hangs
forever. To be on the safe side just exclude anything which might be
running on windows.
Boosted performance, board.json metadata, more mimxrt, rp2, samd features
This release of MicroPython sees a boost to the overall performance of the
VM and runtime. This is achieved by the addition of an optional cache to
speed up general hash table lookups, as well as a fast path in the VM for
the LOAD_ATTR opcode on instance types. The new configuration options are
MICROPY_OPT_MAP_LOOKUP_CACHE and MICROPY_OPT_LOAD_ATTR_FAST_PATH. As part
of this improvement the MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE option has
been removed, which provided a similar map caching mechanism but with the
cache stored in the bytecode, which made it not useful on bare metal ports.
The new mechanism is measured to be at least as good as the old one,
applies to more map lookups, has a constant RAM overhead, and applies to
native code as well as bytecode.
These performance options are enabled on the esp32, mimxrt, rp2, stm32 and
unix ports. For esp32 and mimxrt some code is also moved to RAM to further
boost performance. On stm32, performance increases by about 20% for
benchmarks that are heavy on name lookups, like misc_pystone.py and
misc_raytrace.py. On esp32 performance can increase by 2-3x, and on mimxrt
it is up to 6x.
All boards in all ports now have a board.json metadata file, which is used
to automatically build firmware and generate a webpage for that board
(among other possibilities). Auto-build scripts have been added for this
purpose and they build all esp32, mimxrt, rp2, samd and stm32 boards. The
generated output is available at https://micropython.org/download.
Support for FROZEN_DIR and FROZEN_MPY_DIR has been deprecated for some time
and was finally removed in this release. Instead of these, FROZEN_MANIFEST
can be used. The io.resource_stream() function is also removed, replaced
by the pure Python version in micropython-lib.
The search order for importing frozen Python modules is now controlled by
the ".frozen" entry in sys.path. This string is added by default in the
second position in sys.path. User code should adjust sys.path depending on
the desired behaviour. Putting ".frozen" first in sys.path will speed up
importing frozen modules.
A bug in multiple precision integers with bitwise of -0 was fixed in commit
2c139bbf4e.
The platform module has been added to allow querying the compiler and
underlying SDK/HAL/libc version. This is enabled on esp32, mimxrt and
stm32 ports.
The mpremote tool now supports seek, flush, mkdir and rmdir on PC-mounted
filesystems. And a help command has been added.
The documentation has seen many additions and improvements thanks (for a
second time) to the Google Season of Docs project. The rp2 documentation
now includes a reference for PIO assembly instructions, a PIO quick
reference and a PIO tutorial. The random and stm modules have been
documented, along with sys.settrace, manifest.py files and mpremote. There
is also now more detail about the differences between MicroPython and
standard Python 3.5 and above.
The esp32 port sees support for ESP32-S3 SoCs, and new boards GENERIC_S3,
ESP32_S2_WROVER, LOLIN_S2_MINI, LOLIN_S2_PICO and UM_FEATHERS2NEO. The PWM
driver has been improved and now supports all PWM timers and channels, and
the duty_u16() and duty_ns() methods, and it keeps the duty constant when
changing frequency. The machine.bitstream() function has been improved to
use RMT, with an option to select the original bit-banging implementation.
The mimxrt port gained new hardware features: SDRAM and SD card support, as
well as network integration with a LAN driver. The machine.WDT class was
added along with the machine.reset_cause(), machine.soft_reset(),
machine.unique_id() add machine.bitstream() functions. DHT sensor support
was added, and f-strings were enabled.
The rp2 port now has support for networking, and bluetooth using NimBLE.
The Nina-W10 WiFi/BT driver is fully integrated and supported by the new
Arduino Nano RP2040 connect board. I2S protocol support is added along
with a machine.bitstream() driver and DHT sensor support. The PWM driver
had a bug fix with the accuracy of setting/getting the frequency, and the
duty value is now retained when changing the frequency.
On the samd port there is now support for the internal flash being a block
device, and for filesystems and the os module. Pin and LED classes have
been implemented. There are more time functions, more Python features
enabled, and the help() function is added. SEEED_WIO_TERMINAL and
SEEED_XIAO board definitions are now available.
The stm32 port now has support for F427, F479 and H7A3(Q)/H7B3(Q) MCUs, and
new board definitions for VCC_GND_H743VI, OLIMEX_H407, MIKROE_QUAIL,
GARATRONIC_PYBSTICK26_F411, STM32H73B3I_DK. A bug was fixed in the SPI
driver where a SPI transfer could fail if the CYW43 WiFi driver was also
active at the same time.
On the windows port the help() function has been enabled, and support for
build variants added, to match the unix port.
The zephyr port upgraded Zephyr to v2.7.0.
The change in code size since the previous release for various ports is
(absolute and percentage change in the text section):
bare-arm: -1520 -2.605%
minimal x86: -2256 -1.531%
unix x64: -457 -0.089%
unix nanbox: -925 -0.204%
stm32: +312 +0.079% PYBV10
cc3200: -176 -0.096%
esp8266: +532 +0.076% GENERIC
esp32: +27096 +1.820% GENERIC
nrf: -212 -0.121% pca10040
rp2: +9904 +2.051% PICO
samd: +35332 +33.969% ADAFRUIT_ITSYBITSY_M4_EXPRESS
The changes that dominate these numbers are:
- bare-arm, minimal: use of new MICROPY_CONFIG_ROM_LEVEL_MINIMUM option and
subsequent disabling of remaining optional features
- unix, cc3200, nrf: general code size reductions of the core
- stm32: performance improvements, addition of platform module
- esp8266: enabling f-strings
- esp32: use of -O2 instead of -Os
- rp2: machine.I2S and other new hardware features
- samd: filesystem support and other new hardware features
Thanks to everyone who contributed to this release: Alan Dragomirecký,
Alexey Shvetsov, Andrew Leech, Andrew Scheller, Antoine Aubert, Boris
Vinogradov, Chris Boudacoff, Chris Fiege, Christian Decker, Damien George,
Daniel Gorny, Dave Hylands, David Michieli, Emilie Feral, Frédéric Pierson,
gibbonsc, Henk Vergonet, iabdalkader, Ihor Nehrutsa, Jan Hrudka, Jan Staal,
jc_.kim, Jim Mussared, Jonathan Hogg, Laurens Valk, leo chung, Lorenzo
Cappelletti, Magnus von Wachenfeldt, Matt Trentini, Matt van de Werken,
Maureen Helm, Michael Bentley, Michael Buesch, Mike Causer, Mike Teachman,
Mike Wadsten, Ned Konz, NitiKaur, oli, patrick, Patrick Van Oosterwijck,
Peter Boin, Peter Hinch, Peter van der Burg, Philipp Ebensberger, Pooya
Moradi, retsyo, robert-hh, roland van straten, Scott Armitage, Sebastian
Wicki, Seon Rozenblum, Sergei Silnov, Simon Baatz, Stewart Bonnick, stijn,
Tobias Thyrrestrup, Tomas Vanek, YoungJoon Chun.
What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.
Main components
===============
all:
- remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE
- update Python formatting to latest Black version 21.12b0
- remove support for FROZEN_DIR and FROZEN_MPY_DIR
py core:
- parse: simplify parse nodes representing a list
- emitnative: ensure load_subscr does not clobber existing REG_RET
- mpconfig.h: define initial templates for "feature levels"
- vm: add a fast path for LOAD_ATTR on instance types
- map: add an optional cache of (map+index) to speed up map lookups
- builtinimport: forward all debug printing to MICROPY_DEBUG_PRINTER
- add wrapper macros so hot VM functions can go in fast code location
- runtime: fix crash when exc __new__ doesn't return an exc instance
- mpconfig.h: define the "extra" feature level
- mpconfig.h: revert MICROPY_REPL_INFO to disabled at all levels
- gc: add hook to run code during time consuming GC operations
- showbc: print unary-op string when dumping bytecode
- modsys: replace non-ASCII quote char with ASCII char
- runtime: allow types to use both .attr and .locals_dict
- lexer: support nested [] and {} characters within f-string params
- objfun.h: remove obsolete comments about entries in extra_args
- builtinimport: refactor module importing
- showbc: fix printing of raw bytecode header on nanbox builds
- modio: remove io.resource_stream function
- only search frozen modules when '.frozen' is found in sys.path
- mkrules.cmake: set frozen preprocessor defs early
- runtime: allow initialising sys.path/argv with defaults
- mpstate.h: only include sys.path/argv objects in state when enabled
- mpz: fix bugs with bitwise of -0 by ensuring all 0's are positive
- qstr: reset mpstate.qstr_last_chunk before raising an error
- modbuiltins: add additional macro for extending builtins
- mpconfig.h: define MICROPY_PY_USSL_FINALISER only if not defined
extmod:
- machine_i2c: make SoftI2C configurable via macro option
- machine_spi: make SoftSPI configurable via macro option
- modonewire: make _onewire module configurable via macro option
- machine_pwm: factor out machine.PWM bindings to common code
- move modnetwork and modusocket from stm32 to extmod
- modnetwork: add STA_IF and AP_IF constants
- modnetwork: add extended socket state
- modusocket: add read/write stream methods to socket object
- modnetwork: define network interfaces in port config files
- network_cyw43: make consistent use of STA and AP constants
- modnetwork: remove STM32 references
- modnetwork: remove modnetwork socket u_state member
- mpbthci.h: add mp_bluetooth_hci_uart_any prototype
- nimble: add nimble CMake fragment file
- add platform module
- moduplatform: improve implementation for PC ports
- vfs_posix_file: support MP_STREAM_POLL in vfs_posix_file_ioctl
- modbluetooth: add connection interval to gap_connect
- nimble: update to NimBLE v1.4
- nimble: remove workaround for OS_ENOMEM
- uasyncio: fix gather returning exceptions from a cancelled task
- uplatform: remove unused definitions
- uplatform: use generic custom platform string
- network_ninaw10: fix scan list order to match other NICs
- modbluetooth: support gap_connect(None) to cancel a connection
- modure: redirect regex debug printing to mp_printf
- network_ninaw10: fix config of AP mode
- network_ninaw10: disable active connections before connecting
- network_ninaw10: make NIC state persistent
- network_ninaw10: return -1 on timeout from recv/send
- network_ninaw10: make recv/recvfrom interchangeable
- moduplatform: detect xtensa arch
- modusocket: allow setting timeout on unbound sockets
- modusocket: initialise accepted socket state
- network_ninaw10: use socket timeout preset in modusocket
- modbluetooth: fix conditional compilation of ringbuf_put_uuid
- modbluetooth: put declaration of connect_cancel in correct place
shared:
- libc/string0: don't include string.h, and provide __memcpy_chk
- runtime/pyexec: cleanup EXEC_FLAG flag constants
drivers:
- ninaw10: add ublox Nina-W10 WiFi/BT module driver
- lsm6dsox: add LSM6DSOX driver and examples
- neopixel: avoid heap alloc in fill()
- ninaw10: fix BSSID byte order, and add null byte to ESSID
- ninaw10/nina_wifi_drv: fix DNS resolution
mpy-cross: no changes specific to this component/port
lib:
- mynewt-nimble: switch to the MicroPython fork of NimBLE
- asf4: point submodule to latest commit on circuitpython branch
- update pico-sdk to 1.3.0 and tinyusb to 0.12.0
- stm32lib: update library for L4 v1.17.0, new G4, WL, and MMC fixes
- stm32lib: update library for fix to F7 USB HS
Support components
==================
docs:
- library/os.rst: clarify littlefs requirements for block erase
- library/bluetooth.rst: update incorrect link to gatts_write
- make.bat: change Windows output dir from '_build' to 'build'
- library/machine.I2S.rst: specify that I2S.shift args are kw-only
- esp32: explain ESP32 PWM modes, timers, and channels
- rp2: add reference for PIO assembly instructions, and PIO tutorial
- library/random.rst: document the random module
- reference/mpremote.rst: add docs for mpremote
- reference/manifest.rst: add docs for manifest.py files
- library/stm.rst: document the stm module
- esp32/tutorial: add an example of peripheral control via regs
- rp2/general.rst: fix typo with missing spaces
- library/framebuf.rst: adjust dimensions in example
- library/rp2.rst: update function asm_pio_encode to add sideset_opt
- reference/filesystem.rst: add detail on how to use littlefs fuse
- rp2/quickref.rst: add section on PIO
- library/sys.rst: add docs for sys.settrace
- esp8266/tutorial: fix comments of FrameBuffer examples
- library/uasyncio.rst: detail exception behaviour in cancel/timeout
- library/machine.Timer.rst: document 'id' as positional-only arg
- library/machine.SPI.rst: add example SPI usage
- library/machine.Timer.rst: document `period` and `callback` args
- library/machine.Pin.rst: add Pin.ANALOG mode constant
- remove trailing spaces and convert tabs to spaces
- library/sys.rst: add note about '.frozen' as an entry in sys.path
- differences: document details of new PEPs/features in Python 3.5+
- update copyright year range to include 2022
- esp32: update RMT quickref example to match latest code
examples: no changes specific to this component/port
tests:
- perf_bench: use math.log instead of math.log2
- basics: add tests for type-checking subclassed exc instances
- micropython/const.py: add comment about required config for test
- cpydiff: clarify f-string diffs regarding concatenation
- basics/int_big_cmp.py: add more tests for big-int comparison
- extmod: skip uselect_poll_udp when poll() is not available
tools:
- autobuild: add auto build for GENERIC_C3_USB
- ci.sh: use IDF v4.4 as part of esp32 CI and build GENERIC_S3
- autobuild: add the MIMXRT1010_EVK board to autobuild
- ci.sh: use a specific ESP IDF v4.4 commit
- autobuild: add script to generate website board metadata
- dfu.py: make tool work with python3 when parsing DFU files
- autobuild: automatically build all mimxrt, rp2 and samd boards
- autobuild: automatically build all stm32 boards
- mpremote: implement seek and flush in ioctl method
- autobuild: automatically build all esp32 boards
- upip.py: support == to specify exact package version
- makemanifest.py: make str conversion compatible with Python 2
- makemanifest.py: merge make-frozen.py
- mpremote: add mkdir and rmdir to RemoteFS
- mpremote: add help command
- mpremote: add link to mpremote docs URL in help message
- upip.py: skip '.frozen' entry in sys.path for install path
- autobuild: build esp8266 OTA image with GENERIC_1M board
- ci.sh: upgrade Zephyr docker image to v0.21.0
- ci.sh: build zephyr nucleo_wb55rg to test zephyr bluetooth build
CI:
- workflows: use Python 3.8 for macos workflow
- workflows: add new workflow to build ports download metadata
The ports
=========
all ports:
- add board.json for all boards
- add images, features and urls to board.json
- add '.frozen' as the first entry in sys.path
- move '.frozen' to second entry in sys.path
bare-arm port:
- mpconfigport.h: use MICROPY_CONFIG_ROM_LEVEL_MINIMUM
- mpconfigport.h: disable remaining optional features
cc3200 port: no changes specific to this component/port
esp8266 port:
- boards/GENERIC: enable f-strings
- extract qstr from object when comparing keys in config()
- etshal.h: remove unneeded function declarations
- allow building a board to any dest directory
esp32 port:
- boards: add new FeatherS2-Neo board definition
- machine_timer: use tx_update member for IDF 4.4 and above
- add support for ESP32-S3 SoCs
- boards: add new GENERIC_S3 board definition
- machine_hw_spi: fix hardware SPI DMA channels for S2/S3
- boards: add board definition for ESP32-S2-WROVER module
- boards: add LOLIN_S2_MINI ESP32-S2 board
- machine_pwm: add support for all PWM timers and channels
- README: updated readme with req IDF vers for ESP32-S2, C3 and S3
- usb: add USB host connection detection for CDC serial output
- machine_pin: block out IO16 and IO17 when using SPIRAM on ESP32
- mpthreadport: fix TCB cleanup function so thread_mutex is ready
- main: add option for a board to hook code into startup sequence
- split out WLAN code from modnetwork.c to network_wlan.c
- enable optimisations and move code to iRAM to boost performance
- usb: improve speed of USB CDC output
- add specific deploy_s2.md instructions for esp32-s2
- boards/LOLIN_S2_MINI: add image to board.json
- boards: update board and deploy metadata for UM_xxx boards
- usb: further improve speed of USB CDC output
- boards/LOLIN_S2_PICO: add LOLIN_S2_PICO board definition files
- boards/ESP32_S2_WROVER: link to specific deploy_s2 instructions
- support building with latest IDF v5
- in machine_i2s, send null samples in underflow situations
- in machine_i2s, make object reference arrays root pointers
- add SDCard support for S3, and a GENERIC_S3_SPIRAM board
- boards/GENERIC_S3: enable BLE on ESP32 S3
- machine_pwm: implement duty_u16() and duty_ns() PWM methods
- extract qstr from object when comparing keys in config()
- machine_pin: make GPIO 26 usable for S2,S3 if SPIRAM not config'd
- machine_hw_spi: fix SPI default pins reordering on ESP32-S2/S3
- machine_hw_spi: set proper default SPI(id=1) pins on S2,S3 and C3
- machine_hw_spi: set proper default SPI(id=2) pins on S2 and S3
- boards: remove SPI pin defaults from GENERIC S2/S3 boards
- modnetwork: synchronize WiFi AUTH_xxx constants with IDF values
- machine_pwm: keep duty constant when changing frequency
- machine_bitstream: replace bit-bang code with RMT-based driver
- machine_i2s: add support for ESP-IDF 4.4
- machine_bitstream: fix signal duplication on output pins
- esp32: enable platform module with IDF version
- boards/GENERIC_D2WD: build with -Os optimisation
- esp32_rmt: install RMT driver on core 1
- machine_bitstream: reinstate bitstream bit-bang implementation
javascript port: no changes specific to this component/port
mimxrt port:
- sdcard: implement SDCard driver
- machine_bitstream: add bitstream function to machine module
- rework flash configuration
- sdram: add SDRAM support
- eth: add LAN support and integrate the network module
- modmachine: implement machine.WDT() and machine.reset_cause()
- boards: fix the D14/D15 pin assignment of MIMXRT1050/60/64_EVK
- hal: remove duplicate definitions from flexspi_hyper_flash.h
- dma_channel: fix the DMA channel management
- fix cycle counter for time.ticks_cpu() and machine.bitstream()
- add dht_readinto() to the mimxrt module, and freeze dht.py
- extend the help() message and README.md
- mpconfigport.h: enable f-strings
- modmachine: implement soft_reset() and unique_id() functions
- boards/make-pins.py: allow empty lines and comments in pins.csv
- optimize the runtime speed
- enable the platform module
- boards: add the Seeed ARCH MIX board
- boards: update the board.json files and add deploy_xx.md files
- fix mp_hal_quiet_timing_enter()/exit() so timer still runs
- support PWM using the FLEXPWM and QTMR modules
- define UART 0 on MIMXRT boards
- support selection of PHY type and address
- re-enable eth checksum creation by HW
- fix a tiny unnoticed bug in sdcard.c
- add a driver for the DP83848 PHY device
- refactor the reading of the machine id
- enable ticks_cpu at boot time for NDEBUG builds only
- use -Og instead of -O0 for DEBUG builds
- tidy up the board flash related files
- hal: allow readSampleClkSrc to be configured by a board
- enable MICROPY_PY_USSL_FINALISER
minimal port:
- mpconfigport.h: use MICROPY_CONFIG_ROM_LEVEL_MINIMUM
- Makefile: don't force a 32-bit build
- mpconfigport.h: disable features that are not needed
nrf port:
- Makefile: improve Black Magic Probe commands
- main: use VFS helper function to mount fs and chdir
pic16bit port: no changes specific to this component/port
powerpc port: no changes specific to this component/port
qemu-arm port: no changes specific to this component/port
rp2 port:
- mpconfigport.h: enable heapq module
- add support for bluetooth module using NimBLE
- add framework for networking
- mpconfigport.h: use the "extra" feature level
- enable optimisations (comp goto, map cache, fast attr)
- machine_i2s: add I2S protocol support
- add support for Nina-W10 WiFi/BT module
- boards: add support for Arduino Nano RP2040
- machine_bitstream: implement the machine.bitstream driver
- boards: add neopixel.py to manifest.py
- rp2_pio: support exec with sideset
- boards/PIMORONI_PICOLIPO_16MB: fix 16MB flash size
- boards: add PYBSTICK26 RP2040 board definition
- machine_uart: handle and clear UART RX timeout IRQ
- boards/ARDUINO_NANO_RP2040_CONNECT: set default I2C pins
- machine_pwm: fix PWM frequency setting
- machine_pwm: keep duty value when changing the frequency
- add support for DHT11 and DHT22 sensors
- CMakeLists.txt: allow a board to override PICO_BOARD
- boards/GARATRONIC_PYBSTICK26_RP2040: use correct pico-sdk board cfg
samd port:
- integrate latest asf4, add help, more time funcs and uPy features
- samd_soc: allow a board to configure the low-level MCU config
- add internal flash block device, filesystem and uos support
- add Pin and LED classes, and machine.unique_id
- boards/ADAFRUIT_FEATHER_M0_EXPRESS: update for flash and pins
- boards/ADAFRUIT_ITSYBITSY_M4_EXPRESS: update for flash and pins
- boards/MINISAM_M4: update for flash and pins
- boards/ADAFRUIT_TRINKET_M0: update for flash and pins
- boards/SAMD21_XPLAINED_PRO: update for flash and pins
- boards/SEEED_WIO_TERMINAL: add new board definition
- boards/SEEED_XIAO: add new board definition
- README.md: update README to reflect new features and boards
stm32 port:
- pin: enable GPIO clock of pin if it's constructed without init
- main: don't unconditionally enable GPIO A,B,C,D clocks
- boards/VCC_GND_H743VI: add board definition for VCC_GND_H743VI
- boards/OLIMEX_E407: add Ethernet RMII support
- boards/LEGO_HUB_NO6: remove user paths from cc2564 init file
- boards: remove trailing spaces, and add newline at end of file
- add basic support for STM32H750
- add support for H7A3(Q)/H7B3(Q), and STM32H73B3I_DK board defn
- suggest putting code in main.py not boot.py
- boards/make-pins.py: allow a CPU pin to be hidden
- boards/make-pins.py: allow empty lines and comments in pins.csv
- dma: add functions for external users of DMA to enable clock
- enable LOAD_ATTR fast path, and map lookup caching on >M0
- boards: add OLIMEX H407 board definition
- enable platform module
- extended flash filesystem space to 512K on H743 boards
- boards/NUCLEO_H743ZI: enable VfsLfs2 on NUCLEO_H743ZI(2) boards
- boards: add PF11-BOOT0 to stm32f091_af.csv
- machine_i2c: use hardware I2C for STM32H7
- sdram: enforce gcc opt, and use volatile and DSB in sdram_test
- usbd_cdc_interface: allow a board to hook into USBD CDC RX events
- mpbthciport: allow a board to hook BT HCI poll functions
- pendsv: allow a board to add entries for pendsv_schedule_dispatch
- boards: add images to board.json for Adafruit and VCC_GND boards
- uart: fix race conditions and clearing status in IRQ handler
- mpconfigport.h: use the "extra" feature level
- in machine_i2s, send null samples in underflow situations
- in machine_i2s, make object reference arrays root pointers
- led: support an extra 2 LEDs in board configuration
- boards/MIKROE_CLICKER2_STM32: add more detail to board.json
- boards: add new board MikroElektronika Quail, and F427 support
- main: run optional frozen module at boot
- sdio: don't explicitly disable DMA2 on deinit of SDIO
- dma: make DMA2_Stream3 exclusive to SDIO when CYW43 enabled
- boards: build NUCLEO_WB55 and STM32F769DISC without mboot enabled
- boards: add PYBSTICK26 F411 board definition
- boards/NADHAT_PYBF405: rename board to GARATRONIC_NADHAT_F405
- usb: use a table of allowed values to simplify usb_mode get/set
- boards/NUCLEO_WB55: update rfcore_firmwre for new WS
- flashbdev: support generic flash storage config via link symbols
- boards: convert F413,F439,H743,L4xx,WB55 to new flash FS config
- add support for F479 MCUs
- include HAL MMC code in F4 builds
- boards/make-pins.py: use cpu pins to define static alt-fun macros
- boards/NUCLEO_WB55: fix LED ordering
- boards/LEGO_HUB_NO6: set filesystem label as HUB_NO6
- boards: remove stray '+' characters at start of lines in ld files
- boards: remove unused MICROPY_HW_ENABLE_TIMER config
- boards: enable MICROPY_HW_ENABLE_SERVO on various boards
- update L4 code to build with latest stm32lib and L4 HAL 1.17.0
- main: call sdcard_init when only MICROPY_HW_ENABLE_MMCARD enabled
- sdcard: support 8-bit wide SDIO bus
- sdcard: add config option to force MM card capacity
- factoryreset: init vfs flags before calling pyb_flash_init_vfs
- qspi: fix typo in address comment
- boards/make-pins.py: generate empty ADC table if needed
- boards/OLIMEX_H407: fix typo in OLIMEX H407 board.json
- network_wiznet5k: fix build error with wiznet5k and lwip enabled
- enable MICROPY_PY_USSL_FINALISER
teensy port:
- switch to use manifest.py instead of FROZEN_DIR
unix port:
- enable LOAD_ATTR fast path, and map lookup caching
- modusocket: support MP_STREAM_POLL in unix socket_ioctl
- modos: add support for uos.urandom(n)
- coverage: change remaining printf to mp_printf
- Makefile: use -Og instead of -O0 for debug builds
windows port:
- README: remove unsupported Python instructions for Cygwin
- mpconfigport.h: enable help and help("modules")
- add support for build variants to windows port
- run tests via Makefile
- appveyor: build both standard and dev variants
- appveyor: build mpy-cross only once for mingw-w64
- msvc: run qstr preprocessing phase in parallel
zephyr port:
- mphalport.h: remove unused and unimplemented C-level pin API
- increase minimum CMake version to 3.20.0
- update include path to reboot.h
- get UART console device from devicetree instead of Kconfig
- use CONFIG_USB_DEVICE_STACK for conditional USB device support
- upgrade to Zephyr v2.7.0
- modbluetooth_zephyr: provide dummy connect_cancel function
A script will print "SKIP" if it wants to be skipped, so the test runner
must also use uppercase SKIP.
Signed-off-by: Damien George <damien@micropython.org>
The current test depends on a specific number and order of packets to pass,
which can't be reproduced every run due to the unreliable UDP protocol.
This patch adds simple packets sequencing, retransmits with timeouts, and a
packet loss threshold, to make the test more tolerant to UDP protocol
packet drops and reordering.
This allows encoding things (eg a Basic-Auth header for a request) without
slicing the \n from the string, which allocates additional memory.
Co-authored-by: David Lechner <david@lechnology.com>
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.
See CPython version of the issue at https://bugs.python.org/issue27772
Signed-off-by: Jeff Epler <jepler@gmail.com>
BLE still functions correctly even though these messages are sometimes
printed by the IDF. Ignoring them allows the multi_bluetooth tests to pass
on an esp32 board.
Signed-off-by: Damien George <damien@micropython.org>
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.
But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.
This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).
Signed-off-by: Damien George <damien@micropython.org>
This commit makes sure that the value zero is always encoded in an mpz_t as
neg=0 and len=0 (previously it was just len=0).
This invariant is needed for some of the bitwise operations that operate on
negative numbers, because they cannot handle -0. For example
(-((1<<100)-(1<<100)))|1 was being computed as -65535, instead of 1.
Fixes issue #8042.
Signed-off-by: Damien George <damien@micropython.org>
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
This combines some test vectors from the implementation & a NIST
standards document, plus the code from the docstring.
The test vectors were eyeball-verified.
Expected result of const.py will be matched only when MICROPY_COMP_CONST is
enabled. For easy understanding, added description at the first of the
test code.
F-strings, new machine.I2S class, ESP32-C3 support and LEGO_HUB_NO6 board
This release of MicroPython adds support for f-strings (PEP-498), with a
few limitations compared to normal Python. F-strings are essentially
syntactic sugar for "".format() and make formatting strings a lot more
convenient. Other improvements to the core runtime include pretty printing
OSError when it has two arguments (an errno code and a string), scheduling
of KeyboardInterrupt on the main thread, and support for a single argument
to the optimised form of StopIteration.
In the machine module a new I2S class has been added, with support for
esp32 and stm32 ports. This provides a consistent API for transmit and
receive of audio data in blocking, non-blocking and asyncio-based
operation. Also, the json module has support for the "separators" argument
in the dump and dumps functions, and framebuf now includes a way to blit
between frame buffers of different formats using a palette. A new,
portable machine.bitstream function is also added which can output a stream
of bits with configurable timing, and is used as the basis for driving
WS2812 LEDs in a common way across ports.
There has been some restructuring of the repository directory layout, with
all third-party code now in the lib/ directory. And a new top-level
directory shared/ has been added with first-party code that was previously
in lib/ moved there.
The docs have seen further improvement with enhancements and additions to
the rp2 parts, as well as a new quick reference for the zephyr port.
The terms master/slave have been replaced with controller/peripheral,
mainly relating to I2C and SPI usage. And u-module references have been
replaced with just the module name without the u-prefix to help clear up
the intended usage of modules in MicroPython.
For the esp8266 and esp32 ports, hidden networks are now included in WLAN
scan results. On the esp32 the RMT class is enhanced with idle_level and
write_pulses modes. There is initial support for ESP32-C3 chips with
GENERIC_C3 and GENERIC_C3_USB boards.
The javascript port has had its Makefile and garbage collector
implementation reworked so it compiles and runs with latest the Emscripten
using asyncify.
The mimxrt port sees the addition of hardware I2C and SPI support, as well
as some additional methods to the machine module. There is also support
for Hyperflash chips.
The nrf port now has full VFS storage support, enables source-line on
traceback, and has .mpy features consistent with other ports.
For the rp2 port there is now more configurability for boards, and more
boards added.
The stm32 port has a new LEGO_HUB_NO6 board definition with detailed
information how to get this LEGO Hub running stock MicroPython. There is
also now support to change the CPU frequency on STM32WB MCUs. And USBD_xxx
descriptor options have been renamed to MICROPY_HW_USB_xxx.
Thanks to everyone who contributed to this release: Amir Gonnen, Andrew
Scheller, Bryan Tong Minh, Chris Wilson, Damien George, Daniel Mizyrycki,
David Lechner, David P, Fernando, finefoot, Frank Pilhofer, Glenn Ruben
Bakke, iabdalkader, Jeff Epler, Jim Mussared, Jonathan Hogg, Josh Klar,
Josh Lloyd, Julia Hathaway, Krzysztof Adamski, Matúš Olekšák, Michael
Weiss, Michel Bouwmans, Mike Causer, Mike Teachman, Ned Konz, NitiKaur,
oclyke, Patrick Van Oosterwijck, Peter Hinch, Peter Züger, Philipp
Ebensberger, robert-hh, Roberto Colistete Jr, Sashkoiv, Seon Rozenblum,
Tobias Thyrrestrup, Tom McDermott, Will Sowerbutts, Yonatan Goldschmidt.
What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.
Main components
===============
all:
- fix signed shifts and NULL access errors from -fsanitize=undefined
- update to point to files in new shared/ directory
py core:
- mpstate: make exceptions thread-local
- mpstate: schedule KeyboardInterrupt on main thread
- mperrno: add MP_ECANCELED error code
- makeqstrdefs.py: don't include .h files explicitly in preprocessing
- mark unused arguments from bytecode decoding macros
- objexcept: pretty print OSError also when it has 2 arguments
- makeversionhdr: add --tags arg to git describe
- vm: simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode
- objexcept: make mp_obj_exception_get_value support subclassed excs
- support single argument to optimised MP_OBJ_STOP_ITERATION
- introduce and use mp_raise_type_arg helper
- modsys: optimise sys.exit for code size by using exception helpers
- objexcept: make mp_obj_new_exception_arg1 inline
- obj: fix formatting of comment for mp_obj_is_integer
- emitnative: reuse need_reg_all func in need_stack_settled
- emitnative: ensure stack settling is safe mid-branch
- runtime: fix bool unary op for subclasses of native types
- builtinimport: fix condition for including do_execute_raw_code()
- mkrules: automatically build mpy-cross if it doesn't exist
- implement partial PEP-498 (f-string) support
- lexer: clear fstring_args vstr on lexer free
- mkrules.mk: do submodule sync in "make submodules"
extmod:
- btstack: add missing call to mp_bluetooth_hci_uart_deinit
- btstack: check that BLE is active before performing operations
- uasyncio: get addr and bind server socket before creating task
- axtls-include: add axtls_os_port.h to customise axTLS
- update for move of crypto-algorithms, re1.5, uzlib to lib
- moduselect: conditionally compile select()
- nimble: fix leak in l2cap_send if send-while-stalled
- btstack/btstack.mk: use -Wno-implicit-fallthrough, not =0
- utime: always invoke mp_hal_delay_ms when >= to 0ms
- modbluetooth: clamp MTU values to 32->UINT16_MAX
- nimble: allow modbluetooth binding to hook "sent HCI packet"
- nimble: add "memory stalling" mechanism for l2cap_send
- uasyncio: in open_connection use address info in socket creation
- modujson: add support for dump/dumps separators keyword-argument
- modlwip: fix close and clean up of UDP and raw sockets
- modbluetooth: add send_update arg to gatts_write
- add machine.bitstream
- modframebuf: enable blit between different formats via a palette
lib:
- tinyusb: update to version 0.10.1
- pico-sdk: update to version 1.2.0
- utils/stdout_helpers: make mp_hal_stdout_tx_strn_cooked efficient
- axtls: switch to repo at micropython/axtls
- axtls: update to latest axtls 2.1.5 wih additional commits
- re1.5: move re1.5 code from extmod to lib
- uzlib: move uzlib code from extmod to lib
- crypto-algorithms: move crypto-algorithms code from extmod to lib
- update README's based on contents of these dirs
drivers:
- neopixel: add common machine.bitstream-based neopixel module
- neopixel: optimize fill() for speed
- neopixel: reduce code size of driver
- cyw43: fix cyw43_deinit so it can be called many times in a row
- cyw43: make wifi join fail if interface is not active
mpy-cross:
- disable stack check when building with Emscripten
Support components
==================
docs:
- library: document new esp32.RMT features and fix wait_done
- library: warn that ustruct doesn't handle spaces in format strings
- esp8266/tutorial: change flash mode from dio to dout
- replace master/slave with controller/peripheral in I2C and SPI
- rp2: enhance quickref and change image to Pico pinout
- rp2: update general section to give a brief technical overview
- library/utime.rst: clarify behaviour and precision of sleep ms/us
- library/uasyncio.rst: document stream readexactly() method
- library/machine.I2S.rst: fix use of sd pin in examples
- zephyr: add quick reference for the Zephyr port
- library/zephyr: add libraries specific to the Zephyr port
- templates: add unix and zephyr quickref links to top-index
- rename ufoo.rst to foo.rst
- replace ufoo with foo in all docs
- library/index.rst: clarify module naming and purpose
- library/builtins.rst: add module title
- library/network.rst: simplify socket import
- add docs for machine.bitstream and neopixel module
- library: fix usage of :term: for frozen module reference
- esp8266: use monospace for software tools
- reference: mention that slicing a memoryview causes allocation
examples: no changes specific to this component/port
tests:
- extmod: make uasyncio_heaplock test more deterministic
- cpydiff/modules_struct_whitespace_in_format: run black
- extmod/ujson: add tests for dump/dumps separators argument
- run-multitests.py: add broadcast and wait facility
- multi_bluetooth/ble_subscribe.py: add test for subscription
- extmod/vfs_fat_finaliser.py: ensure alloc at never-used GC blocks
- basics: split f-string debug printing to separate file with .exp
- pybnative: make while.py test run on boards without pyb.delay
tools:
- autobuild: add scripts to build release firmware
- remove obsolete build-stm-latest.sh script
- ci.sh: run apt-get update in ci_powerpc_setup
- makemanifest.py: allow passing flags to mpy-tool.py
- autobuild: add mimxrt port to build scripts for nightly builds
- pyboard.py: add cmd-line option to make soft reset configurable
- mpremote: swap order of PID and VID in connect-list output
- ci.sh: build unix dev variant as part of macOS CI
- ci.sh: build GENERIC_C3 board as part of esp32 CI
- autobuild: use separate IDF version to build newer esp32 SoCs
- autobuild: add FeatherS2 and TinyS2 to esp32 auto builds
- mpremote: add seek whence for mounted files
- mpremote: raise OSError on unsupported RemoteFile.seek
- autobuild: add the MIMXRT1050_EVKB board to the daily builds
- ci.sh: add mpy-cross build to nrf port
- codeformat.py: include ports/nrf/modules/nrf in code formatting
- gen-cpydiff.py: don't rename foo to ufoo in diff output
- autobuild: add auto build for Silicognition wESP32
- mpremote: fix connect-list in case VID/PID are None
- mpremote: add "devs" shortcut for "connect list"
- mpremote: remove support for pyb.USB_VCP in/out specialisation
- autobuild: don't use "-B" for make, it's already a fresh build
- pyboard.py: move --no-exclusive/--soft-reset out of mutex group
- pyboard.py: make --no-follow use same variable as --follow
- pyboard.py: add --exclusive to match --no-exclusive
- pyboard.py: make --no-soft-reset consistent with other args
- uncrustify: force 1 newline at end of file
- mpremote: bump version to 0.0.6
CI:
- workflows: add workflow to build and test javascript port
- workflows: switch from Coveralls to Codecov
- workflows: switch from lcov to gcov
- workflows: add workflow to build and test unix dev variant
The ports
=========
all ports:
- use common mp_hal_stdout_tx_strn_cooked instead of custom one
- update for move of crypto-algorithms, uzlib to lib
- rename USBD_VID/PID config macros to MICROPY_HW_USB_VID/PID
bare-arm port: no changes specific to this component/port
cc3200 port: no changes specific to this component/port
esp8266 port:
- add __len__ to NeoPixel driver to support iterating
- Makefile: add more libm files to build
- include hidden networks in WLAN.scan results
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
esp32 port:
- boards/sdkconfig.base: disable MEMPROT_FEATURE to alloc from IRAM
- add __len__ to NeoPixel driver to support iterating
- main: allow MICROPY_DIR to be overridden
- esp32_rmt: fix RMT looping in newer IDF versions
- esp32_rmt: enhance RMT with idle_level and write_pulses modes
- add new machine.I2S class for I2S protocol support
- machine_spi: calculate actual attained baudrate
- machine_hw_spi: use a 2 item SPI queue for long transfers
- machine_dac: add MICROPY_PY_MACHINE_DAC option, enable by default
- machine_i2s: add MICROPY_PY_MACHINE_I2S option, enable by default
- fix use of mp_int_t, size_t and uintptr_t
- add initial support for ESP32C3 SoCs
- boards/GENERIC_C3: add generic C3-based board
- modmachine: release the GIL in machine.idle()
- mphalport: always yield at least once in delay_ms
- machine_uart: add flow kw-arg to enable hardware flow control
- boards: add Silicognition wESP32 board configuration
- mpconfigport.h: enable reverse and inplace special methods
- include hidden networks in WLAN.scan results
- makeimg.py: get bootloader and partition offset from sdkconfig
- enable MICROPY_PY_FSTRINGS by default
- machine_hw_spi: release GIL during transfers
- machine_pin: make check for non-output pins respect chip variant
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
- boards: add GENERIC_C3_USB board with USB serial/JTAG support
javascript port:
- rework Makefile and GC so it works with latest Emscripten
- Makefile: suppress compiler errors from array bounds
- Makefile: change variable to EXPORTED_RUNTIME_METHODS
mimxrt port:
- move calc_weekday helper function to timeutils
- machine_spi: add the SPI class to the machine module
- moduos: seed the PRNG on boot using the TRNG
- boards: set vfs partition start to 1 MBbyte
- main: skip running main.py if boot.py failed
- main: extend the information returned by help()
- mimxrt_flash: remove commented-out code
- modmachine: add a few minor methods to the machine module
- machine_led: use mp_raise_msg_varg helper
- machine_i2c: add hardware-based machine.I2C to machine module
- add support for Hyperflash chips
- boards: add support for the MIMXRT1050_EVKB board
- machine_pin: implement ioctl for Pin
minimal port:
- Makefile: add support for building with user C modules
nrf port:
- modules: replace master/slave with controller/peripheral in SPI
- boards/common.ld: calculate unused flash region
- modules/nrf: add new nrf module with flash block device
- drivers: add support for using flash block device with SoftDevice
- mpconfigport.h: expose nrf module when MICROPY_PY_NRF is set
- README: update README.md to reflect internal file systems
- mpconfigport.h: tune FAT FS configuration
- Makefile: add _fs_size linker script override from make
- modules/uos: allow a board to configure MICROPY_VFS_FAT/LFS1/LFS2
- mpconfigport.h: enable MICROPY_PY_IO_FILEIO when an FS is enabled
- qstrdefsport.h: add entries for in-built FS mount points
- main: add auto mount and auto format hook for internal flash FS
- boards: enable needed features for FAT/LFS1/LFS2
- facilitate use of freeze manifest
- boards: set FROZEN_MANIFEST blank when SD present on nrf51 targets
- modules/scripts: add file system formatting script
- Makefile: set default manifest file for all targets
- mphalport: add dummy function for mp_hal_time_ns()
- boards: enable MICROPY_VFS_LFS2 for all target boards
- modules/uos: add ilistdir to uos module
- modules/nrf: add function to enable/disable DCDC
- enable source line on tracebacks
- set .mpy features consistent with documentation and other ports
pic16bit port: no changes specific to this component/port
powerpc port: no changes specific to this component/port
qemu-arm port: no changes specific to this component/port
rp2 port:
- use 0=Monday datetime convention in RTC
- machine_rtc: in RTC.datetime, compute weekday automatically
- CMakeLists.txt: suppress compiler errors for pico-sdk and tinyusb
- tusb_config.h: set CFG_TUD_CDC_EP_BUFSIZE to 256
- machine_uart: add hardware flow control support
- machine_uart: allow overriding default machine UART pins
- machine_i2c: allow boards to configure I2C pins using new macros
- machine_spi: allow boards to configure SPI pins using new macros
- machine_uart: fix poll ioctl to also check hardware FIFO
- machine_uart: fix read when FIFO has chars but ringbuf doesn't
- tusb_port: allow boards to configure USB VID and PID
- boards/ADAFRUIT_FEATHER_RP2040: configure custom VID/PID
- boards/ADAFRUIT_FEATHER_RP2040: configure I2C/SPI default pins
- boards/SPARKFUN_PROMICRO: configure UART/I2C/SPI default pins
- boards/SPARKFUN_THINGPLUS: configure I2C/SPI default pins
- boards: add Adafruit ItsyBitsy RP2040
- boards: add Adafruit QT Py RP2040
- boards: add Pimoroni Pico LiPo 4MB
- boards: add Pimoroni Pico LiPo 16MB
- boards: add Pimoroni Tiny 2040
- CMakeLists.txt: allow a board's cmake to set the manifest path
- enable MICROPY_PY_FSTRINGS by default
- Makefile: add "submodules" target, to match other ports
- rp2_flash: disable IRQs while calling flash_erase/program
- CMakeLists.txt: add option to enable double tap reset to bootrom
- mpconfigport.h: allow boards to add root pointers
samd port:
- add support for building with user C modules
stm32 port:
- softtimer: add soft_timer_reinsert() helper function
- mpbthciport: change from systick to soft-timer for BT scheduling
- provide a custom BTstack runloop that integrates with soft timer
- usb: make irq's default trigger enable all events
- boardctrl: skip running main.py if boot.py had an error
- sdio: fix undefined reference to DMA stream on H7
- dma: add DMAMUX configuration for H7 to fix dma_nohal_init
- main: call mp_deinit() at end of main
- adc: allow using ADC12 and ADC3 for H7
- adc: define the ADC instance used for internal channels
- adc: simplify and generalise how pin_adcX table is defined
- add new machine.I2S class for I2S protocol support
- boards/NUCLEO_F446RE: fix I2C1 pin assignment to match datasheet
- replace master/slave with controller/peripheral in I2C and SPI
- systick: always POLL_HOOK when delaying for milliseconds
- sdram: make SDRAM test cache aware, and optional failure with msg
- boards/NUCLEO_F446RE: enable CAN bus support
- boards: add support for SparkFun STM32 MicroMod Processor board
- uart: fix LPUART1 baudrate set/get
- uart: support low baudrates on LPUART1
- boards/STM32F429DISC: set correct UART2 pins and add UART3/6
- boards/NUCLEO_F439ZI: add board definition for NUCLEO_F439ZI
- boards/LEGO_HUB_NO6: add board definition for LEGO_HUB_NO6
- Makefile: update to only pull in used Bluetooth library
- README.md: update supported MCUs, and submodule and mboot use
- usbd_desc: rename USBD_xxx descriptor opts to MICROPY_HW_USB_xxx
- usbd_cdc_interface: rename USBD_CDC_xx opts to MICROPY_HW_USB_xx
- powerctrl: support changing frequency on WB MCUs
- boards/NUCLEO_H743ZI2: add modified version of NUCLEO_H743ZI
- mbedtls: fix compile warning about uninitialized val
- enable MICROPY_PY_FSTRINGS by default
- add implementation of machine.bitstream
- Makefile: allow GIT_SUBMODULES and LIBS to be extended
- stm32_it: support TIM17 IRQs on WB MCUs
- disable computed goto on constrained boards
- storage: make extended-block-device more configurable
- boards/LEGO_HUB_NO6: change SPI flash storage to use hardware SPI
- boards/LEGO_HUB_NO6: skip first 1MiB of SPI flash for storage
- boards/LEGO_HUB_NO6: add make commands to backup/restore firmware
teensy port: no changes specific to this component/port
unix port:
- modffi: add option to lock GC in callback, and cfun access
- Makefile: add back LIB_SRC_C to list of object files
- variants: enable help and help("modules") on standard and dev
- Makefile: disable error compression on arm-linux-gnueabi-gcc
windows port:
- Makefile: add .exe extension to executables name
- appveyor: update to VS 2017 and use Python 3.8 for build/test
zephyr port:
- machine_spi: add support for hardware SPI
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.
This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.
The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).
It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.
For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:
diff of scores (higher is better)
N=2000 M=2000 bccache -> attrmapcache diff diff% (error%)
bm_chaos.py 13742.56 -> 13905.67 : +163.11 = +1.187% (+/-3.75%)
bm_fannkuch.py 60.13 -> 61.34 : +1.21 = +2.012% (+/-2.11%)
bm_fft.py 113083.20 -> 114793.68 : +1710.48 = +1.513% (+/-1.57%)
bm_float.py 256552.80 -> 243908.29 : -12644.51 = -4.929% (+/-1.90%)
bm_hexiom.py 521.93 -> 625.41 : +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py 197544.25 -> 217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py 8072.98 -> 8198.75 : +125.77 = +1.558% (+/-3.22%)
misc_aes.py 17283.45 -> 16480.52 : -802.93 = -4.646% (+/-0.82%)
misc_mandel.py 99083.99 -> 128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py 83860.10 -> 82592.56 : -1267.54 = -1.511% (+/-2.27%)
misc_raytrace.py 21490.40 -> 22227.23 : +736.83 = +3.429% (+/-1.88%)
This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).
The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):
diff of scores (higher is better)
N=2000 M=2000 native -> nat-attrmapcache diff diff% (error%)
bm_chaos.py 14130.62 -> 15464.68 : +1334.06 = +9.441% (+/-7.11%)
bm_fannkuch.py 74.96 -> 76.16 : +1.20 = +1.601% (+/-1.80%)
bm_fft.py 166682.99 -> 168221.86 : +1538.87 = +0.923% (+/-4.20%)
bm_float.py 233415.23 -> 265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py 628.59 -> 734.17 : +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py 225418.44 -> 232926.45 : +7508.01 = +3.331% (+/-3.10%)
bm_pidigits.py 6322.00 -> 6379.52 : +57.52 = +0.910% (+/-5.62%)
misc_aes.py 20670.10 -> 27223.18 : +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py 138221.11 -> 152014.01 : +13792.90 = +9.979% (+/-2.46%)
misc_pystone.py 85032.14 -> 105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py 19800.01 -> 23350.73 : +3550.72 = +17.933% (+/-2.79%)
In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.
See #7680 for further discussion. And see also #7653 for a discussion
about simplifying mpy-cross options.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This feature {x=} was introduced in Python 3.8 so needs a separate .exp
file to run on earlier Python versions.
See https://bugs.python.org/issue36817
Signed-off-by: Damien George <damien@micropython.org>
This achieves a substantial performance improvement when rendering glyphs
to color displays, the benefit increasing proportional to the number of
pixels in the glyph.
This also tweaks the repr for unicode strings to only escape a few
utf-8 code points. This makes emoji show in os.listdir() for
example.
Also, enable exfat support on full builds.
Fixes#5146
Prevents the finaliser from being missed if there's a dangling reference
on the stack to one of the blocks for the files (that this test checks
that they get finalised).
See github.com/micropython/micropython/pull/7659#issuecomment-899479793
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This tests both sending indications/notifications from a server to
subscribed clients via gatts_write(...,send_update=True) and subscribing
from a client.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.
It is implemented in the lexer as a syntax translation to `str.format`:
f"{a}" --> "{}".format(a)
It also supports:
f"{a=}" --> "a={}".format(a)
This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.
There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
`SyntaxError: raw f-strings are not supported`.
- literal concatenation of f-strings with adjacent strings will fail
"{}" f"{a}" --> "{}{}".format(a) (str.format will incorrectly use
the braces from the non-f-string)
f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)
- PEP-498 requires the full parser to understand the interpolated
argument, however because this entirely runs in the lexer it cannot
resolve nested braces in expressions like
f"{'}'}"
- The !r, !s, and !a conversions are not supported.
Includes tests and cpydiffs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The correct callback-deregister functions must be called dependent on the
socket type, otherwise resources may not be freed correctly.
Signed-off-by: Damien George <damien@micropython.org>
Test instances can now use the following methods to synchronise their
execution:
multitest.broadcast("sync message")
multitest.wait("sync message")
Signed-off-by: Damien George <damien@micropython.org>
Previously a subclass of a type that didn't implement unary_op, or didn't
handle MP_UNARY_OP_BOOL, would raise TypeError on bool conversion.
Fixes#5677.
Replace "master" with "controller" and "slave" with "peripheral" in
comments, errors, and debug messages.
Add CONTROLLER and PERIPHERAL constants to pyb.SPI and pyb.I2C classes;
retain MASTER and SLAVE constants for backward compatiblity.
The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a
StopIteration() exception object, and means that heap memory does not need
to be allocated for the exception (in cases where it can be used). This
commit allows this optimised object to take an optional argument (before,
it could only have no argument).
The commit also adds some new tests to cover corner cases with
StopIteration and generators that previously did not work.
Signed-off-by: Damien George <damien@micropython.org>
This test snuck through without proper formatting and is causing CI for
other unrelated changes to fail.
Signed-off-by: David Lechner <david@pybricks.com>
Currently when using uasyncio.start_server() the socket configuration is
done inside a uasyncio.create_task() background function. If the address
and port are already in use however this throws an OSError which cannot be
cleanly caught behind the create_task().
This commit moves the getaddrinfo and socket binding to the start_server()
function, and only creates the task if that succeeds. This means that any
OSError from the initial socket configuration is propagated directly up the
call stack, compatible with CPython behaviour.
See #7444.
Signed-off-by: Damien George <damien@micropython.org>
2.5 can be represented correctly in object representation C, but 2.3 cannot
(it is slightly truncated).
Signed-off-by: Damien George <damien@micropython.org>
This commit fixes a problem with a race between cancellation of task A and
completion of task B, when A waits on B. If task B completes just before
task A is cancelled then the cancellation of A does not work. Instead,
the CancelledError meant to cancel A gets passed through to B (that's
expected behaviour) but B handles it as a "Task exception wasn't retrieved"
scenario, printing out such a message (this is because finished tasks point
their "coro" attribute to themselves to indicate they are done, and
implement the throw() method, but that method inadvertently catches the
CancelledError). The correct behaviour is for B to bounce that
CancelledError back out.
This bug is mainly seen when wait_for() is used, and in that context the
symptoms are:
- occurs when using wait_for(T, S), if the task T being waited on finishes
at exactly the same time as the wait-for timeout S expires
- task T will have run to completion
- the "Task exception wasn't retrieved message" is printed with
"<class 'CancelledError'>" as the error (ie no traceback)
- the wait_for(T, S) call never returns (it's never put back on the
uasyncio run queue) and all tasks waiting on this are blocked forever
from running
- uasyncio otherwise continues to function and other tasks continue to be
scheduled as normal
The fix here reworks the "waiting" attribute of Task to be called "state"
and uses it to indicate whether a task is: running and not awaited on,
running and awaited on, finished and not awaited on, or finished and
awaited on. This means the task does not need to point "coro" to itself to
indicate finished, and also allows removal of the throw() method.
A benefit of this is that "Task exception wasn't retrieved" messages can go
back to being able to print the name of the coroutine function.
Fixes issue #7386.
Signed-off-by: Damien George <damien@micropython.org>
With docs and a multi-test using TCP server/client.
This method is a MicroPython extension, although there is discussion of
adding it to CPython: https://bugs.python.org/issue41305
Signed-off-by: Mike Teachman <mike.teachman@gmail.com>
Take changes from https://github.com/micropython/micropython/pull/3694
(expected to be merged soon) as well as other accumulated stuff from
upstream that we want.
Leave our desired differences, including:
* silencing warnings in python3
* renaming the file descriptors returned by openpty()
* adding ulab tests
* Adding "." to the import path for skip_if
This speeds up `make test_full` and should also reduce the time in CI
a little bit.
The random module's getrandbits() method didn't give a proper error message
when calling it with a value that was outside of the range of 1-32, which
can lead to confusion using this function (which under CPython can accept
numbers larger than 32). Now instead of simply giving a ValueError it
gives an error message that states that the number of bits is constrained.
Also, since the random module's functions getrandbits() and randint()
differ from CPython, tests have been added to describe these differences.
For getrandbits the relevant documentation is shown and added to the docs.
The same is given for randint method so that the information is more easily
found.
Finally, since the int object lacks the bit_length() method there is a test
for that method also to include within the docs, showing the difference to
CPython.
If digest is called then the hash object is put in a "final" state and
calling update() or digest() again will raise a ValueError (instead of
silently producing the wrong result).
See issue #4119.
Signed-off-by: Damien George <damien@micropython.org>
MicroPython does not store any reference from a function object to the
module it was defined in, but there is a way to use function.__globals__ to
indirectly get the module.
See issue #7259.
Signed-off-by: Damien George <damien@micropython.org>
This significantly reduces the time taken to run the test suite (on the
unix port). Use `-j1` to disable this feature.
Signed-off-by: Jeff Epler <jepler@gmail.com>
Array equality is defined as each element being equal but to keep
code size down MicroPython implements a binary comparison. This
can only be used correctly for elements with the same binary layout
though so turn it into an NotImplementedError when comparing types
for which the binary comparison yielded incorrect results: types
with different sizes, and floating point numbers because nan != nan.
The RP2040 has 2 cores and supports running at most 2 Python threads (the
main one plus another), and will raise OSError if a thread cannot be
created because core1 is already in use. This commit adjusts some thread
tests to be robust against such OSError's. These tests now pass on rp2
boards.
Signed-off-by: Damien George <damien@micropython.org>
This commit makes gc_lock_depth have one counter per thread, instead of one
global counter. This makes threads properly independent with respect to
the GC, in particular threads can now independently lock the GC for
themselves without locking it for other threads. It also means a given
thread can run a hard IRQ without temporarily locking the GC for all other
threads and potentially making them have MemoryError exceptions at random
locations (this really only occurs on MCUs with multiple cores and no GIL,
eg on the rp2 port).
The commit also removes protection of the GC lock/unlock functions, which
is no longer needed when the counter is per thread (and this also fixes the
cas where a hard IRQ calling gc_lock() may stall waiting for the mutex).
It also puts the check for `gc_lock_depth > 0` outside the GC mutex in
gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from
waiting on a mutex if it does attempt to allocate heap memory (and putting
the check outside the GC mutex is now safe now that there is a
gc_lock_depth per thread).
Signed-off-by: Damien George <damien@micropython.org>
* modframebuf: _mp_framebuf_p_t is not "really" a protocol, but the
QSTR assignment caused problems when building as a dynamic module
* modure: str_index_to_ptr is not in the natmod API, disable URE match
spans when dynamic. mp_obj_len() is a bugfix, we should throw here
if the object is not string-like
* moduzlib: Correct paths to uzlib headers & sources. this relative
path (from moduzlib.c to the referenced file) works in all cases,
the other only worked from ports/PORTNAME.
* dynruntime: Handle 2-arg m_malloc, assert_native_inited, add a
micropythonish mp_arg_check_num_mp, fix mp_raise_msg to use dumb
strings, add mp_raise_arg1
* nativeglue: ad assert_native_inited
* translate: MP_ERROR_TEXT evaluates to its argument for DYNRUNTIME
* mpy-tool: A straggling magic number change
* mpy_ld: Have to renumber manually after dynruntime change
* import_mpy_native_gc.py: Update copy of features0 baked into this test
So a lock can be acquired on one Python thread and then released on
another. A test for this is added.
Signed-off-by: Damien George <damien@micropython.org>
uctypes.FLOAT32 has a special value representation and
uctypes_struct_scalar_size() should be used instead of GET_SCALAR_SIZE().
Signed-off-by: Damien George <damien@micropython.org>
This fixes a bug where double arguments on a 32-bit architecture would not
be passed correctly because they only had 4 bytes of storage (not 8). It
also fixes a compiler warning/error in return_ffi_value on certian
architectures: array subscript 'double[0]' is partly outside array bounds
of 'ffi_arg[1]' {aka 'long unsigned int[1]'}.
Fixes issue #7064.
Signed-off-by: Damien George <damien@micropython.org>
we changed this way back at fab634e3ee for Rosie Pi and the merge
mangled it. I think at this point it makes sense just to take upstream's
version again.
Doing "import <tab>" will now complete/list built-in modules.
Originally at adafruit#4548 and adafruit#4608
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
Anything beginning with "_" will now only be tab-completed if there is
already a partial match for such an entry. In other words, entering
foo.<tab> will no longer complete/list anything beginning with "_".
Originally at adafruit#1850
Signed-off-by: Kathryn Lingel <kathryn@lingel.net>
It's now possible to specify a device serial port using shorcuts like:
$ ./run-multitests.py -i pyb:a0 -i pyb:u1 multi_bluetooth/*.py
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the errno attribute to exceptions, so code can retrieve
errno codes from an OSError using exc.errno.
The implementation here simply lets `errno` (and the existing `value`)
attributes work on any exception instance (they both alias args[0]). This
is for efficiency and to keep code size down. The pros and cons of this
are:
Pros:
- more compatible with CPython, less difference to document and learn
- OSError().errno will correctly return None, whereas the current way of
doing it via OSError().args[0] will raise an IndexError
- it reduces code size on most bare-metal ports (because they already have
the errno qstr)
- for Python code that uses exc.errno the generated bytecode is 2 bytes
smaller and more efficient to execute (compared with exc.args[0]); so
bytecode loaded to RAM saves 2 bytes RAM for each use of this attribute,
and bytecode that is frozen saves 2 bytes flash/ROM for each use
- it's easier/shorter to type, and saves 2 bytes of space in .py files that
use it (for each use)
Cons:
- increases code size by 4-8 bytes on minimal ports that don't already have
the `errno` qstr
- all exceptions now have .errno and .value attributes (a cpydiff test is
added to address this)
See also #2407.
Signed-off-by: Damien George <damien@micropython.org>
Commit cb68a5741a broke automatic Python
feature detection when running tests, because some detection relied on a
crash of a feature script returning exactly b"CRASH".
This commit fixes this and improves the situation by testing for the lack
of a known pass result, rather than an exact failure result.
Signed-off-by: Damien George <damien@micropython.org>
Per CPython everything which comes after the command, module or file
argument is not an option for the interpreter itself. Hence the processing
of options should stop when encountering those, and the remainder be passed
as sys.argv. Note the latter was already the case for a module or file but
not for a command.
This fixes issues like 'micropython myfile.py -h' showing the help and
exiting instead of passing '-h' as sys.argv[1], likewise for
'-X <something>' being treated as a special option no matter where it
occurs on the command line.
The generated regex code is limited in the range of jumps and counts, and
this commit checks all cases which can overflow given the right kind of
input regex, and returns an error in such a case.
This change assumes that the results that overflow an int8_t do not
overflow a platform int.
Closes: #7078
Signed-off-by: Jeff Epler <jepler@gmail.com>
# Conflicts:
# extmod/re1.5/compilecode.c
The generated regex code is limited in the range of jumps and counts, and
this commit checks all cases which can overflow given the right kind of
input regex, and returns an error in such a case.
This change assumes that the results that overflow an int8_t do not
overflow a platform int.
Closes: #7078
Signed-off-by: Jeff Epler <jepler@gmail.com>
Jeff Epler
Dan Halbert
Jim Mussared
Angus Gratton
Andrew Leech
stijn
Takeo Takahashi
Mat Booth
Peter Hinch
Damien George
Scott Shawcroft
Dan Ellis
Ned Konz
Daniel Jour
Dan Ellis
David Lechner
Carlosgg
Lars Haulin
Angus Gratton
Andrew Leech
robert-hh
iabdalkader
Jon Bjarni Bjarnason
Christian Zietz
Mark
David Lechner
gamblor21
foamyguy
Jeff Epler
Christian Decker
microDev
jc_.kim
Scott Shawcroft
Alec Delaney
Kattni Rembor
Mike Wadsten
Peter Züger
David P
Tom McDermott
Dan Halbert
Lucian Copeland
Mike Teachman
Zoltán Vörös
Macarthur Inbody
Artyom Skrobov
Kathryn Lingel