When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally. Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.
If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed. In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.
Fixes issues #6568 and #8506.
In case the version from pypi is installed or some other version is
available in sys.path, prepend `$(TOP)/mpy-cross` to sys.path instead.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previous the build directory just used the board name, now make it use the
variant name too.
This shouldn't have any change because the existing directory should not
exist (all builds run by these scripts remove their build directory after
completion), but it makes debugging easier.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The metadata can be version, description, and license.
After executing a manifest, the top-level metadata can be queried, and also
each file output from the manifest will have the metadata of the
containing manifest.
Use the version metadata to "tag" files before freezing such that they have
__version__ available.
By default, don't include micropython-lib/unix-ffi in the search.
If unix_ffi=True is passed to require(), then include unix-ffi and make it
take precedence over the other locations (e.g. python-stdlib).
This does two things:
- Prevents non-unix builds from using unix-only packages.
- Allows the unix build to optionally use a more full-featured (e.g. ffi)
based package, even with the same name as one from e.g. stdlib.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
If an include path is a directory, then it implicitly grabs the manifest.py
file inside that directory. This simplifies most manifest.py files.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This splits the manifest file loading logic from makemanifest.py and
updates makemanifest.py to use it.
This will allow non-freezing uses of manifests, such as defining packages
and dependencies in micropython-lib.
Also adds additional methods to the manifest "API":
- require() - to get a package from micropython-lib.
- module() - to define a single-file module
- package() - to define a multi-file package
module() and package() should replace most uses of freeze() and can also
be also used in non-freezing scenarios.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Having two separate manifests is confusing. It's simpler to have the daily
builds use the same configuration as the stable, release builds.
Signed-off-by: Damien George <damien@micropython.org>
It has been about 8 years since support for this chip was added. Reasons
to remove it are:
- It is no longer easy to obtain this part.
- There are now many other options for WiFi.
- It's not a good use of developer time to maintain it.
Signed-off-by: Damien George <damien@micropython.org>
Rather than having the autobuild know about the particular variants, have
the mpconfigboard.mk describe them and make autobuild discover them
automatically.
Adds a "query-variants" target to stm32/Makefile to allow the set of
possible variants to be queried.
Removes pybv3 from the autobuild as this isn't use by the downloads page.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
- Add lib/wiznet5k into the 'make submodules' step.
- Split the stm32 builds for wiznet5k and cc3k.
- Run 'make .... clean' after making the wiznet5k build.
This allows a remote file to be edited locally by copying it over, running
the local editor, then copying it back.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.
Signed-off-by: Damien George <damien@micropython.org>
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory. Thus
some paths had to be adjusted.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
The CI scripts were using a PPA to get a backported version of uncrustify
on Ubuntu 20.04. However, this causes CI to intermittently fail due to
connection issues to launchpad.net or the key server.
Ubuntu 22.04 has a newer version of uncrustify removing the need for the
PPA. Ubuntu 22.04 is now in beta on GitHub actions, so it can be used.
Signed-off-by: David Lechner <david@pybricks.com>
Updates the Zephyr port build instructions and CI to use the latest Zephyr
release tag.
Tested on frdm_k64f.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.
Signed-off-by: Damien George <damien@micropython.org>
Frozen identifiers now include their full name hierarchy, eg their class
name. This makes it easier to understand the generated code.
Signed-off-by: Damien George <damien@micropython.org>
Now that the native qstr link table is gone, merging a native .mpy file
with a bytecode .mpy file is not as simple as concatenating the .mpy data.
The qstr_table and obj_table tables from all merged .mpy files must now be
joined together, because they are global to the .mpy file (and hence global
to the merged .mpy file). This means the bytecode needs to be be decoded,
qstr_table and obj_table indices updated to point to the correct entries in
the new tables, and then the bytecode re-encoded.
This commit makes this change to the merging feature in mpy-tool.py. This
can now merge an arbitrary number of bytecode .mpy files, and up to one
native .mpy file.
Signed-off-by: Damien George <damien@micropython.org>
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.
Signed-off-by: Damien George <damien@micropython.org>
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table. The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py. But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.
This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table. mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.
Signed-off-by: Damien George <damien@micropython.org>
This enables the new `-X realtime` runtime option when running tests on
macOS. This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers. This
makes tests that depend on the passage of time more likely to succeed.
CI tests that were disabled because of this are now enabled again.
Signed-off-by: David Lechner <david@pybricks.com>
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms. More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.
Signed-off-by: Damien George <damien@micropython.org>
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory. This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise. Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was
then placed in a bytes object, pointed to by the module's constant table.
This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons
for doing this are:
1. It decouples the native emitter from runtime requirements, the emitted
code no longer needs to know if the system it runs on can/can't read
byte-wise from executable memory.
2. It makes all ports have the same emitter behaviour, there is no longer
the N_PRELUDE_AS_BYTES_OBJ option.
3. The module's constant table is now used only for actual constants in the
Python code. This allows further optimisations to be done with the
constants (eg constant deduplication).
Code size change for those ports that enable the native emitter:
unix x64: +80 +0.015%
stm32: +24 +0.004% PYBV10
esp8266: +88 +0.013% GENERIC
esp32: -20 -0.002% GENERIC[incl -112(data)]
rp2: +32 +0.005% PICO
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
This will add a space after a comma if it doesn't have one, but will allow
more than one space if the spaces are already there.
Signed-off-by: Damien George <damien@micropython.org>
This adds a rule to cover all ports/**/*.[ch] file to the code formatting
list. Explicit exclusions are also added for code in ports/ which is third
party, or which requires a lot of reformatting.
Signed-off-by: Damien George <damien@micropython.org>
To keep the separate parts of the code that use these values in sync. And
make it easier to add new object types.
Signed-off-by: Damien George <damien@micropython.org>
Specifying the option `--unsafe-links` (or `-l`) to `mpremote mount` will
allow symlinks to be followed in the local directory that point outside of
the base directory path.
For the unsafe case the `path_check()` method of `PyboardCommand` still
checks for a common path but without expanding symlinks. While this check
is currently redundant, it makes the purpose of the method clearer for
possible future uses or extensions.
With the existing code problems can occur with remounting, the "if t -
t_last_activity > QUIET_TIMEOUT:" check can be triggered early before the
REPL string comes through, meaning that the remount doesn't happen.
On certain boards the "MPY: soft reboot" line comes through immediately
(getting the routine past initial timeout) but then there's a slightly
longer delay while the board restarts before it prints out the startup
header and the REPL prompt.
This commit adds some extra pattern monitoring during the timeout loop to
track the state if a soft restart is actually started.
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument. These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
Since cpydiff is code used as documentation, there are cases where we may
want to use Black's `fmt: on/off/skip` comments to avoid automatic
formatting. However, we don't want these comments to be distracting in the
generated documentation.
This rewrites the code to omit these comments when generating the docs.
Signed-off-by: David Lechner <david@pybricks.com>
Tested on PYBV10 and PYBD_SF6, with MBOOT_FSLOAD enabled and programming
new firmware from a .dfu.gz file stored on the SD card.
Signed-off-by: Damien George <damien@micropython.org>
That caused the compile of frozen_content.c to fail if characters like
backslash were in a short string. Thanks to @hippy for identifying the
spot to change.
This makes the auto soft-reset behaviour of mpremote more logical, and now
configurable via these new commands.
Signed-off-by: Damien George <damien@micropython.org>
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
This changes the git commit message line length check to ignore lines that
contain URLs, since these cannot be wrapped without breaking tools that
detect URLs and create a link.
Signed-off-by: David Lechner <david@pybricks.com>
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.
No functional change.
The size reduction depends on the number of qstrs in the build. The change
this commit brings is:
bare-arm: -4 -0.007%
minimal x86: +150 +0.092% [incl +48(data)]
unix x64: -608 -0.118%
unix nanbox: -572 -0.126% [incl +32(data)]
stm32: -1392 -0.352% PYBV10
cc3200: -448 -0.244%
esp8266: -1208 -0.173% GENERIC
esp32: -1028 -0.068% GENERIC[incl -1020(data)]
nrf: -440 -0.252% pca10040
rp2: -1072 -0.217% PICO
samd: -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.
Originally at adafruit#4583
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
Changes are:
- decision to remount local filesystem on remote device is made only if
"MPY: soft reboot" is seen in the output after sending a ctrl-D
- a nice message is printed to the user when the remount occurs
- soft reset during raw REPL is now handled correctly
Fixes issue #7731.
Signed-off-by: Damien George <damien@micropython.org>
A backslash in the directory name will end up being passed through to the
device and becoming a backslash in a filename, rather than being
interpreted as directories. This makes "cp -r" problematic on Windows.
Changing to simply "/",join() fixes this.
As a prerequisite to upgrading to Zephyr v2.7.0, upgrade CI to use
Zephyr docker image v0.21.0. In particular, this is needed to pick up a
newer CMake version because Zephyr v2.7.0 increased the minimum CMake
version required to 3.20.0.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
This allows the remote MicroPython instance to create and delete
directories from the mounted host filesystem in addition to the already
existing functionality of reading, creating, and modifying files.
Signed-off-by: Michael Bentley <mikebentley15@gmail.com>
This changes makemanifest.py & mpy-tool.py to merge string and mpy names
into the same list (now mp_frozen_names).
The various paths for loading a frozen module (mp_find_frozen_module) and
checking existence of a frozen module (mp_frozen_stat) use a common
function that searches this list.
In addition, the frozen lookup will now only take place if the path starts
with ".frozen", which needs to be added to sys.path.
This fixes issues #1804, #2322, #3509, #6419.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Takes the functionality from tools/make-frozen.py, adds support for
multiple frozen directories, and moves it to tools/makemanifest.py.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Any board with a board.json file will be built. ESP32-based boards will be
built using the IDF at $IDF_PATH_V42, all other MCU variants (S2, S3, C3)
will be built using the IDF at $IDF_PATH_V44.
Signed-off-by: Damien George <damien@micropython.org>
Any board with a board.json file will be built. Additional variants for
certain pyboards will also be built by the explicit build-stm32-extra.sh
script. Both .dfu and .hex files will be made available.
Also build boards in a sorted order, and don't stop building if a single
board fails.
Signed-off-by: Damien George <damien@micropython.org>
This is to make the builds for all nucleo/discovery boards uniform, so they
can be treated the same by the auto build scripts.
The CI script is updated to explicitly enable mboot and packing, to test
these features.
Signed-off-by: Damien George <damien@micropython.org>
There is no release of IDF v4.4 yet but master is now on v5.0-dev so a
specific commit must be chosen to stick to v4.4.
Signed-off-by: Damien George <damien@micropython.org>
IDF v4.4 does not have an official release so for now use the latest
master. Also remove building GENERIC with no options (all the other boards
are no-option builds), to keep CI time reasonable.
Signed-off-by: Damien George <damien@micropython.org>
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.
This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.
The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).
It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.
For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:
diff of scores (higher is better)
N=2000 M=2000 bccache -> attrmapcache diff diff% (error%)
bm_chaos.py 13742.56 -> 13905.67 : +163.11 = +1.187% (+/-3.75%)
bm_fannkuch.py 60.13 -> 61.34 : +1.21 = +2.012% (+/-2.11%)
bm_fft.py 113083.20 -> 114793.68 : +1710.48 = +1.513% (+/-1.57%)
bm_float.py 256552.80 -> 243908.29 : -12644.51 = -4.929% (+/-1.90%)
bm_hexiom.py 521.93 -> 625.41 : +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py 197544.25 -> 217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py 8072.98 -> 8198.75 : +125.77 = +1.558% (+/-3.22%)
misc_aes.py 17283.45 -> 16480.52 : -802.93 = -4.646% (+/-0.82%)
misc_mandel.py 99083.99 -> 128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py 83860.10 -> 82592.56 : -1267.54 = -1.511% (+/-2.27%)
misc_raytrace.py 21490.40 -> 22227.23 : +736.83 = +3.429% (+/-1.88%)
This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).
The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):
diff of scores (higher is better)
N=2000 M=2000 native -> nat-attrmapcache diff diff% (error%)
bm_chaos.py 14130.62 -> 15464.68 : +1334.06 = +9.441% (+/-7.11%)
bm_fannkuch.py 74.96 -> 76.16 : +1.20 = +1.601% (+/-1.80%)
bm_fft.py 166682.99 -> 168221.86 : +1538.87 = +0.923% (+/-4.20%)
bm_float.py 233415.23 -> 265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py 628.59 -> 734.17 : +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py 225418.44 -> 232926.45 : +7508.01 = +3.331% (+/-3.10%)
bm_pidigits.py 6322.00 -> 6379.52 : +57.52 = +0.910% (+/-5.62%)
misc_aes.py 20670.10 -> 27223.18 : +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py 138221.11 -> 152014.01 : +13792.90 = +9.979% (+/-2.46%)
misc_pystone.py 85032.14 -> 105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py 19800.01 -> 23350.73 : +3550.72 = +17.933% (+/-2.79%)
In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.
See #7680 for further discussion. And see also #7653 for a discussion
about simplifying mpy-cross options.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
- Moves definition of BOARD_FLASH_SIZE and other header files related to
flash configuration into the Makefile.
- Adds board specific clock_config.h.
- Adds board.h, pin_mux.h, and peripherals.h as they are
required by NXP MCU SDK in order to use our own clock_config.h.
- Renames board specific FlexSPI configuration files.
- Updates flash frequency of MIMXRT1020_EVK
- Creates separated flash_config files for QSPI NOR and
QSPI Hyper flash.
- Unifies VFS start address to be @ 1M for 1010 and 1020 boards.
- Unifies 1050EVK boards
- Adds support to both NOR and HyperFlash on boards with
both capabilities.
- Adds automatic FlexRAM initialization to start-up code based on
linker script and NXP HAL.
- Applies code formatting to all files in mimxrt port.
With this change the flash configuration is restructured and
organized. This simplifies the configuration process and
provides a better overview of each board's settings. With the integration
of clock_config.h, board.h, pin_mux.h, and peripherals.h we gain better
control of the settings and clock configurations. Furthermore the
implementation of an explicit FlexRAM setup improves the system
performance and allows for performance tuning.
Signed-off-by: Philipp Ebensberger
To keep things neat and tidy, we ensure that each file has 1 and only 1
newline at the end of each file.
Signed-off-by: David Lechner <david@pybricks.com>
This feature {x=} was introduced in Python 3.8 so needs a separate .exp
file to run on earlier Python versions.
See https://bugs.python.org/issue36817
Signed-off-by: Damien George <damien@micropython.org>
This makes it work like --no-follow and --no-exclusive using a mutex group
and dest. Although the current implementation with BooleanOptionAction is
neater it requires Python 3.9, so don't use this feature.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The --no-exclusive flag was accidentally added to the mutex group in
178198a01d.
The --soft-reset flag was accidentally added to the mutex group in
41adf17830.
These flags can be specified independently to --[no-]follow so should not
be in that mutex group.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
And using "-B" means mpy-cross is forcefully rebuilt, sometimes with
invalid CFLAGS_EXTRA options which makes the auto-build fail.
Signed-off-by: Damien George <damien@micropython.org>
The sys.stdin.buffer and sys.stdout.buffer streams work just as well (and
are just as fast) as pyb.USB_VCP on stm32 devices, so there's no need to
have the USB_VCP specialisation code, which just adds complexity.
Also, on stm32 devices with both USB and UART (or other serial interface),
if something other than the USB_VCP port is used for the serial connection
then mpremote mount will not work because it will default to reading and
writing on USB_VCP instead of the other connected serial stream.
As part of this simplification, support for a second port as input is
removed (this feature was never exposed to the user).
Signed-off-by: Damien George <damien@micropython.org>
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.
It is implemented in the lexer as a syntax translation to `str.format`:
f"{a}" --> "{}".format(a)
It also supports:
f"{a=}" --> "a={}".format(a)
This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.
There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
`SyntaxError: raw f-strings are not supported`.
- literal concatenation of f-strings with adjacent strings will fail
"{}" f"{a}" --> "{}{}".format(a) (str.format will incorrectly use
the braces from the non-f-string)
f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)
- PEP-498 requires the full parser to understand the interpolated
argument, however because this entirely runs in the lexer it cannot
resolve nested braces in expressions like
f"{'}'}"
- The !r, !s, and !a conversions are not supported.
Includes tests and cpydiffs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
