The recent rework of bytecode made all constants global with respect to the
module (previously, each function had its own constant table). That means
the constant table for a module is shared among all functions/methods/etc
within the module.
This commit add support to the compiler to de-duplicate constants in this
module constant table. So if a constant is used more than once -- eg 1.0
or (None, None) -- then the same object is reused for all instances.
For example, if there is code like `print(1.0, 1.0)` then the parser will
create two independent constants 1.0 and 1.0. The compiler will then (with
this commit) notice they are the same and only put one of them in the
constant table. The bytecode will then reuse that constant twice in the
print expression. That allows the second 1.0 to be reclaimed by the GC,
also means the constant table has one less entry so saves a word.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, all qstrs were required to be allocated (by calling
mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one). But
this is an unnecessary restriction, which is lifted by this commit.
Lifting the restriction simplifies the compiler because it can allocate
qstrs in later passes.
This also generates better code, because in some cases (eg when a variable
is closed over) the scope of an identifier is not known until a bit later
and then the identifier no longer needs its qstr allocated in the global
table.
Code size is reduced for all ports with this commit.
Signed-off-by: Damien George <damien@micropython.org>
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory. This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise. Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was
then placed in a bytes object, pointed to by the module's constant table.
This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons
for doing this are:
1. It decouples the native emitter from runtime requirements, the emitted
code no longer needs to know if the system it runs on can/can't read
byte-wise from executable memory.
2. It makes all ports have the same emitter behaviour, there is no longer
the N_PRELUDE_AS_BYTES_OBJ option.
3. The module's constant table is now used only for actual constants in the
Python code. This allows further optimisations to be done with the
constants (eg constant deduplication).
Code size change for those ports that enable the native emitter:
unix x64: +80 +0.015%
stm32: +24 +0.004% PYBV10
esp8266: +88 +0.013% GENERIC
esp32: -20 -0.002% GENERIC[incl -112(data)]
rp2: +32 +0.005% PICO
Signed-off-by: Damien George <damien@micropython.org>
mpy-cross will now generate native code based on the size of
mp_code_state_native_t, and the runtime will use this struct to calculate
the offset of the .state field. This makes native code generation and
execution (which rely on this struct) independent to the settings
MICROPY_STACKLESS and MICROPY_PY_SYS_SETTRACE, both of which change the
size of the mp_code_state_t struct.
Fixes issue #5059.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
This replaces occurences of
foo_t *foo = m_new_obj(foo_t);
foo->base.type = &foo_type;
with
foo_t *foo = mp_obj_malloc(foo_t, &foo_type);
Excludes any places where base is a sub-field or when new0/memset is used.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is to replace the following:
mp_foo_obj_t *self = m_new_obj(mp_foo_obj_t);
self->base.type = &mp_type_foo;
with:
mp_foo_obj_t *self = mp_obj_malloc(mp_foo_obj_t, &mp_type_foo);
Calling the function is less code than inlining setting the type
everywhere, adds up to ~100 bytes on PYBV11.
It also helps to avoid an easy mistake of forgetting to set the type.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When in a class body or at the module level don't implicitly close over
variables that have been assigned to.
Fixes issue #8603.
Signed-off-by: Damien George <damien@micropython.org>
This contains a string useful for identifying the underlying machine. This
string is kept consistent with the second part of the REPL banner via the
new config option MICROPY_BANNER_MACHINE.
This makes os.uname() more or less redundant, as all the information in
os.uname() is now available in the sys module.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the git hash and build date to sys.version. This is
allowed according to CPython docs, and is what PyPy does. The docs state:
A string containing the version number of the Python interpreter plus
additional information on the build number and compiler used.
Eg on CPython:
Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]'
and PyPy:
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
[PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.version
'2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ...
With this commit on MicroPython we now have:
MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v...
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import sys
>>> sys.version
'3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28'
Note that the start of the banner is the same as the end of sys.version.
This helps to keep code size under control because the string can be reused
by the compiler.
Signed-off-by: Damien George <damien@micropython.org>
Entering tab at the REPL will now make it insert an indent (4 spaces) in
the following cases:
- after any whitespace on a line
- at the start of a line that is not the first line
This changes the existing behaviour where a tab would insert an indent only
if there were no matches in the auto-complete search, and it was the start
of the line. This means, if there were any symbols in the global
namespace, tab could never be used to indent.
Note that entering tab at the start of the first line will still do
auto-completion, but will now do nothing if there are no symbols in the
global namespace, which is more consistent than before.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support to the parser so that tuples which contain only
constant elements (bool, int, str, bytes, etc) are immediately converted to
a tuple object. This makes it more efficient to use tuples containing
constant data because they no longer need to be created at runtime by the
bytecode (or native code).
Furthermore, with this improvement constant tuples that are part of frozen
code are now able to be stored fully in ROM (this will be implemented in
later commits).
Code size is increased by about 400 bytes on Cortex-M4 platforms.
See related issue #722.
Signed-off-by: Damien George <damien@micropython.org>
To keep the separate parts of the code that use these values in sync. And
make it easier to add new object types.
Signed-off-by: Damien George <damien@micropython.org>
This reverts commit 7e8222ae06.
The prelude data must exist somewhere in the native code so load_raw_code
and mpy-tool.py can access and parse it.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_SCHEDULER_STATIC_NODES is enabled then C code can declare a
static mp_sched_node_t and schedule a callback using
mp_sched_schedule_node(). In contrast to using mp_sched_schedule(), the
node version will have at most one pending callback outstanding, and will
always be able to schedule if there is nothing already scheduled on this
node. This guarantees that the the callback will be called exactly once
after it is scheduled.
Signed-off-by: Damien George <damien@micropython.org>
The Thumb instruction set has special 16 bit encodings for PUSH involving
LR and POP involving PC, which are commonly used in nested functions.
Using this encoding is particularly important for ARMv6-M, where the more
general 32 bit encoding of PUSH and POP is unavailable.
- The classification of source files in makeqstrdefs.py has been moved into
functions to consolidate the logic for that classification into a single
place.
- Classification of source files (into C or C++ or "other" files) is based
on the filename extension.
- For C++ there are many more common filename extensions than just ".cpp";
see "Options Controlling the Kind of Output" in man gcc for example. All
common extensions for C++ source files which need preprocessing have been
added.
The values are always real objects, only the key can be MP_OBJ_NULL to
indicate a **kwargs entry.
Signed-off-by: Damien George <damien@micropython.org>
There were two issues with the existing code:
1. "1 << i" is computed as a 32-bit number so would overflow when
executed on 64-bit machines (when mp_uint_t is 64-bit). This meant that
*args beyond 32 positions would not be handled correctly.
2. star_args must fit as a positive small int so that it is encoded
correctly in the emitted code. MP_SMALL_INT_BITS is too big because it
overflows a small int by 1 bit. MP_SMALL_INT_BITS - 1 does not work
because it produces a signed small int which is then sign extended when
extracted (even by mp_obj_get_int_truncated), and this sign extension
means that any position arg after *args is also treated as a star-arg.
So the maximum bit position is MP_SMALL_INT_BITS - 2. This means that
MP_OBJ_SMALL_INT_VALUE() can be used instead of
mp_obj_get_int_truncated() to get the value of star_args.
These issues are fixed by this commit, and a test added.
Signed-off-by: Damien George <damien@micropython.org>
This replaces instances of uint with size_t and int with ssize_t in
the mp_call_prepare_args_n_kw_var() function since all of the variables
are used as array offsets.
Also sort headers while we are touching this.
Signed-off-by: David Lechner <david@pybricks.com>
To reach this check, n_kw has to be >= 1 and therefore args2_alloc has
to be >= 2. Therefore new_alloc will always be >= 4. So this check will
never be true and can be removed.
Signed-off-by: David Lechner <david@pybricks.com>
This fixes overallocating an extra mp_obj_t when the length of *args and
**args is known. Previously we were allocating 1 mp_obj_t for each
n_args and n_kw plus the length of each *arg and **arg (if they are
known). Since n_args includes *args and n_kw includes **args, this was
allocating an extra mp_obj_t in addition to the length of these args
when unpacked.
To fix this, we just subtract 1 from the length to account for the 1
already implicitly allocated by n_args and n_kw.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow unpacking multiple
star args in a function or method call.
This is implemented by changing the emitted bytecodes so that both
positional args and star args are stored as positional args. A bitmap is
added to indicate if an argument at a given position is a positional
argument or a star arg.
In the generated code, this new bitmap takes the place of the old star arg.
It is stored as a small int, so this means only the first N arguments can
be star args where N is the number of bits in a small int.
The runtime is modified to interpret this new bytecode format while still
trying to perform as few memory reallocations as possible.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow multiple ** unpackings
when calling a function or method.
The compiler is modified to encode the argument as a None: obj key-value
pair (similar to how regular keyword arguments are encoded as str: obj
pairs). The extra object that was pushed on the stack to hold a single **
unpacking object is no longer used and is removed.
The runtime is modified to decode this new format.
Signed-off-by: David Lechner <david@pybricks.com>
This warning can happen on clang 13.0.1 building mpy-cross:
../py/vm.c:748:25: error: array index -3 refers past the last possible
element for an array in 64-bit address space containing 64-bit (8-byte)
elements (max possible 2305843009213693952 elements)
[-Werror,-Warray-bounds]
sp[-MP_OBJ_ITER_BUF_NSLOTS + 1] = MP_OBJ_NULL;
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using pointer access instead of array access works around this warning.
Fixes issue #8467.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds optimised l32i/s32i functions that select the best load/
store encoding based on the size of the offset, and uses the function when
necessary in code generation.
Without this, ASM_LOAD_REG_REG_OFFSET() could overflow the word offset
(using a narrow encoding), for example when loading the prelude from the
constant table when there are many (>16) constants.
Fixes issue #8458.
Signed-off-by: Damien George <damien@micropython.org>
The sys module should always be available (if it's compiled in), eg to
change sys.path for importing. So provide an explicit alias from "sys" to
"usys" so that "import sys" can always work.
Signed-off-by: Damien George <damien@micropython.org>
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument. These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
Some compilers will warn about unused variables like scope_flags. So use
MP_BC_PRELUDE_SIG_DECODE() which will silence these warnings.
Signed-off-by: Damien George <damien@micropython.org>
This adds a new MP_SMALL_INT_BITS macro that is a compile-time constant
that contains the number of bits available in an MP_SMALL_INT.
We can use this in place of the runtime function mp_small_int_bits().
Signed-off-by: David Lechner <david@pybricks.com>
The bytecode state variables mp_showbc_code_start and mp_showbc_constants
have been removed and made local variables passed into the various
functions.
As part of this, the DECODE_PTR macro is fixed so it extracts the relevant
pointer from the child_table (a regression introduced in
f2040bfc7e).
Signed-off-by: Damien George <damien@micropython.org>
This means that all constants for EMIT_ARG(load_const_obj, obj) are created
in the parser (rather than some in the compiler).
Signed-off-by: Damien George <damien@micropython.org>
This commit adds generic support for mutable module attributes on built in
modules, by adding support for an optional hook function for module
attribute lookup. If a module wants to support additional attribute load/
store/delete (beyond what is in the constant, globals dict) then it should
add at the very end of its globals dict MP_MODULE_ATTR_DELEGATION_ENTRY().
This should point to a custom function which will handle any additional
attributes.
The mp_module_generic_attr() function is provided as a helper function for
additional attributes: it requires an array of qstrs (terminated in
MP_QSTRnull) and a corresponding array of objects (with a 1-1 mapping
between qstrs and objects). If the qstr is found in the array then the
corresponding object is loaded/stored/deleted.
Signed-off-by: Damien George <damien@micropython.org>
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.
No functional change.
The size reduction depends on the number of qstrs in the build. The change
this commit brings is:
bare-arm: -4 -0.007%
minimal x86: +150 +0.092% [incl +48(data)]
unix x64: -608 -0.118%
unix nanbox: -572 -0.126% [incl +32(data)]
stm32: -1392 -0.352% PYBV10
cc3200: -448 -0.244%
esp8266: -1208 -0.173% GENERIC
esp32: -1028 -0.068% GENERIC[incl -1020(data)]
nrf: -440 -0.252% pca10040
rp2: -1072 -0.217% PICO
samd: -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.
Originally at adafruit#4583
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c. Enabling MICROPY_PY_MACHINE will include this
module.
As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).
Signed-off-by: Damien George <damien@micropython.org>
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.
See CPython version of the issue at https://bugs.python.org/issue27772
Signed-off-by: Jeff Epler <jepler@gmail.com>
The qstr_last_chunk is not collected by the garbage collector. This relies
on the assertion that qstr_pool_t also references the qstr_last_chunk. If
an exception is raised while allocating the qstr_pool_t, qstr_last_chunk
has to be invalidated not to become a dangling reference at the next
garbage collection.
Signed-off-by: Emilie Feral <emilie.feral@numworks.com>
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.
But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.
This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).
Signed-off-by: Damien George <damien@micropython.org>
This commit makes sure that the value zero is always encoded in an mpz_t as
neg=0 and len=0 (previously it was just len=0).
This invariant is needed for some of the bitwise operations that operate on
negative numbers, because they cannot handle -0. For example
(-((1<<100)-(1<<100)))|1 was being computed as -65535, instead of 1.
Fixes issue #8042.
Signed-off-by: Damien George <damien@micropython.org>
The mp_sys_path_obj and mp_sys_argv_obj objects are only used by the
runtime and accessible from Python if MICROPY_PY_SYS is enabled. So
exclude them from the runtime state if this option is disabled.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_PY_SYS_PATH_ARGV_DEFAULTS is enabled (which it is by default)
then sys.path and sys.argv will be initialised and populated with default
values. This keeps all bare-metal ports aligned.
Signed-off-by: Damien George <damien@micropython.org>
This ensures MICROPY_QSTR_EXTRA_POOL and MICROPY_MODULE_FROZEN_MPY are set
if necessary before the CFLAGS are extracted for QSTR generation.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This changes makemanifest.py & mpy-tool.py to merge string and mpy names
into the same list (now mp_frozen_names).
The various paths for loading a frozen module (mp_find_frozen_module) and
checking existence of a frozen module (mp_frozen_stat) use a common
function that searches this list.
In addition, the frozen lookup will now only take place if the path starts
with ".frozen", which needs to be added to sys.path.
This fixes issues #1804, #2322, #3509, #6419.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
Make it possible to proceed to a regular lookup in locals_dict if the
custom type->attr fails. This allows type->attr to extend rather than
completely replace the lookup in locals_dict.
This is useful for custom builtin classes that have mostly regular methods
but just a few special attributes/properties. This way, type->attr needs
to deal with the special cases only and the default lookup will be used for
generic methods.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
This makes it possible for cooperative multitasking systems to keep running
event loops during garbage collector operations.
For example, this can be used to ensure that a motor control loop runs
approximately each 5 ms. Without this hook, the loop time can jump to
about 15 ms.
Addresses #3475.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
This is an stm32-specific feature that's accessed via the pyb module, so
not something that will be widely enabled.
Signed-off-by: Damien George <damien@micropython.org>
Some of these will later be moved to CORE or BASIC, but EXTRA is a good
starting point based on what stm32 uses.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.
This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.
The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).
It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.
For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:
diff of scores (higher is better)
N=2000 M=2000 bccache -> attrmapcache diff diff% (error%)
bm_chaos.py 13742.56 -> 13905.67 : +163.11 = +1.187% (+/-3.75%)
bm_fannkuch.py 60.13 -> 61.34 : +1.21 = +2.012% (+/-2.11%)
bm_fft.py 113083.20 -> 114793.68 : +1710.48 = +1.513% (+/-1.57%)
bm_float.py 256552.80 -> 243908.29 : -12644.51 = -4.929% (+/-1.90%)
bm_hexiom.py 521.93 -> 625.41 : +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py 197544.25 -> 217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py 8072.98 -> 8198.75 : +125.77 = +1.558% (+/-3.22%)
misc_aes.py 17283.45 -> 16480.52 : -802.93 = -4.646% (+/-0.82%)
misc_mandel.py 99083.99 -> 128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py 83860.10 -> 82592.56 : -1267.54 = -1.511% (+/-2.27%)
misc_raytrace.py 21490.40 -> 22227.23 : +736.83 = +3.429% (+/-1.88%)
This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).
The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):
diff of scores (higher is better)
N=2000 M=2000 native -> nat-attrmapcache diff diff% (error%)
bm_chaos.py 14130.62 -> 15464.68 : +1334.06 = +9.441% (+/-7.11%)
bm_fannkuch.py 74.96 -> 76.16 : +1.20 = +1.601% (+/-1.80%)
bm_fft.py 166682.99 -> 168221.86 : +1538.87 = +0.923% (+/-4.20%)
bm_float.py 233415.23 -> 265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py 628.59 -> 734.17 : +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py 225418.44 -> 232926.45 : +7508.01 = +3.331% (+/-3.10%)
bm_pidigits.py 6322.00 -> 6379.52 : +57.52 = +0.910% (+/-5.62%)
misc_aes.py 20670.10 -> 27223.18 : +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py 138221.11 -> 152014.01 : +13792.90 = +9.979% (+/-2.46%)
misc_pystone.py 85032.14 -> 105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py 19800.01 -> 23350.73 : +3550.72 = +17.933% (+/-2.79%)
In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.
See #7680 for further discussion. And see also #7653 for a discussion
about simplifying mpy-cross options.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The existing inline bytecode caching optimisation, selected by
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, reserves an extra byte in the
bytecode after certain opcodes, which at runtime stores a map index of the
likely location of this field when looking up the qstr. This scheme is
incompatible with bytecode-in-ROM, and doesn't work with native generated
code. It also stores bytecode in .mpy files which is of a different format
to when the feature is disabled, making generation of .mpy files more
complex.
This commit provides an alternative optimisation via an approach that adds
a global cache for map offsets, then all mp_map_lookup operations use it.
It's less precise than bytecode caching, but allows the cache to be
independent and external to the bytecode that is executing. It also works
for the native emitter and adds a similar performance boost on top of the
gain already provided by the native emitter.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When the LOAD_ATTR opcode is executed there are quite a few different cases
that have to be handled, but the common case is accessing a member on an
instance type. Typically, built-in types provide methods which is why this
is common.
Fortunately, for this specific case, if the member is found in the member
map then there's no further processing.
This optimisation does a relatively cheap check (type is instance) and then
forwards directly to the member map lookup, falling back to the regular
path if necessary.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is the beginning of a set of changes to simplify enabling/disabling
features. The goals are:
- Remove redundancy from mpconfigport.h (never set a value to the default
-- make it clear exactly what's being enabled).
- Improve consistency between ports. All "similar" ports (i.e. approx same
flash size) should get the same features.
- Simplify mpconfigport.h -- just get default/sensible options for the size
of the port.
- Make it easy for defining constrained boards (e.g. STM32F0/L0), they can
just set a lower level.
This commit makes a step towards this and defines the "core" level as the
current default feature set, and a "minimal" level to turn off everything.
And a few placeholder levels are added for where the other ports will
roughly land.
This is a no-op change for all ports.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit simplifies and optimises the parse tree in-memory
representation of lists of expressions, for tuples and lists, and when
tuples are used on the left-hand-side of assignments and within del
statements. This reduces memory usage of the parse tree when such code is
compiled, and also reduces the size of the compiler.
For example, (1,) was previously the following parse tree:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=2)
int(1)
testlist_comp_3b(149) (n=1)
NULL
NULL
and with this commit is now:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=1)
int(1)
NULL
Similarly, (1, 2, 3) was previously:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=2)
int(1)
testlist_comp_3c(150) (n=2)
int(2)
int(3)
NULL
and is now:
expr_stmt(5) (n=2)
atom_paren(45) (n=1)
testlist_comp(146) (n=3)
int(1)
int(2)
int(3)
NULL
Signed-off-by: Damien George <damien@micropython.org>
This commit refactors machine.PWM and creates extmod/machine_pwm.c. The
esp8266, esp32 and rp2 ports all use this and provide implementations of
the required PWM functionality. This helps to reduce code duplication and
keep the same Python API across ports.
This commit does not make any functional changes.
Signed-off-by: Damien George <damien@micropython.org>
The zephyr port doesn't support SoftI2C so it's not enabled, and the legacy
I2C constructor check can be removed.
Signed-off-by: Damien George <damien@micropython.org>
This is a generic API for synchronously bit-banging data on a pin.
Initially this adds a single supported encoding, which supports controlling
WS2812 LEDs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This was missed in 692d36d779. It's not
strictly necessary as the GC will clean it anyway, but it's good to
pre-emptively gc_free() all the blocks used in lexing/parsing.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.
It is implemented in the lexer as a syntax translation to `str.format`:
f"{a}" --> "{}".format(a)
It also supports:
f"{a=}" --> "a={}".format(a)
This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.
There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
`SyntaxError: raw f-strings are not supported`.
- literal concatenation of f-strings with adjacent strings will fail
"{}" f"{a}" --> "{}{}".format(a) (str.format will incorrectly use
the braces from the non-f-string)
f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)
- PEP-498 requires the full parser to understand the interpolated
argument, however because this entirely runs in the lexer it cannot
resolve nested braces in expressions like
f"{'}'}"
- The !r, !s, and !a conversions are not supported.
Includes tests and cpydiffs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Commit 4173950658 removed automatic building
of mpy-cross, which rebuilt it whenever any of its dependent source files
changed.
But needing to build mpy-cross, and not knowing how, is a frequent issue.
This commit aims to help by automatically building mpy-cross only if it
doesn't exist. For Makefiles it uses an order-only prerequisite, while
for CMake it uses a custom command.
If MICROPY_MPYCROSS (which is what makemanifest.py uses to locate the
mpy-cross executable) is defined in the environment then automatic build
will not be attempted, allowing a way to prevent this auto-build if needed.
Thanks to Trammell Hudson aka @osresearch for the original idea; see #5760.
Signed-off-by: Damien George <damien@micropython.org>
Optionally enabled via MICROPY_PY_UJSON_SEPARATORS. Enabled by default.
For dump, make sure mp_get_stream_raise is called after
mod_ujson_separators since CPython does it in this order (if both
separators and stream are invalid, separators will raise an exception
first).
Add separators argument in the docs as well.
Signed-off-by: Peter Züger <zueger.peter@icloud.com>
Signed-off-by: Damien George <damien@micropython.org>
Commit e33bc597 ("py: Remove calls to file reader functions when these
are disabled.") changed the condition for one caller of
do_execute_raw_code() from
MICROPY_PERSISTENT_CODE_LOAD
to
MICROPY_HAS_FILE_READER && MICROPY_PERSISTENT_CODE_LOAD
The condition that enables compiling the function itself needs to be
changed to match.
Signed-off-by: David Lechner <david@pybricks.com>
Previously a subclass of a type that didn't implement unary_op, or didn't
handle MP_UNARY_OP_BOOL, would raise TypeError on bool conversion.
Fixes#5677.
This adds #if MICROPY_PY_USELECT_SELECT around the uselect.select()
function. According to the docs, this function is only for CPython
compatibility and should not normally be used. So we can disable it
and save a few bytes of flash space where possible.
Signed-off-by: David Lechner <david@pybricks.com>
The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a
StopIteration() exception object, and means that heap memory does not need
to be allocated for the exception (in cases where it can be used). This
commit allows this optimised object to take an optional argument (before,
it could only have no argument).
The commit also adds some new tests to cover corner cases with
StopIteration and generators that previously did not work.
Signed-off-by: Damien George <damien@micropython.org>
This adds the --tags argument to the git describe command that is used
to define the MICROPY_GIT_TAG macro. This makes it match non-annotated
tags. This is useful for MicroPython derivatives that don't use
annotated tags.
Signed-off-by: David Lechner <david@pybricks.com>
Only include .c and .cpp files explicitly in the list of files passed to
the preprocessor for QSTR extraction. All relevant .h files will be
included in this process by "#include" from the .c(pp) files. In
particular for moduledefs.h, this is included by py/objmodule.c (and
doesn't actually contain any extractable MP_QSTR_xxx, but rather defines
macros with MP_QSTR_xxx's in them which are then part of py/objmodule.c).
The main reason for this change is to simplify the preprocessing step on
the javascript port, which tries to compile .h files as C++ precompiled
headers if they are passed with -E to clang.
Signed-off-by: Damien George <damien@micropython.org>
Fixes the following (the line numbers match commit 0e87459e2b):
../../extmod/crypto-algorithms/sha256.c:49:19: runtime error: left shif...
../../extmod/moduasyncio.c:106:35: runtime error: member access within ...
../../py/binary.c:210:13: runtime error: left shift of negative value -...
../../py/mpz.c:744:16: runtime error: negation of -9223372036854775808 ...
../../py/objint.c:109:22: runtime error: left shift of 1 by 31 places c...
../../py/objint_mpz.c:374:9: runtime error: left shift of 4611686018427...
../../py/objint_mpz.c:374:9: runtime error: left shift of negative valu...
../../py/parsenum.c:106:14: runtime error: left shift of 46116860184273...
../../py/runtime.c:395:33: runtime error: left shift of negative value ...
../../py/showbc.c:177:28: runtime error: left shift of negative value -...
../../py/vm.c:321:36: runtime error: left shift of negative value -1```
Testing was done on an amd64 Debian Buster system using gcc-8.3 and these
settings:
CFLAGS += -g3 -Og -fsanitize=undefined
LDFLAGS += -fsanitize=undefined
The introduced TASK_PAIRHEAP macro's conditional (x ? &x->i : NULL)
assembles (under amd64 gcc 8.3 -Os) to the same as &x->i, since i is the
initial field of the struct. However, for the purposes of undefined
behavior analysis the conditional is needed.
Signed-off-by: Jeff Epler <jepler@gmail.com>
This introduces a new macro to get the main thread and uses it to ensure
that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only
scheduled on the main thread. This is more deterministic than being
scheduled on a random thread and is more in line with CPython that only
allow signal handlers to run on the main thread.
Fixes issue #7026.
Signed-off-by: David Lechner <david@pybricks.com>
This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t.
This allows exceptions to be scheduled on a specific thread.
Signed-off-by: David Lechner <david@pybricks.com>
Prior to this commit, cache flushing for ARM native code was done only in
the assembler code asm_thumb_end_pass()/asm_arm_end_pass(), at the last
pass of the assembler. But this misses flushing the cache when loading
native code from an .mpy file, ie in persistentcode.c.
The change here makes sure the cache is always flushed/cleaned/invalidated
when assigning native code on ARM architectures.
This problem was found running tests/micropython/import_mpy_native_gc.py on
the mimxrt port.
Signed-off-by: Damien George <damien@micropython.org>
asan considers that memcmp(p, q, N) is permitted to access N bytes at each
of p and q, even for values of p and q that have a difference earlier.
Accessing additional values is frequently done in practice, reading 4 or
more bytes from each input at a time for efficiency, so when completing
"non_exist<TAB>" in the repl, this causes a diagnostic:
==16938==ERROR: AddressSanitizer: global-buffer-overflow on
address 0x555555cd8dc8 at pc 0x7ffff726457b bp 0x7fffffffda20 sp 0x7fff
READ of size 9 at 0x555555cd8dc8 thread T0
#0 0x7ffff726457a (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xb857a)
#1 0x555555b0e82a in mp_repl_autocomplete ../../py/repl.c:301
#2 0x555555c89585 in readline_process_char ../../lib/mp-readline/re
#3 0x555555c8ac6e in readline ../../lib/mp-readline/readline.c:513
#4 0x555555b8dcbd in do_repl /home/jepler/src/micropython/ports/uni
#5 0x555555b90859 in main_ /home/jepler/src/micropython/ports/unix/
#6 0x555555b90a3a in main /home/jepler/src/micropython/ports/unix/m
#7 0x7ffff619a09a in __libc_start_main ../csu/libc-start.c:308
#8 0x55555595fd69 in _start (/home/jepler/src/micropython/ports/uni
0x555555cd8dc8 is located 0 bytes to the right of global variable
'import_str' defined in '../../py/repl.c:285:23' (0x555555cd8dc0) of
size 8
'import_str' is ascii string 'import '
Signed-off-by: Jeff Epler <jepler@gmail.com>
The proper way to do this is to test for __APPLE__ and __MACH__, where
__APPLE__ tests for an Apple OS and __MACH__ tests that it is based on CMU
Mach. Using both tests ensures that just Darwin (Apple's open source base
for MacOS, iOS, etc.) is recognized. __APPLE__ by itself will test for any
Apple OS, which can include older OS 7-9 and any future Apple OS. __MACH__
tests for any OS based on CMU Mach, including Darwin and GNU Hurd.
Fixes#7232.
