The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a
StopIteration() exception object, and means that heap memory does not need
to be allocated for the exception (in cases where it can be used). This
commit allows this optimised object to take an optional argument (before,
it could only have no argument).
The commit also adds some new tests to cover corner cases with
StopIteration and generators that previously did not work.
Signed-off-by: Damien George <damien@micropython.org>
This adds the --tags argument to the git describe command that is used
to define the MICROPY_GIT_TAG macro. This makes it match non-annotated
tags. This is useful for MicroPython derivatives that don't use
annotated tags.
Signed-off-by: David Lechner <david@pybricks.com>
Only include .c and .cpp files explicitly in the list of files passed to
the preprocessor for QSTR extraction. All relevant .h files will be
included in this process by "#include" from the .c(pp) files. In
particular for moduledefs.h, this is included by py/objmodule.c (and
doesn't actually contain any extractable MP_QSTR_xxx, but rather defines
macros with MP_QSTR_xxx's in them which are then part of py/objmodule.c).
The main reason for this change is to simplify the preprocessing step on
the javascript port, which tries to compile .h files as C++ precompiled
headers if they are passed with -E to clang.
Signed-off-by: Damien George <damien@micropython.org>
Fixes the following (the line numbers match commit 0e87459e2b):
../../extmod/crypto-algorithms/sha256.c:49:19: runtime error: left shif...
../../extmod/moduasyncio.c:106:35: runtime error: member access within ...
../../py/binary.c:210:13: runtime error: left shift of negative value -...
../../py/mpz.c:744:16: runtime error: negation of -9223372036854775808 ...
../../py/objint.c:109:22: runtime error: left shift of 1 by 31 places c...
../../py/objint_mpz.c:374:9: runtime error: left shift of 4611686018427...
../../py/objint_mpz.c:374:9: runtime error: left shift of negative valu...
../../py/parsenum.c:106:14: runtime error: left shift of 46116860184273...
../../py/runtime.c:395:33: runtime error: left shift of negative value ...
../../py/showbc.c:177:28: runtime error: left shift of negative value -...
../../py/vm.c:321:36: runtime error: left shift of negative value -1```
Testing was done on an amd64 Debian Buster system using gcc-8.3 and these
settings:
CFLAGS += -g3 -Og -fsanitize=undefined
LDFLAGS += -fsanitize=undefined
The introduced TASK_PAIRHEAP macro's conditional (x ? &x->i : NULL)
assembles (under amd64 gcc 8.3 -Os) to the same as &x->i, since i is the
initial field of the struct. However, for the purposes of undefined
behavior analysis the conditional is needed.
Signed-off-by: Jeff Epler <jepler@gmail.com>
This introduces a new macro to get the main thread and uses it to ensure
that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only
scheduled on the main thread. This is more deterministic than being
scheduled on a random thread and is more in line with CPython that only
allow signal handlers to run on the main thread.
Fixes issue #7026.
Signed-off-by: David Lechner <david@pybricks.com>
This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t.
This allows exceptions to be scheduled on a specific thread.
Signed-off-by: David Lechner <david@pybricks.com>
Prior to this commit, cache flushing for ARM native code was done only in
the assembler code asm_thumb_end_pass()/asm_arm_end_pass(), at the last
pass of the assembler. But this misses flushing the cache when loading
native code from an .mpy file, ie in persistentcode.c.
The change here makes sure the cache is always flushed/cleaned/invalidated
when assigning native code on ARM architectures.
This problem was found running tests/micropython/import_mpy_native_gc.py on
the mimxrt port.
Signed-off-by: Damien George <damien@micropython.org>
asan considers that memcmp(p, q, N) is permitted to access N bytes at each
of p and q, even for values of p and q that have a difference earlier.
Accessing additional values is frequently done in practice, reading 4 or
more bytes from each input at a time for efficiency, so when completing
"non_exist<TAB>" in the repl, this causes a diagnostic:
==16938==ERROR: AddressSanitizer: global-buffer-overflow on
address 0x555555cd8dc8 at pc 0x7ffff726457b bp 0x7fffffffda20 sp 0x7fff
READ of size 9 at 0x555555cd8dc8 thread T0
#0 0x7ffff726457a (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xb857a)
#1 0x555555b0e82a in mp_repl_autocomplete ../../py/repl.c:301
#2 0x555555c89585 in readline_process_char ../../lib/mp-readline/re
#3 0x555555c8ac6e in readline ../../lib/mp-readline/readline.c:513
#4 0x555555b8dcbd in do_repl /home/jepler/src/micropython/ports/uni
#5 0x555555b90859 in main_ /home/jepler/src/micropython/ports/unix/
#6 0x555555b90a3a in main /home/jepler/src/micropython/ports/unix/m
#7 0x7ffff619a09a in __libc_start_main ../csu/libc-start.c:308
#8 0x55555595fd69 in _start (/home/jepler/src/micropython/ports/uni
0x555555cd8dc8 is located 0 bytes to the right of global variable
'import_str' defined in '../../py/repl.c:285:23' (0x555555cd8dc0) of
size 8
'import_str' is ascii string 'import '
Signed-off-by: Jeff Epler <jepler@gmail.com>
The proper way to do this is to test for __APPLE__ and __MACH__, where
__APPLE__ tests for an Apple OS and __MACH__ tests that it is based on CMU
Mach. Using both tests ensures that just Darwin (Apple's open source base
for MacOS, iOS, etc.) is recognized. __APPLE__ by itself will test for any
Apple OS, which can include older OS 7-9 and any future Apple OS. __MACH__
tests for any OS based on CMU Mach, including Darwin and GNU Hurd.
Fixes#7232.
Array equality is defined as each element being equal but to keep
code size down MicroPython implements a binary comparison. This
can only be used correctly for elements with the same binary layout
though so turn it into an NotImplementedError when comparing types
for which the binary comparison yielded incorrect results: types
with different sizes, and floating point numbers because nan != nan.
This commit makes gc_lock_depth have one counter per thread, instead of one
global counter. This makes threads properly independent with respect to
the GC, in particular threads can now independently lock the GC for
themselves without locking it for other threads. It also means a given
thread can run a hard IRQ without temporarily locking the GC for all other
threads and potentially making them have MemoryError exceptions at random
locations (this really only occurs on MCUs with multiple cores and no GIL,
eg on the rp2 port).
The commit also removes protection of the GC lock/unlock functions, which
is no longer needed when the counter is per thread (and this also fixes the
cas where a hard IRQ calling gc_lock() may stall waiting for the mutex).
It also puts the check for `gc_lock_depth > 0` outside the GC mutex in
gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from
waiting on a mutex if it does attempt to allocate heap memory (and putting
the check outside the GC mutex is now safe now that there is a
gc_lock_depth per thread).
Signed-off-by: Damien George <damien@micropython.org>
Doing "import <tab>" will now complete/list built-in modules.
Originally at adafruit#4548 and adafruit#4608
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
Anything beginning with "_" will now only be tab-completed if there is
already a partial match for such an entry. In other words, entering
foo.<tab> will no longer complete/list anything beginning with "_".
Originally at adafruit#1850
Signed-off-by: Kathryn Lingel <kathryn@lingel.net>
These commented-out lines of code have been unused for a long time, so
remove them to avoid confusion as to why they are there.
mp_obj_dict_free() never existed, this line was converted from
mp_map_deinit() and commented out as soon as it was added. The call to
mp_map_deinit(mp_loaded_modules_map) was commented in
1a1d11fa32.
Fixes issue #3507.
Signed-off-by: Damien George <damien@micropython.org>
This helper is added to properly set a pending exception, to mirror
mp_sched_schedule(), which schedules a function.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_ENABLE_SCHEDULER is enabled then MP_STATE_VM(sched_state) must
be updated after handling the pending exception, which is done by the
mp_handle_pending() function.
Signed-off-by: Damien George <damien@micropython.org>
This introduces a new option, MICROPY_ERROR_REPORTING_NONE, which
completely disables all error messages. To be used in cases where
MicroPython needs to fit in very limited systems.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the errno attribute to exceptions, so code can retrieve
errno codes from an OSError using exc.errno.
The implementation here simply lets `errno` (and the existing `value`)
attributes work on any exception instance (they both alias args[0]). This
is for efficiency and to keep code size down. The pros and cons of this
are:
Pros:
- more compatible with CPython, less difference to document and learn
- OSError().errno will correctly return None, whereas the current way of
doing it via OSError().args[0] will raise an IndexError
- it reduces code size on most bare-metal ports (because they already have
the errno qstr)
- for Python code that uses exc.errno the generated bytecode is 2 bytes
smaller and more efficient to execute (compared with exc.args[0]); so
bytecode loaded to RAM saves 2 bytes RAM for each use of this attribute,
and bytecode that is frozen saves 2 bytes flash/ROM for each use
- it's easier/shorter to type, and saves 2 bytes of space in .py files that
use it (for each use)
Cons:
- increases code size by 4-8 bytes on minimal ports that don't already have
the `errno` qstr
- all exceptions now have .errno and .value attributes (a cpydiff test is
added to address this)
See also #2407.
Signed-off-by: Damien George <damien@micropython.org>
This allows configuring the pre-allocated size of sys.modules dict, in
order to prevent unwanted reallocations at run-time (3 sys-modules is
really not quite enough for a larger project).
When building with STATIC undefined (e.g., -DSTATIC=), there are two
instances of mp_type_code that collide at link time: in profile.c and in
builtinevex.c. This patch resolves the collision by renaming one of them.
The parts that are generic are added to py/ so they can be used by other
ports that use CMake.
py/usermod.cmake:
* Creates a usermod target to hang user C/CXX modules from.
* Gathers sources from user C/CXX modules and libs for QSTR scan.
ports/rp2/CMakeLists.txt:
* Includes py/usermod.cmake.
* Links the resulting usermod library to the MicroPython target.
py/mkrules.cmake:
Add cxxflags to qstr.i.last custom command for CXX modules:
* MICROPY_CPP_FLAGS so CXX modules will find includes.
* -DNO_QSTR to fix fatal error missing "genhdr/qstrdefs.generated.h".
Usage:
The rp2 port can be linked against user C modules by running:
make USER_C_MODULES=/path/to/module/micropython.cmake
CMake will print a list of included modules.
Co-authored-by: Graham Sanderson <graham.sanderson@raspberrypi.org>
Co-authored-by: Michael O'Cleirigh <michael.ocleirigh@rivulet.ca>
Signed-off-by: Phil Howard <phil@pimoroni.com>
mp_printf should be used to print the prefix because it's also used in
mp_bytecode_print2 (otherwise, depending on the system, different output
streams may be used).
Also print the current thread state when threading is enabled to easily see
which thread executes what opcode.
Signed-off-by: Damien George <damien@micropython.org>
This allows a port to specify a custom qstrdefsport.h file, the same as the
QSTR_DEFS variable in a Makefile.
Signed-off-by: Damien George <damien@micropython.org>
The core cmake rules use custom commands to invoke qstr processing
scripts. For the zephyr port, it's possible that list arguments to these
commands may contain generator expressions, therefore we need to expand
them properly.
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
For certain operands to mpn_div, the existing code path for
`DIG_SIZE == MPZ_DBL_DIG_SIZE / 2` had a bug in it where borrow could still
overflow in the `(x >= *n || *n - x <= borrow)` branch, ie
`borrow + x - (mpz_dbl_dig_t)*n` overflows the borrow variable. In such
cases the subsequent right-shift of borrow would not bring in the overflow
bit, leading to an error in the result. An example division that had
overflow when MPZ_DIG_SIZE = 16 is `(2 ** 48 - 1) ** 2 // (2 ** 48 - 1)`.
This is fixed in this commit by simplifying the code and handling the low
digits of borrow first, and then the upper bits (to shift down) separately.
There is no longer a distinction between `DIG_SIZE < MPZ_DBL_DIG_SIZE / 2`
and `DIG_SIZE == MPZ_DBL_DIG_SIZE / 2`.
This commit also simplifies the second part of the calculation so that
borrow does not need to be negated (instead the code just works knowing
that borrow is negative and using + instead of - in calculations involving
borrow).
Fixes#6777.
Signed-off-by: Damien George <damien@micropython.org>
The "word" referred to by BYTES_PER_WORD is actually the size of mp_obj_t
which is not always the same as the size of a pointer on the target
architecture. So rename this config value to better reflect what it
measures, and also prefix it with MP_.
For uses of BYTES_PER_WORD in setting the stack limit this has been
changed to sizeof(void *), because the stack usually grows with
machine-word sized values (eg an nlr_buf_t has many machine words in it).
Signed-off-by: Damien George <damien@micropython.org>
It's only used in one location, to test if << or >> will overflow when
shifting mp_uint_t. For such a test it's clearer to use sizeof(lhs_val),
which will be valid even if the type of lhs_val changes.
Signed-off-by: Damien George <damien@micropython.org>
This environment variable, if defined during the build process,
indicates a fixed time that should be used in place of "now" when
such a time is explicitely referenced.
This allows for reproducible builds of micropython.
See https://reproducible-builds.org/specs/source-date-epoch/
Signed-off-by: iTitou <moiandme@gmail.com>
This should be enabled when the mp_raw_code_save_file function is needed.
It is enabled for mpy-cross, and a check for defined(__APPLE__) is added to
cover Mac M1 systems.
It practically does the same as qstr_from_str and was only used in one
place, which should actually use the compile-time MP_QSTR_XXX form for
consistency; qstr_from_str is for runtime strings only.
Adds a new compile-time option MICROPY_EMIT_THUMB_ARMV7M which is enabled
by default (to get existing behaviour) and which should be disabled (set to
0) when building native emitter support (@micropython.native) on ARMv6M
targets.
This returns a reference to the globals dict associated with the function,
ie the global scope that the function was defined in. This attribute is
read-only but the dict itself is modifiable, per CPython behaviour.
Signed-off-by: Damien George <damien@micropython.org>
As a general pattern, required positional arguments that are not named do
not need to be parsed using mp_arg_parse_all().
Signed-off-by: Damien George <damien@micropython.org>
Two issues are tackled:
1. The calculation of the correct length to print is fixed to treat the
precision as a maximum length instead as the exact length.
This is done for both qstr (%q) and for regular str (%s).
2. Fix the incorrect use of mp_printf("%.*s") to mp_print_strn().
Because of the fix of above issue, some testcases that would print
an embedded null-byte (^@ in test-output) would now fail.
The bug here is that "%s" was used to print null-bytes. Instead,
mp_print_strn is used to make sure all bytes are outputted and the
exact length is respected.
Test-cases are added for both %s and %q with a combination of precision
and padding specifiers.
Also known as L2CAP "connection oriented channels". This provides a
socket-like data transfer mechanism for BLE.
Currently only implemented for NimBLE on STM32 / Unix.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This gives a substantial speedup of the preprocessing step, i.e. the
generation of qstr.i.last. For example on a clean build, making
qstr.i.last:
21s -> 4s on STM32 (WB55)
8.9 -> 1.8s on Unix (dev).
Done in collaboration with @stinos.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Support C++ code in .cpp files by providing CXX counterparts of the
_USERMOD_ flags we have for C already. This merely enables the Makefile of
user C modules to use variables specific to C++ compilation, it is still up
to each port's main Makefile to also include these in the build.
When SCR_QSTR contains C++ files they should be preprocessed with the same
compiler flags (CXXFLAGS) as they will be compiled with, to make sure code
scanned for QSTR occurrences is effectively the code used in the rest of
the build. The 'split SCR_QSTR in .c and .cpp files and process each with
different flags' logic isn't trivial to express in a Makefile and the
existing principle for deciding which files to preprocess was already
rather complicated, so the actual preprocessing is moved into
makeqstrdefs.py completely.
When process_file() is passed a preprocessed C++ file for instance it won't
find any lines containing .c files and the last_fname variable remains
None, so handle that gracefully.
Newer GCC versions are able to warn about switch cases that fall
through. This is usually a sign of a forgotten break statement, but in
the few cases where a fall through is intended we annotate it with this
macro to avoid the warning.
Like Clang, GCC warns about this file, but only with -Woverride-init
which is enabled by -Wextra. Disable the warnings for this file just
like we do for Clang to make -Wextra happy.
When compiling with -Wextra which includes -Wmissing-field-initializers
GCC will warn that the defval field of mp_arg_val_t is not initialized.
This is just a warning as it is defined to be zero initialized, but since
it is a union it makes sense to be explicit about which member we're
going to use, so add the explicit initializers and get rid of the
warning.
On x86 chars are signed, but we're comparing a char to '0' + unsigned int,
which is promoted to an unsigned int. Let's promote the char to unsigned
before doing the comparison to avoid weird corner cases.
The function scope_find_or_add_id used to take a scope_kind_t enum and
save it in an uint8_t. Saving an enum in a uint8_t is fine, but
everywhere this function is called it is not actually given a
scope_kind_t but an anonymous enum instead. Let's give this enum a name
and use that as the argument type.
This doesn't change the generated code, but is a C type mismatch that
unfortunately doesn't show up unless you enable -Wenum-conversion.
Damien George
Bryan Tong Minh
David Lechner
Yonatan Goldschmidt
Jeff Epler
Bob Abeles
stijn
Artyom Skrobov
scottbelden
Kathryn Lingel
matejcik
Phil Howard
Maureen Helm
Xiang Xiao
iTitou
David CARLIER
graham sanderson
Joris Peeraer
Jim Mussared
Emil Renner Berthing