Prior to this commit, importing a module that exists but has a syntax error
or some other problem that happens at import time would result in a
potentially-incomplete module object getting added to sys.modules.
Subsequent imports would use that object, resulting in confusing error
messages that hide the root cause of the problem.
This commit fixes that issue by removing the failed module from sys.modules
using the new NLR callback mechanism.
Note that it is still important to add the module to sys.modules while the
import is happening so that we can support circular imports just like
CPython does.
Fixes issue #967.
Signed-off-by: David Grayson <davidegrayson@gmail.com>
The changed functions now use less stack, and don't have any issues with
local variables needing to be declared volatile.
Testing on a PYBv1.0, imports (of .py, .mpy and frozen code) now use 64
less bytes of C stack per import depth.
Signed-off-by: Damien George <damien@micropython.org>
NLR buffers are usually quite large (use lots of C stack) and expensive to
push and pop. Some of the time they are only needed to perform clean up if
an exception happens, and then they re-raise the exception.
This commit allows optimizing that scenario by introducing a linked-list of
NLR callbacks that are called automatically when an exception is raised.
They are essentially a light-weight NLR handler that can implement a
"finally" block, i.e. clean-up when an exception is raised, or (by passing
`true` to nlr_pop_jump_callback) when execution leaves the scope.
Signed-off-by: Damien George <damien@micropython.org>
This can lead to duplicate initialisations if a module can be imported
via multiple names, so the module must track this itself anyway.
This reduces code size (diff is -40 bytes), and avoids special treatment of
builtin-modules-with-init with respect to sys.modules. No other builtin
modules get put into sys.modules.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
To use this:
- Create a built-in module, and add the module object as a member of the
parent module's globals dict.
- The submodule can set its `__name__` to either `QSTR_foo_dot_bar` or
`QSTR_bar`. The former requires using qstrdefs(port).h to make the qstr.
Because `bar` is a member of `foo`'s globals, it is possible to write
`import foo` and then immediately use `foo.bar` without importing it
explicitly. This means that if `bar` has an `__init__`, it will not be
called in this situation, and for that reason, sub-modules should not have
`__init__` methods. If this is required, then all initalisation for
sub-modules should be done by the top-level module's (i.e. `foo`'s)
`__init__` method.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This makes it so that sub-packages are resolved relative to their parent's
`__path__`, rather than re-resolving each parent's filesystem path.
The previous behavior was that `import foo.bar` would first re-search
`sys.path` for `foo`, then use the resulting path to find `bar`.
For already-loaded and u-prefixed modules, because we no longer need to
build the path from level to level, we no longer unnecessarily search
the filesystem. This should improve startup time.
Explicitly makes the resolving process clear:
- Loaded modules are returned immediately without touching the filesystem.
- Exact-match of builtins are also returned immediately.
- Then the filesystem search happens.
- If that fails, then the weak-link handling is applied.
This maintains the existing behavior: if a user writes `import time` they
will get time.py if it exits, otherwise the built-in utime. Whereas `import
utime` will always return the built-in.
This also fixes a regression from a7fa18c203
where we search the filesystem for built-ins. It is now only possible to
override u-prefixed builtins. This will remove a lot of filesystem stats
at startup, as micropython-specific modules (e.g. `pyb`) will no longer
attempt to look at the filesystem.
Added several improvements to the comments and some minor renaming and
refactoring to make it clearer how the import mechanism works. Overall
code size diff is +56 bytes on STM32.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
If sys.path is enabled, but empty, this will now no longer search the
filesystem. Previously an empty sys.path was equivalent to having
`sys.path=[""]`. This is a breaking change, but this behavior now matches
CPython.
This also provides an alternative mechanism to the u-prefix to force an
import of a builtin module:
```
import sys
_path = sys.path[:]
sys.path.clear()
import foo # Forces the built-in foo.
sys.path.extend(_path)
del _path
```
Code size diff is -32 bytes on PYBV11.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This generalises and simplifies the code and follows CPython behaviour.
See similar change for floats in a07fc5b640.
Signed-off-by: Damien George <damien@micropython.org>
This is possible now that MP_UNARY_OP_INT_MAYBE exists.
As a consequence mp_obj_get_int now also supports user types, which was
previously possible with MP_UNARY_OP_INT but no tests existed for it.
Signed-off-by: Damien George <damien@micropython.org>
To be consistent with MP_UNARY_OP_INT_FLOAT and MP_UNARY_OP_INT_COMPLEX,
and allow int() to first check if a type supports __int__ before trying
other things (as per CPython).
Signed-off-by: Damien George <damien@micropython.org>
The code that handles inplace-operator to normal-binary-operator fallback
is moved in this commit from py/objtype.c to py/runtime.c, making it apply
to all types, not just user classes.
Signed-off-by: Damien George <damien@micropython.org>
So that user types can implement reverse operators and have them work with
str on the left-hand-side, eg `"a" + UserType()`.
Signed-off-by: Damien George <damien@micropython.org>
This adds a unary_op implementation for the dict_view type that makes
the implementation of `hash()` for these types compatible with CPython.
Signed-off-by: David Lechner <david@pybricks.com>
As per https://bugs.python.org/issue408326, the slice object should not be
hashable. Since MicroPython has an implicit fallback when the unary_op
slot is empty, we need to fill this slot.
Signed-off-by: David Lechner <david@pybricks.com>
Since converting to variable sized slots in mp_obj_type_t, we can now
reduce the code size a bit by removing mp_generic_unary_op() and the
corresponding slots where it is used. Instead we just implement the
generic `__hash__` operation in the runtime.
Signed-off-by: David Lechner <david@pybricks.com>
Changes in this commit:
- Add MICROPY_GC_HOOK_LOOP to gc_info() and gc_alloc(). Both of these can
be long running (many milliseconds) which is too long to be blocking in
some applications.
- Pass loop variable to MICROPY_GC_HOOK_LOOP(i) macro so that implementers
can use it, e.g. to improve performance by only calling a function every
X number of iterations.
- Drop outer call to MICROPY_GC_HOOK_LOOP in gc_mark_subtree().
This class allows much more expressive sound synthesis:
* tremolo & vibrato
* arbitrary frequency
* different evelope & waveform per note
* all properties dynamically settable from Python code
When a tuple is the condition of an if statement, it's only possible to
optimise that tuple away when it is a constant tuple (ie all its elements
are constants), because if it's not constant then the elements must be
evaluated in case they have side effects (even though the resulting tuple
will always be "true").
The code before this change handled the empty tuple OK (because it doesn't
need to be evaluated), but it discarded non-empty tuples without evaluating
them, which is incorrect behaviour (as show by the updated test).
This optimisation is anyway rarely applied because it's not common Python
coding practice to write things like `if (): ...` and `if (1, 2): ...`, so
removing this optimisation completely won't affect much code, if any.
Furthermore, when MICROPY_COMP_CONST_TUPLE is enabled, constant tuples are
already optimised by the parser, so expression with constant tuples like
`if (): ...` and `if (1, 2): ...` will continue to be optimised properly
(and so when this option is enabled the code that's deleted in this commit
is actually unreachable when the if condition is a constant tuple).
Signed-off-by: Damien George <damien@micropython.org>
This adds a script to generate the peripherals files (except clock).
It adds support for the 1015, 1020, 1040, and 1050 EVKs.
Some work was started on 1176 but it isn't working. So, the board
def is in a separate branch.
Fixes#3521. Fixes#2477.
Based on extmod/utime_mphal.c, with:
- a globals dict added
- time.localtime wrapper added
- time.time wrapper added
- time.time_ns function added
New configuration options are added for this module:
- MICROPY_PY_UTIME (enabled at basic features level)
- MICROPY_PY_UTIME_GMTIME_LOCALTIME_MKTIME
- MICROPY_PY_UTIME_TIME_TIME_NS
Signed-off-by: Damien George <damien@micropython.org>
This adds a mechanism to track a pending notify/indicate operation that
is deferred due to the send buffer being full. This uses a tracked alloc
that is passed as the content arg to the callback.
This replaces the previous mechanism that did this via the global pending
op queue, shared with client read/write ops.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
`locals_dict` and `attr` are incompatible, so just use circuitpython-style
properties so that a property and a method are both available.
this makes no difference in circuitpython, where
`MICROPY_PY_BUILTINS_MEMORYVIEW_ITEMSIZE` is never enabled
PicoDVI in CP support 640x480 and 800x480 on Feather DVI, Pico and
Pico W. 1 and 2 bit grayscale are full resolution. 8 and 16 bit
color are half resolution.
Memory layout is modified to give the top most 4k of ram to the
second core. Its MPU is used to prevent flash access after startup.
The port saved word is moved to a watchdog scratch register so that
it doesn't get overwritten by other things in RAM.
Right align status bar and scroll area. This normally gives a few
pixels of padding on the left hand side and improves the odds it is
readable in a case. Fixes#7562
Fixes c stack checking. The length was correct but the top was being
set to the current stack pointer instead of the correct top.
Fixes#7643
This makes Bitmap subscr raise IndexError instead of ValueError
when the index arguments are wrong.
The GreenHills preprocessor produces #line directives without a file name,
which the regular expression used to distiguish between
"# <number> file..." (GCC and similar) and "#line <number> file..."
(Microsoft C and similar) does not match, aborting processing.
Besides, the regular expression was unnecessarily wide, matching lines
containing a "#", followed by any number of 'l','i','n', and 'e'
characters.
Signed-off-by: Alex Riesen <alexander.riesen@cetitec.com>
In #7497 port_background_task was renamed to port_background_tick
but the actual call site wasn't changed. This meant that it was
no longer called!
Rename more functions from task to tick to make it clearer which is
which.
In contrast to MidiTrack, this can be controlled from Python code,
turning notes on/off as desired.
Not tested on real HW yet, just the acceptance test based on checking
which notes it thinks are held internally.
This is intended to be used by the very outer caller of the VM/runtime. It
allows setting a top-level NLR handler that can be jumped to directly, in
order to forcefully abort the VM/runtime.
Enable using:
#define MICROPY_ENABLE_VM_ABORT (1)
Set up the handler at the top level using:
nlr_buf_t nlr;
nlr.ret_val = NULL;
if (nlr_push(&nlr) == 0) {
nlr_set_abort(&nlr);
// call into the VM/runtime
...
nlr_pop();
} else {
if (nlr.ret_val == NULL) {
// handle abort
...
} else {
// handle other exception that propagated to the top level
...
}
}
nlr_set_abort(NULL);
Schedule an abort, eg from an interrupt handler, using:
mp_sched_vm_abort();
Signed-off-by: Damien George <damien@micropython.org>
As the comment in py/obj.h says:
> Implementing this as a call rather than inline saves 8 bytes per usage.
So in order to get this savings, we need to tell the compiler to never
inline the function.
Signed-off-by: David Lechner <david@pybricks.com>
* Enable dcache for OCRAM where the VM heap lives.
* Add CIRCUITPY_SWO_TRACE for pushing program counters out over the
SWO pin via the ITM module in the CPU. Exempt some functions from
instrumentation to reduce traffic and allow inlining.
* Place more functions in ITCM to handle errors using code in RAM-only
and speed up CP.
* Use SET and CLEAR registers for digitalio. The SDK does read, mask
and write.
* Switch to 2MiB reserved for CircuitPython code. Up from 1MiB.
* Run USB interrupts during flash erase and write.
* Allow storage writes from CP if the USB drive is disabled.
* Get perf bench tests running on CircuitPython and increase timeouts
so it works when instrumentation is active.
Without this, building the unix port variants gives:
ports/unix/main.c:667: undefined reference to `mp_obj_is_package',
when MICROPY_ENABLE_EXTERNAL_IMPORT is 0.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
The C-level printf is usually used for internal debugging prints, and a
port/board may want to redirect this somewhere other than stdout.
Signed-off-by: Damien George <damien@micropython.org>
When := is used in a comprehension the target variable is bound to the
parent scope, so it's either a global or a nonlocal. Prior to this commit
that was handled by simply using the parent scope's id_info for the
target variable. That's completely wrong because it uses the slot number
for the parent's Python stack to store the variable, rather than the slot
number for the comprehension. This will in most cases lead to incorrect
behaviour or memory faults.
This commit fixes the scoping of the target variable by explicitly
declaring it a global or nonlocal, depending on whether the parent is the
global scope or not. Then the id_info of the comprehension can be used to
access the target variable. This fixes a lot of cases of using := in a
comprehension.
Code size change for this commit:
bare-arm: +0 +0.000%
minimal x86: +0 +0.000%
unix x64: +152 +0.019% standard
stm32: +96 +0.024% PYBV10
cc3200: +96 +0.052%
esp8266: +196 +0.028% GENERIC
esp32: +156 +0.010% GENERIC[incl +8(data)]
mimxrt: +96 +0.027% TEENSY40
renesas-ra: +88 +0.014% RA6M2_EK
nrf: +88 +0.048% pca10040
rp2: +104 +0.020% PICO
samd: +88 +0.033% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Fixes issue #10895.
Signed-off-by: Damien George <damien@micropython.org>
This is handy when you are doing builds outside of the Git repository but
still want to record that information.
Signed-off-by: David Grayson <davidegrayson@gmail.com>
Without this it's possible to get a compiler error about the comparison
always being true, because MP_BINARY_OP_LESS is 0. And it seems that gcc
optimises these 6 equality comparisons into the same size machine code as
before.
Prior to this fix, pow(1.5, inf) and pow(0.5, -inf) (among other things)
would incorrectly raise a ValueError, because the result is inf with the
first argument being finite. This commit fixes this by allowing the result
to be infinite if the first or second (or both) argument is infinite.
This fix doesn't affect the other three math functions that have two
arguments:
- atan2 never returns inf, so always fails isinf(ans)
- copysign returns inf only if the first argument x is inf, so will never
reach the isinf(y) check
- fmod never returns inf, so always fails isinf(ans)
Signed-off-by: Damien George <damien@micropython.org>
Only uppercase ASCII letters a-z. This saves ~900 bytes. Previously
written files with other unicode letters will only be accessible
from their upper cased path.
If an exception's chain or context can refer to a pointer from a different
VM, a crash would typically result.
This couldn't turn up on UNIX testing because the VM is never torn
down and rebuilt like it is on hardware.
Because in the 'static' case the GeneratorObject is now fully initialized
whenever it's raised, the initialization can be dropped, which reduces
the flash size slightly.
Closes: #7565
This ensures that all builds unconditionally run makeversionhdr.py and
makemanifest.py to generate mpversion.h and frozen_content.c respectively.
This now matches the Makefile behavior, and in particular this fixes the
issue on ESP32 builds that changes in code-to-be-frozen will cause the
build to update. Both these already tools know not to touch their output
if there is no change.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Unless MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D, these macros only work with
values and "->"/"." expressions as their sole argument. In other words,
the macros are broken with expressions which contain operations of lower
precedence than the cast operator.
Depending on situation, the old code either results in compiler error:
MP_OBJ_TO_PTR(flag ? o1 : o2) expands into "(void *)flag ? o1 : o2",
which some compiler configurations will reject (e.g. GCC -Wint-conversion
-Wint-to-pointer-cast -Werror)
Or in an incorrect address calculation:
For ptr declared as "uint8_t *" the MP_OBJ_FROM_PTR(ptr + off)
expands into ((mp_obj_t)ptr) + off, resulting in an obviously
wrong address.
Signed-off-by: Alex Riesen <alexander.riesen@cetitec.com>
This is important for literal tuples, e.g.
f"{a,b,}, {c}" --> "{}".format((a,b), (c),)
which would otherwise result in either a syntax error or the wrong result.
Fixes issue #9635.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
32-bit platforms only support a slice offset start of 24 bit max due to the
limited size of the mp_obj_array_t.free member. Similarly on 64-bit
platforms the limit is 56 bits.
This commit adds an OverflowError if the user attempts to slice a
memoryview beyond this limit.
Signed-off-by: Damien George <damien@micropython.org>
Showing 8 digits instead of 5, supporting devices with more than 1 MByte of
RAM (which is common these days). The masking was never needed, and the
related commented-out line can go.
To adhere to the contract of mp_map_lookup, namely:
MP_MAP_LOOKUP_ADD_IF_NOT_FOUND behaviour:
- returns slot, with key non-null and value=MP_OBJ_NULL if it was added
cpython actually makes sure the newly chained exception doesn't create
a cycle (even indirectly); see _PyErr_SetObject use of "Floyd's cycle
detection algo". We'll go for the simpler solution of just checking
one level deep until it's clear we need to do more.
Closes: #7414
This may help address #7409 if the underlying cause is the deterministic
volume ID. However, not all boards have working urandom (samd21
at least does not) so a couple of fallbacks are attempted when it fails.
I verified that on a pico_w, each `storage.erase_filesystem()` gives
a distinct 32-bit volume ID (pico_w's urandom can never fail)
In @micropython.native code the types of variables and expressions are
always Python objects, so they can be initialised as such. This prevents
problems with compiling optimised code like while-loops where a local may
be referenced before it is assigned to.
Signed-off-by: Damien George <damien@micropython.org>
This was previously used for the definition of NIC types, but they have
been updated to use a protocol instead.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The assertion that is added here (to gc.c) fails when running this new test
if ALLOC_TABLE_GAP_BYTE is set to 0.
Signed-off-by: Jeff Epler <jepler@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Prior to this fix the follow crash occurred. With a GC layout of:
GC layout:
alloc table at 0x3fd80428, length 32001 bytes, 128004 blocks
finaliser table at 0x3fd88129, length 16001 bytes, 128008 blocks
pool at 0x3fd8bfc0, length 2048064 bytes, 128004 blocks
Block 128003 is an AT_HEAD and eventually is passed to gc_mark_subtree.
This causes gc_mark_subtree to call ATB_GET_KIND(128004). When block 1 is
created with a finaliser, the first byte of the finaliser table becomes
0x2, but ATB_GET_KIND(128004) reads these bits as AT_TAIL, and then
gc_mark_subtree references past the end of the heap, which happened to be
past the end of PSRAM on the esp32-s2.
The fix in this commit is to ensure there is a one-byte gap after the ATB
filled permanently with AT_FREE.
Fixes issue #7116.
See also https://github.com/adafruit/circuitpython/issues/5021
Signed-off-by: Jeff Epler <jepler@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
When you want to use the valgrind memory analysis tool on MicroPython, you
can arrange to define MICROPY_DEBUG_VALGRIND to enable use of special
valgrind macros. For now, this only fixes `gc_get_ptr` so that it never
emits the diagnostic "Conditional jump or move depends on uninitialised
value(s)".
Signed-off-by: Jeff Epler <jepler@gmail.com>
This change makes it so the compiler and persistent code loader take a
mp_compiled_module_t* as their last argument, instead of returning this
struct. This eliminates a duplicate context variable for all callers of
these functions (because the context is now stored in the
mp_compiled_module_t by the caller), and also eliminates any confusion
about which context to use after the mp_compile_to_raw_code or
mp_raw_code_load function returns (because there is now only one context,
that stored in mp_compiled_module_t.context).
Reduces code size by 16 bytes on ARM Cortex-based ports.
Signed-off-by: Damien George <damien@micropython.org>
These unimplemented features may never be implemented, and having the word
"yet" there takes up space.
Signed-off-by: Damien George <damien@micropython.org>
This module is useful, but it is not always needed. Disabling it saves
several kilobytes of build size, depending on other config options.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
The conversion of characters like _space_ in qstrs is a bit
ad-hoc. Because "_not_" stands for the logical negation character ¬
the recently added message was displayed incorrectly:
```
>>> socket.getaddrinfo('does.not.exist', 0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
gaierror: (-2, 'Name or service_space¬space_known')
```
I had noticed this, but evidently failed to include the fix in
the problem in #7269.
MP_REGISTER_MODULE would use identifiers like
"MODULE_DEF_MP_QSTR___FUTURE__" which would in turn cause
a QSTR to be generated for it. This wasn't desirable, because the
qstr would never be used.
This clears out quite a bit of flash storage on the proxlight trinkey.
The code was already checking for duplicate kwargs for named parameters but
if `**kwargs` was given as a parameter, it did not check for multiples of
the same argument name.
This fixes the issue by adding an addition test to catch duplicates and
adds a test to exercise the code.
Fixes issue #10083.
Signed-off-by: David Lechner <david@pybricks.com>
Implements dictionary union according to PEP 584's specifications, minus
the fact that dictionary entries are not guaranteed to be in insertion
order. This feature is enabled with MICROPY_CPYTHON_COMPAT.
Includes a new test.
With the assistance of Fangrui Qin <qinf@purdue.edu>
Signed-off-by: Rayane Chatrieux <rayane.chatrieux@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
If a CMake-build is run with `make BUILD=/outside/path` then
makeversionheader.py is run with the CWD set to the build directory, which
means the git version lookup will fail and silently fall back to the
mpconfig.h mode (giving the wrong result).
This commit:
- Uses the location of makeversionheader.py to find the repo path.
- Allows overriding this path via --repo-path.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This can be tested using ports/minimal and qemu:
make CC=mips-linux-gnu-gcc-8
Then run with qemu-mips:
stty raw opost -echo;
QEMU_LD_PREFIX=/usr/mips-linux-gnu/ qemu-mips build/firmware.elf;
sleep 1; reset
Signed-off-by: Jan Willeke <willeke@smartmote.de>
This adds the __cause__, __context__ and __suppress_context__
members to exception objects and makes e.g., `raise exc from cause`
set them in the same way as standard Python.
.. a fast helper for animations. It is similar to and inspired by the
PixelMap helper in Adafruit LED Animation library, but with an extremely
fast 'paste' method for setting a series of pixels. This is a common
operation for many animations, and can give a substantial speed improvement.
It's named `adafruit_pixelmap` so that we can package a compatible version
in pure Python for systems that can't fit it in C in flash, or for
Blinka.
This is a proof of concept and can make a very fast comet animation:
```python
import time
import adafruit_pixelbuf
import adafruti_pixelmap
import board
import neopixel
from supervisor import ticks_ms
from adafruit_led_animation.animation.solid import Solid
from adafruit_led_animation import color
pixel_pin = board.GP0
pixel_num = 96
pixels = neopixel.NeoPixel(pixel_pin, pixel_num, brightness=1, auto_write=False, pixel_order="RGB")
evens = adafruit_pixelmap.PixelMap(pixels, tuple(range(0, pixel_num, 2)))
odd_indices = tuple((i, i+2) for i in range(1, pixel_num, 4))
print(odd_indices)
odds = adafruit_pixelbuf.PixelMap(pixels, odd_indices)
assert len(odds) == len(odd_indices)
comet_length = 16
comet1 = [color.calculate_intensity(color.GREEN, ((1+i) / comet_length) ** 2.4)
for i in range(comet_length)]
comet2 = [color.calculate_intensity(color.PURPLE, ((1+i) / comet_length) ** 2.4)
for i in range(comet_length)]
pos1 = 0
pos2 = 96//4
while True:
evens.paste(comet1, pos1, wrap=True, reverse=False, others=0)
pos1 = (pos1 + 1) % len(evens)
odds.paste(comet2, pos2, wrap=True, reverse=True, others=0)
pos2 = (pos2 - 1) % len(odds)
pixels.show()
m = ticks_ms()
if m % 2000 > 1000:
time.sleep(.02)
```
This prevents a very subtle bug caused by writing e.g. `bytearray('\xfd')`
which gives you `(0xc3, 0xbd)`.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
There are two calls to mp_builtin___import__():
1. ports/unix/main.c:main_() which provides a str in args[0]
2. py/runtime.c:mp_import_name() which provides a qstr in args[0]
The default implementation of mp_builtin___import__() is
mp_builtin___import___default() which has a different implementation based
on MICROPY_ENABLE_EXTERNAL_IMPORT.
If MICROPY_ENABLE_EXTERNAL_IMPORT is disabled then the handling of weak
links assumes that args[0] is a `const char *`, when it is either a str or
qstr object.
Use the existing qstr of the module name instead, and also use a vstr
instead of strcpy() to ensure no overflow occurs.
Commit 64af916c11 removed the version string
from docs/conf.py. py/mpconfig.h is a better place to get the version
from, so use that (when there is no git repository).
Signed-off-by: Damien George <damien@micropython.org>
In order for v1.19.1 to load a .mpy, the formerly-feature-flags which are
now used for the sub-version must be zero.
The sub-version is only used to indicate a native version change, so it
should be zero when emitting bytecode-only .mpy files.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prevents double-precision floats being enabled on 32-bit architectures
where they will not fit into the mp_obj_t encoding.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
We adopted the file "py/ioctl.h" and the ioctl names beginning
with MP_IOCTL_POLL while micropython went with "py/stream.h" and
MP_STREAM_POLL.
Align with upstream.
Closes#6711
Since there is only one flag, we don't need to use a bitfield in vstr_t.
Compilers emit extra instructions to access a bitfield, so this should
reduce the binary size a small amount.
Signed-off-by: David Lechner <david@pybricks.com>
This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc.
Make all extmod variables (src, flags, etc) private to extmod.mk.
Also move common/shared, extmod-related fragments (e.g. wiznet, cyw43,
bluetooth) into extmod.mk.
Now that SRC_MOD, CFLAGS_MOD, CXXFLAGS_MOD are unused by both extmod.mk
(and user-C-modules in a previous commit), remove all uses of them from
port makefiles.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Removes the need for the port to add anything to OBJS or SRC_QSTR.
Also makes it possible for user-C-modules to differentiate between code
that should be processed for QSTR vs other files (e.g. helpers and
libraries).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Only intended to be used on Unix and other "OS" ports. Matches CPython.
This should give the absolute path to the executing binary.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Note: at this time, the ssl module on pico_w never verifies the server
certificate. This means it does not actually provide a higher security
level than regular socket / http protocols.
My pings go out, and then they come back
```py
import os
import wifi
import ipaddress
wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
ipv4 = ipaddress.ip_address("8.8.4.4")
print("Ping google.com: %f ms" % (wifi.radio.ping(ipv4)*1000))
```
`b'\xaa \xaa'.count(b'\xaa')` now (correctly) returns 2 instead of 1.
Fixes issue #9404.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Allows optimisation of cases like:
import micropython
_DEBUG = micropython.const(False)
if _DEBUG:
print('Debugging info')
Previously the 'if' statement was only optimised out if the type of the
const() argument was integer.
The change is implemented in a way that makes the compiler slightly smaller
(-16 bytes on PYBV11) but compilation will also be very slightly slower.
As a bonus, if const support is enabled then the compiler can now optimise
const truthy/falsey expressions of other types, like:
while "something":
pass
... unclear if that is useful, but perhaps it could be.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This improves error messages in mpy-cross:
- When loading a .py file that doesn't exist (or can't be opened) it now
includes the filename in the OSError.
- When saving a .mpy file that can't be opened it now raises an exception
(prior, it would silently fail), and includes the filename in the
OSError.
Signed-off-by: Damien George <damien@micropython.org>
This matches class `__dict__`, and is similarly gated on
MICROPY_CPYTHON_COMPAT. Unlike class though, because modules's globals are
actually dict instances, the result is a mutable dictionary.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.
With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version. A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION. This allows 3
additional updates to the native ABI per bytecode revision.
The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
The check for make_new (i.e. used to determine something's type) is now
more complicated due to the slot access. This commit changes the inlining
of a few frequently-used helpers to overall improve code size and
performance.
Instead of being an explicit field, it's now a slot like all the other
methods.
This is a marginal code size improvement because most types have a make_new
(100/138 on PYBV11), however it improves consistency in how types are
declared, removing the special case for make_new.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.
This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
to a mp_getiter_iternext_custom_t instance, which then defines both
getiter and iternext.
And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.
Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Rather than reserving a full 12-slot mp_obj_type_t, reserve enough room for
seven and cast as necessary.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
In all cases other than where you have a native base with a protocol, it
now fits into 4 GC blocks (like it did before the slots representation).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The existings mp_obj_type_t uses a sparse representation for slots for the
capability methods of the type (eg print, make_new). This commit adds a
compact slot-index representation. The basic idea is that where the
mp_obj_type_t struct used to have 12 pointer fields, it now has 12 uint8_t
indices, and a variable-length array of pointers. So in the best case (no
fields used) it saves 12x4-12=36 bytes (on a 32-bit machine) and in the
common case (three fields used) it saves 9x4-12=24 bytes.
Overall with all associated changes, this slot-index representation reduces
code size by 1000 to 3000 bytes on bare-metal ports. Performance is
marginally better on a few tests (eg about 1% better on misc_pystone.py and
misc_raytrace.py on PYBv1.1), but overall marginally worse by a percent or
so.
See issue #7542 for further analysis and discussion.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This will always have the maximum/minimum size of a mp_obj_type_t
representation and can be used as a member in other structs.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This will allow the structure of mp_obj_type_t to change while keeping the
definition code the same.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The buffer protocol type only has a single member, and this existing layout
creates problems for the upcoming split/slot-index mp_obj_type_t layout
optimisations.
If we need to make the buffer protocol more sophisticated in the future
either we can rely on the mp_obj_type_t optimisations to just add
additional slots to mp_obj_type_t or re-visit the buffer protocol then.
This change is a no-op in terms of generated code.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Recently(?) github started making it the default to only copy a single
branch (e.g., main) and NO TAGS into new forks.
This makes the step of the build process that determines the CircuitPython
version not work, because tags are expected to be present. When tags are
not present, the version number is only a git hash. The version number
ends up being 0.0.0.
This causes problems with libraries that check for CircuitPython version
to determine compatibility, among other things.
We'll do other things to improve the situation, such as document it.
But it'd also be good if the build stopped when this detectable condition
occurs.
.. the default is intended to be the equivalent of the original,
implementing `DISPLAYIO && TERMINALIO`.
This is a possible alternative to #6889, if I understand the intent.
All uses of this are either tiny strings or not-known-to-be-safe.
Update comments for mp_obj_new_str_copy and mp_obj_new_str_of_type.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The new `mp_obj_new_str_from_utf8_vstr` can be used when you know you
already have a unicode-safe string.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Now that we have `mp_obj_new_str_type_from_vstr` (private helper used by
objstr.c) split from the public API (`mp_obj_new_str_from_vstr`), we can
enforce a unicode check at the public API without incurring a performance
cost on the various objstr.c methods (which are already working on known
unicode-safe strings).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previously the desired output type was specified. Now make the type part
of the function name. Because this function is used in a few places this
saves code size due to smaller call-site.
This makes `mp_obj_new_str_type_from_vstr` a private function of objstr.c
(which is almost the only place where the output type isn't a compile-time
constant).
This saves ~140 bytes on PYBV11.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows ports to override mp_builtin___import__.
This can be useful in MicroPython applications where
MICROPY_ENABLE_EXTERNAL_IMPORT has to be disabled due to its impact on
build size (2% to 2.5% of the minimal port). By overriding the otherwise
very minimal mp_builtin___import__, ports can still allow limited forms
of application-specific imports.
Signed-off-by: Laurens Valk <laurens@pybricks.com>
Since commit e65d1e69e8 there is no longer an
io.FileIO class, so this option is no longer needed.
This option also controlled whether or not files supported being opened in
binary mode (eg 'rb'), and could, if disabled, lead to confusion as to why
opening a file in binary mode silently did the wrong thing (it would just
open in text mode if MICROPY_PY_IO_FILEIO was disabled).
The various VFS implementations (POSIX, FAT, LFS) were the only places
where enabling this option made a difference, and in almost all cases where
one of these filesystems were enabled, MICROPY_PY_IO_FILEIO was also
enabled. So it makes sense to just unconditionally enable this feature
(ability to open a file in binary mode) in all cases, and so just remove
this config option altogether. That makes configuration simpler and means
binary file support always exists (and opening a file in binary mode is
arguably more fundamental than opening in text mode, so if anything should
be configurable then it should be the ability to open in text mode).
Signed-off-by: Damien George <damien@micropython.org>
Rework the conversion of floats to decimal strings so it aligns precisely
with the conversion of strings to floats in parsenum.c. This is to avoid
rendering 1eX as 9.99999eX-1 etc. This is achieved by removing the power-
of-10 tables and using pow() to compute the exponent directly, and that's
done efficiently by first estimating the power-of-10 exponent from the
power-of-2 exponent in the floating-point representation.
Code size is reduced by roughly 100 to 200 bytes by this commit.
Signed-off-by: Dan Ellis <dan.ellis@gmail.com>
Prior to this commit, parsenum would calculate "1e-20" as 1.0*pow(10, -20),
and "1.000e-20" as 1000.0*pow(10, -23); in certain cases, this could make
seemingly-identical values compare as not equal. This commit watches for
trailing zeros as a special case, and ignores them when appropriate, so
"1.000e-20" is also calculated as 1.0*pow(10, -20).
Fixes issue #5831.
Otherwise if the `mpy-cross/build/` directory doesn't exist then
`mpy-cross/build/..` won't work.
Signed-off-by: Damien George <damien@micropython.org>
Since f7f56d4285 consolidated all uses of
these to a single locals dict, they no longer need to be made public.
Signed-off-by: Damien George <damien@micropython.org>
These were added in Python 3.5.
Enabled via MICROPY_PY_BUILTINS_BYTES_HEX, and enabled by default for all
ports that currently have ubinascii.
Rework ubinascii to use the implementation of these methods.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit adds the bytes methods to bytearray, matching CPython. The
existing implementations of these methods for str/bytes are reused for
bytearray with minor updates to match CPython return types.
For details on the CPython behaviour see
https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations
The work to merge locals tables for str/bytes/bytearray/array was done by
@jimmo. Because of this merging of locals the change in code size for this
commit is mostly negative:
bare-arm: +0 +0.000%
minimal x86: +29 +0.018%
unix x64: -792 -0.128% standard[incl -448(data)]
unix nanbox: -436 -0.078% nanbox[incl -448(data)]
stm32: -40 -0.010% PYBV10
cc3200: -32 -0.017%
esp8266: -28 -0.004% GENERIC
esp32: -72 -0.005% GENERIC[incl -200(data)]
mimxrt: -40 -0.011% TEENSY40
renesas-ra: -40 -0.006% RA6M2_EK
nrf: -16 -0.009% pca10040
rp2: -64 -0.013% PICO
samd: +148 +0.105% ADAFRUIT_ITSYBITSY_M4_EXPRESS
The hash is either 8 or 16 bits (depending on MICROPY_QSTR_BYTES_IN_HASH)
so will fit in a size_t.
This saves 268 bytes on the unix nanbox build. Non-nanbox configurations
are unchanged because mp_uint_t is the same size as size_t.
Signed-off-by: Damien George <damien@micropython.org>
Due to inline assembly, wrong instructions were generated. Use
corresponding 32 bit instructions and fix the offsets used.
Signed-off-by: Efi Weiss <efiwiss@gmail.com>
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory. Thus
some paths had to be adjusted.
The rules for lib (static library with name $(LIBMICROPYTHON)) and the
default rule to build a binary (name $(PROG)) produced outputs in the
current working directory. Change this to build these files in the build
directory.
Note: An empty BUILD variable can cause issues (references to the root
directory); this is not addressed by this commit due to multiple other
places having the same issue.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
The reallocation trigger for unpacking star args with unknown length
did not take into account the number of fixed args remaining. So it was
possible that the unpacked iterators could take up exactly the memory
allocated then nothing would be left for fixed args after the star args.
This causes a segfault crash.
This is fixed by taking into account the remaining number of fixed args
in the check to decide whether to realloc yet or not.
Signed-off-by: David Lechner <david@pybricks.com>
* Tweak scroll area position so last line is complete and top is
under the title bar.
* Pick Blinka size based on the font to minimize unused space in
title bar. Related to #2791
* Update the title bar after terminal is started. Fixes#6078Fixes#6668
This uses the esp32-camera code instead of our own homebrewed camera code.
In theory it supports esp32, esp32-s2 and esp32-s3, as long as they have
PSRAM.
This is very basic and doesn't support changing any camera parameters,
including switching resolution or pixelformat.
This is tested on the Kaluga (ESP32-S2) and ESP32-S3-Eye boards.
First, reserve some PSRAM by putting this line in `CIRCUITPY/_env`:
```
CIRCUITPY_RESERVED_PSRAM=524288
```
and hard-reset the board for it to take effect.
Now, the following script will take a very low-resolution jpeg file and print
it in the REPL in escape coded form:
```python
import board
import esp32_camera
c = esp32_camera.Camera(
data_pins=board.CAMERA_DATA,
external_clock_pin=board.CAMERA_XCLK,
pixel_clock_pin=board.CAMERA_PCLK,
vsync_pin=board.CAMERA_VSYNC,
href_pin=board.CAMERA_HREF,
pixel_format=esp32_camera.PixelFormat.JPEG,
i2c=board.I2C(),
external_clock_frequency=20_000_000)
m = c.take()
if m is not None:
print(bytes(m))
```
Then on desktop open a python repl and run something like
```python
>>> with open("my.jpg", "wb") as f: f.write(<BIG PASTE FROM REPL>)
```
and open my.jpg in a viewer.
.. the primary user of which will be the camera, since the framebuffers
must be allocated via esp-idf allocation function and never from the
gc heap.
A board can have a default value, and the value can also be set in the
/.env file using the key CIRCUITPY_RESERVED_PSRAM with the value being
the reserved size in bytes.
Co-authored-by: Dan Halbert <halbert@adafruit.com>
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
On ports with more than one filesystem, the type will be wrong, for example
if using LFS but FAT enabled, then the type will be FAT. So it's not
possible to use these classes to identify a file object type.
Furthermore, constructing an io.FileIO currently crashes on FAT, and
make_new isn't supported on LFS.
And the io.TextIOWrapper class does not match CPython at all.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit simplifies mp_obj_get_complex_maybe() by first calling
mp_obj_get_float_maybe() to handle the cases corresponding to floats.
Only if that fails does it attempt to extra a full complex number.
This reduces code size and also means that mp_obj_get_complex_maybe() now
supports user-defined classes defining __float__; in particular this allows
user-defined classes to be used as arguments to cmath-module function.
Furthermore, complex_make_new() can now be simplified to directly call
mp_obj_get_complex(), instead of mp_obj_get_complex_maybe() followed by
mp_obj_get_float(). This also improves error messages from complex with
an invalid argument, it now raises "can't convert <type> to complex" rather
than "can't convert <type> to float".
Signed-off-by: Damien George <damien@micropython.org>
Use C macros to reduce the size of firmware images when the GC split-heap
feature is disabled.
The code size difference of this commit versus HEAD~2 (ie the commit prior
to MICROPY_GC_SPLIT_HEAP being introduced) when split-heap is disabled is:
bare-arm: +0 +0.000%
minimal x86: +0 +0.000%
unix x64: -16 -0.003%
unix nanbox: -20 -0.004%
stm32: -8 -0.002% PYBV10
cc3200: +0 +0.000%
esp8266: +8 +0.001% GENERIC
esp32: +0 +0.000% GENERIC
nrf: -20 -0.011% pca10040
rp2: +0 +0.000% PICO
samd: -4 -0.003% ADAFRUIT_ITSYBITSY_M4_EXPRESS
The code size difference of this commit versus HEAD~2 split-heap is enabled
with MICROPY_GC_MULTIHEAP=1 (but no extra code to add more heaps):
unix x64: +1032 +0.197% [incl +544(bss)]
esp32: +592 +0.039% GENERIC[incl +16(data) +264(bss)]
This commit adds a new option MICROPY_GC_SPLIT_HEAP (disabled by default)
which, when enabled, allows the GC heap to be split over multiple memory
areas/regions. The first area is added with gc_init() and subsequent areas
can be added with gc_add(). New areas can be added at runtime. Areas are
stored internally as a linked list, and calls to gc_alloc() can be
satisfied from any area.
This feature has the following use-cases (among others):
- The ESP32 has a fragmented OS heap, so to use all (or more) of it the
GC heap must be split.
- Other MCUs may have disjoint RAM regions and are now able to use them
all for the GC heap.
- The user could explicitly increase the size of the GC heap.
- Support a dynamic heap while running on an OS, adding more heap when
necessary.
This uses MP_REGISTER_ROOT_POINTER() to register sched_queue
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register cur_exception,
sys_exitfunc, mp_sys_path_obj, mp_sys_argv_obj and sys_mutable
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register track_reloc_code_list
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register `bluetooth`
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register vfs_cur and
vfs_mount_table instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register lwip_slip_stream
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register dupterm_objs
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
This uses MP_REGISTER_ROOT_POINTER() to register repl_line
instead of using a conditional inside of mp_state_vm_t.
Signed-off-by: David Lechner <david@pybricks.com>
All in-tree uses of MICROPY_PORT_ROOT_POINTERS have been replaced with
MP_REGISTER_ROOT_POINTER(), so now we can remove both
MICROPY_PORT_ROOT_POINTERS and MICROPY_BOARD_ROOT_POINTERS from the code
and remaining config files.
Signed-off-by: David Lechner <david@pybricks.com>
