Commit Graph

3805 Commits

Author SHA1 Message Date
Damien George
c47a3ddf4a py/pairheap: Properly unlink node on pop and delete.
This fixes a bug in the pairing-heap implementation when nodes are deleted
with mp_pairheap_delete and then reinserted later on.
2020-03-26 01:21:04 +11:00
Damien George
f8fc78691d py/mpconfig.h: Enable MICROPY_MODULE_GETATTR by default.
To enable lazy loading of submodules (among other things), which is very
useful for MicroPython libraries that want to have optional subcomponents.

Disabled explicitly on minimal ports.
2020-03-26 01:21:04 +11:00
David Lechner
2f7d2bb3e2 py/stream.h: Include sys/types.h to get size_t and off_t for POSIX API. 2020-03-25 01:00:52 +11:00
Damien George
feb2577585 all: Remove spaces between nested paren and inside function arg paren.
Using new options enabled in the uncrustify configuration.
2020-03-25 00:39:46 +11:00
Damien George
ad9a0ec8ab all: Convert exceptions to use mp_raise_XXX helpers in remaining places. 2020-03-18 17:26:19 +11:00
Andrew Leech
86bfabec11 py/modmicropython: Add heap_locked function to test state of heap.
This commit adds micropython.heap_locked() which returns the current
lock-depth of the heap, and can be used by Python code to check if the heap
is locked or not.  This new function is configured via
MICROPY_PY_MICROPYTHON_HEAP_LOCKED and is disabled by default.

This commit also changes the return value of micropython.heap_unlock() so
it returns the current lock-depth as well.
2020-03-11 16:54:16 +11:00
Andrew Leech
ed93778e00 py/objstringio: Expose tell() on StringIO and BytesIO objects.
To match file objects.

Fixes issue #5581.
2020-03-11 14:43:03 +11:00
David Lechner
8a4ce6b79a tools/codeformat.py: Eliminate need for sizeof fixup.
This eliminates the need for the sizeof regex fixup by rearranging things a
bit.  All other bitfields already use the parentheses around expressions
with sizeof, so one case is fixed by following this convention.

VM_MAX_STATE_ON_STACK is the only remaining problem and it can be worked
around by changing the order of the operands.
2020-03-11 14:34:40 +11:00
Tom Collins
fccf17521a py/objstr: Remove duplicate % in error string.
The double-% was added in 11de8399fe (Jun
2014) when such errors were formatted with printf.  But then
55830dd9bf (Dec 2018) changed
mp_obj_new_exception_msg() to not format the message, as discussed
in #3004.  So such error strings are no longer formatted and a % is just
that.
2020-03-11 14:31:29 +11:00
Damien George
1993c8cf9a py/builtinevex: Support passing in a bytearray/buffer to eval/exec.
CPython allows this and it's a simple generalisation of the existing code
which just supported str/bytes.

Fixes issue #5704.
2020-02-28 12:45:36 +11:00
Damien George
69661f3343 all: Reformat C and Python source code with tools/codeformat.py.
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-28 10:33:03 +11:00
Damien George
3f39d18c2b all: Add *FORMAT-OFF* in various places.
This string is recognised by uncrustify, to disable formatting in the
region marked by these comments.  This is necessary in the qstrdef*.h files
to prevent modification of the strings within the Q(...).  In other places
it is used to prevent excessive reformatting that would make the code less
readable.
2020-02-28 10:31:07 +11:00
Damien George
b86075ef1f py/parse: Add parenthesis around calculated bit-width in struct.
To improve interaction with uncrustify formatter.
2020-02-28 10:30:52 +11:00
Damien George
17aeb43e18 py: Un-nest configuration #if/#endif's for selection of complex code.
Because un-nested #if's are simpler to handle with formatting tools.
2020-02-28 10:30:34 +11:00
Damien George
7768a8c38d py/malloc: Put { on separate line for funcs that have selective sigs.
To ensure there are balanced {}'s in the file, and to help with formatting.
2020-02-28 10:30:29 +11:00
Damien George
a642241a12 py/builtinimport: Adjust if-block order in find_file to clean up #if's. 2020-02-28 10:30:20 +11:00
Damien George
2b50afaee4 py/bc0.h: Shift comment to start of line to improve format consistency. 2020-02-28 10:29:32 +11:00
Damien George
a1b18b3ba7 py: Removing dangling "else" to improve code format consistency. 2020-02-28 10:29:27 +11:00
Jim Mussared
3ccce89b83 py/objarray: Turn on MP_TYPE_FLAG_EQ_CHECKS_OTHER_TYPE for memoryview.
And add corresponding tests.

Fixes #5674 (comparison of memoryview against bytes).
2020-02-21 14:24:07 +11:00
Damien George
f6d99bc795 py/dynruntime.h: Add implementation of mp_obj_cast_to_native_base. 2020-02-21 13:32:48 +11:00
Damien George
9344e876bb py/objtype: Allow mp_instance_cast_to_native_base to take native obj.
And rename it to mp_obj_cast_to_native_base() to indicate this.  This
allows users of this function to easily support native and native-subclass
objects in the same way (by just passing the object through this function).
2020-02-21 13:20:12 +11:00
Damien George
d3b2c6e44c py/objtuple: Remove code that handles tuple-subclass equality test.
Since commit 3aab54bf43 this piece of code is
no longer needed because the top-level function mp_obj_equal_not_equal()
now handles the case of user types, and will never call tuple's binary_op
function with MP_BINARY_OP_EQUAL and a non-tuple on the RHS.
2020-02-20 10:48:03 +11:00
Damien George
1fccda049f py/objexcept: Rename mp_obj_new_exception_msg_varg2 to ..._vlist.
Follow up to recent commit ad7213d3c3, the
name "varg2" is misleading, vlist describes better that the argument is a
va_list.  This name also matches CircuitPython, which already has such
helper functions.
2020-02-18 21:00:42 +11:00
Damien George
ce39c958ef py: Factor out definition of mp_float_union_t to one location. 2020-02-18 13:04:36 +11:00
David Lechner
f020eac6a8 py/obj.h: Remove TODO idea comment about truncated mp_map_t.
It was suggested to move this to a GitHub issue rather than keep it in the
code, which isn't really sustainable for all ideas.
2020-02-13 13:14:17 +11:00
Damien George
ad7213d3c3 py: Add mp_raise_msg_varg helper and use it where appropriate.
This commit adds mp_raise_msg_varg(type, fmt, ...) as a helper for
nlr_raise(mp_obj_new_exception_msg_varg(type, fmt, ...)).  It makes the
C-level API for raising exceptions more consistent, and reduces code size
on most ports:

   bare-arm:   +28 +0.042%
minimal x86:  +100 +0.067%
   unix x64:   -56 -0.011%
unix nanbox:  -300 -0.068%
      stm32:  -204 -0.054% PYBV10
     cc3200:    +0 +0.000%
    esp8266:   -64 -0.010% GENERIC
      esp32:  -104 -0.007% GENERIC
        nrf:  -136 -0.094% pca10040
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2020-02-13 11:52:40 +11:00
Damien George
97eca38c4f py: Add mp_raise_type helper macro and use it where appropriate.
This provides a more consistent C-level API to raise exceptions, ie moving
away from nlr_raise towards mp_raise_XXX.  It also reduces code size by a
small amount on some ports.
2020-02-13 11:03:37 +11:00
Damien George
7679e3be96 py/objmodule.h: Remove obsolete mp_builtin_module_weak_links_map decl.
It was made obsolete in d2384efa80
2020-02-11 15:43:13 +11:00
Damien George
9ec1caf42e py: Expand type equality flags to 3 separate ones, fix bool/namedtuple.
Both bool and namedtuple will check against other types for equality; int,
float and complex for bool, and tuple for namedtuple.  So to make them work
after the recent commit 3aab54bf43 they would
need MP_TYPE_FLAG_NEEDS_FULL_EQ_TEST set.  But that makes all bool and
namedtuple equality checks less efficient because mp_obj_equal_not_equal()
could no longer short-cut x==x, and would need to try __ne__.  To improve
this, this commit splits the MP_TYPE_FLAG_NEEDS_FULL_EQ_TEST flags into 3
separate flags to give types more fine-grained control over how their
equality behaves.  These new flags are then used to fix bool and namedtuple
equality.

Fixes issue #5615 and #5620.
2020-02-11 11:06:00 +11:00
Damien George
abe2caf6df py/scheduler: Move clearing of kbd traceback to mp_keyboard_interrupt.
This is a more logical place to clear the KeyboardInterrupt traceback,
right before it is set as a pending exception.  The clearing is also
optimised from a function call to a simple store of NULL.
2020-02-07 16:08:31 +11:00
Damien George
8fb5c8fdd5 py/scheduler: Allow a port to specify attrs for mp_keyboard_interrupt.
Functions like mp_keyboard_interrupt() may need to be called from an IRQ
handler and may need to be in a special memory section, so provide a
generic wrapping macro for a port to do this.  The macro name is chosen to
be MICROPY_WRAP_<function name in uppercase> so that (in the future with
more wrappers) each function could potentially be handled separately.
2020-02-07 16:08:29 +11:00
Damien George
9efb36bfa6 py/scheduler: Move mp_keyboard_interrupt from lib/utils to py core.
This function is tightly coupled to the state and behaviour of the
scheduler, and is a core part of the runtime: to schedule a pending
exception.  So move it there.
2020-02-07 16:08:26 +11:00
Damien George
98a3911c43 py/scheduler: Add "raise_exc" argument to mp_handle_pending.
Previous behaviour is when this argument is set to "true", in which case
the function will raise any pending exception.  Setting it to "false" will
cancel any pending exception.
2020-02-07 16:08:20 +11:00
Petr Viktorin
e6c9800645 py/compile: Allow 'return' outside function in minimal builds.
A 'return' statement on module/class level is not correct Python, but
nothing terribly bad happens when it's allowed.  So remove the check unless
MICROPY_CPYTHON_COMPAT is on.

This is similar to MicroPython's treatment of 'import *' in functions
(except 'return' has unsurprising behavior if it's allowed).
2020-02-06 00:41:55 +11:00
David Lechner
3e1bbeabaf py/modthread: Fix spelling error in comment. 2020-01-31 23:57:25 +11:00
Nicko van Someren
3aab54bf43 py: Support non-boolean results for equality and inequality tests.
This commit implements a more complete replication of CPython's behaviour
for equality and inequality testing of objects.  This addresses the issues
discussed in #5382 and a few other inconsistencies.  Improvements over the
old code include:

- Support for returning non-boolean results from comparisons (as used by
  numpy and others).
- Support for non-reflexive equality tests.
- Preferential use of __ne__ methods and MP_BINARY_OP_NOT_EQUAL binary
  operators for inequality tests, when available.
- Fallback to op2 == op1 or op2 != op1 when op1 does not implement the
  (in)equality operators.

The scheme here makes use of a new flag, MP_TYPE_FLAG_NEEDS_FULL_EQ_TEST,
in the flags word of mp_obj_type_t to indicate if various shortcuts can or
cannot be used when performing equality and inequality tests.  Currently
four built-in classes have the flag set: float and complex are
non-reflexive (since nan != nan) while bytearray and frozenszet instances
can equal other builtin class instances (bytes and set respectively).  The
flag is also set for any new class defined by the user.

This commit also includes a more comprehensive set of tests for the
behaviour of (in)equality operators implemented in special methods.
2020-01-30 14:53:07 +11:00
Damien George
c3450effd4 py/objtype: Make mp_obj_type_t.flags constants public, moved to obj.h. 2020-01-30 14:53:07 +11:00
David Lechner
b72cb0ca1b py/mpthread.h: Use strong type for mp_thread_set_state() argument.
This modifies the signature of mp_thread_set_state() to use
mp_state_thread_t* instead of void*.  This matches the return type of
mp_thread_get_state(), which returns the same value.

`struct _mp_state_thread_t;` had to be moved before
`#include <mpthreadport.h>` since the stm32 port uses it in its
mpthreadport.h file.
2020-01-29 17:10:32 +11:00
Jim Mussared
c3095b37e9 py/nativeglue: Fix typo about where the native fun table enum is. 2020-01-27 13:22:19 +11:00
Jim Mussared
888ddb81dd py/emitnative: Stop after finding an unwind target.
The loop searches backwards for a target, but doesn't stop after finding
the first result, meaning that it'll always end up at the outermost
exception handler.
2020-01-27 13:22:03 +11:00
Jim Mussared
0de304e7da py/emitnative: Use NULL for pending exception (not None).
This previously made the native emitter incompatible with the bytecode
emitter, and mp_resume (and subsequently mp_obj_generator_resume) expects
the bytecode emitter behavior (i.e. throw==NULL).
2020-01-27 13:21:49 +11:00
David Lechner
62537a18e3 py: Release GIL during syscalls in reader and writer code.
This releases the GIL during POSIX system calls that could block.
2020-01-26 23:26:31 +11:00
Yonatan Goldschmidt
d9433d3e94 py/obj.h: Add and use mp_obj_is_bool() helper.
Commit d96cfd13e3 introduced a regression in
testing for bool objects, that such objects were in some cases no longer
recognised and bools, eg when using mp_obj_is_type(o, &mp_type_bool), or
mp_obj_is_integer(o).

This commit fixes that problem by adding mp_obj_is_bool(o).  Builds with
MICROPY_OBJ_IMMEDIATE_OBJS enabled check if the object is any of the const
True or False objects.  Builds without it use the old method of ->type
checking, which compiles to smaller code (compared with the former
mentioned method).

Fixes #5538.
2020-01-24 10:53:45 +11:00
Damien George
e2c1226da4 py/objexcept: Optimise mp_obj_new_exception[_arg1/_args] functions.
This reduces code size by 10-70 bytes on all ports (except cc3200 which has
no change).
2020-01-23 13:37:25 +11:00
Damien George
edc7a8bf1d py/objgenerator: Use mp_obj_new_exception_arg1 to make StopIteration. 2020-01-23 13:37:25 +11:00
David Lechner
edbb73a411 py/qstr: Don't include or init qstr_mutex when GIL is enabled.
When threads and the GIL are enabled, then the qstr mutex is not needed.
The qstr_mutex field is never used in this case because of:

    #if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL
    #define QSTR_ENTER() mp_thread_mutex_lock(&MP_STATE_VM(qstr_mutex), 1)
    #define QSTR_EXIT() mp_thread_mutex_unlock(&MP_STATE_VM(qstr_mutex))
    #else
    #define QSTR_ENTER()
    #define QSTR_EXIT()
    #endif

So, we can completely remove qstr_mutex everywhere when MICROPY_PY_THREAD
&& !MICROPY_PY_THREAD_GIL.
2020-01-23 13:29:11 +11:00
David Lechner
ccc18f047d py/gc: Don't include or init gc_mutex when GIL is enabled.
When threads and the GIL are enabled, then the GC mutex is not needed.  The
gc_mutex field is never used in this case because of:

    #if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL
    #define GC_ENTER() mp_thread_mutex_lock(&MP_STATE_MEM(gc_mutex), 1)
    #define GC_EXIT() mp_thread_mutex_unlock(&MP_STATE_MEM(gc_mutex))
    #else
    #define GC_ENTER()
    #define GC_EXIT()
    #endif

So, we can completely remove gc_mutex everywhere when MICROPY_PY_THREAD
&& !MICROPY_PY_THREAD_GIL.
2020-01-23 13:28:42 +11:00
Damien George
fe203bb3e2 py/pairheap: Add generic implementation of pairing heap data structure. 2020-01-22 17:31:18 +11:00
Yonatan Goldschmidt
176ab99180 py/objint: Add mp_obj_int_get_uint_checked() helper.
Can be used where mp_obj_int_get_checked() will overflow due to the
sign-bit solely.  This returns an mp_uint_t, so it also verifies the given
integer is not negative.

Currently implemented only for mpz configurations.
2020-01-14 23:35:22 +11:00
Yonatan Goldschmidt
1c849d63a8 py/mpconfig.h: Define BITS_PER_BYTE only if not already defined.
It's a common macro that is possibly defined in headers of systems/SDKs
MicroPython is embedded into.
2020-01-12 20:57:01 +02:00
Damien George
40057600b8 py/obj: Optimise mp_obj_get_type for immediate objs with repr A and C.
This function is called often and with immediate objects enabled it has
more cases, so optimise it for speed.  With this optimisation the runtime
is now slightly faster with immediate objects enabled than with them
disabled.
2020-01-13 01:01:45 +11:00
Damien George
d96cfd13e3 py/obj: Add MICROPY_OBJ_IMMEDIATE_OBJS option to reduce code size.
This option (enabled by default for object representation A, B, C) makes
None/False/True objects immediate objects, ie they are no longer a concrete
object in ROM but are rather just values, eg None=0x6 for representation A.

Doing this saves a considerable amount of code size, due to these objects
being widely used:

   bare-arm:  -392 -0.591%
minimal x86:  -252 -0.170% [incl +52(data)]
   unix x64:  -624 -0.125% [incl -128(data)]
unix nanbox:    +0 +0.000%
      stm32: -1940 -0.510% PYBV10
     cc3200: -1216 -0.659%
    esp8266:  -404 -0.062% GENERIC
      esp32:  -732 -0.064% GENERIC[incl +48(data)]
        nrf:  -988 -0.675% pca10040
       samd:  -564 -0.556% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Thanks go to @Jongy aka Yonatan Goldschmidt for the idea.
2020-01-13 01:01:45 +11:00
Damien George
6f0c83f6e1 py/obj.h: Redefine qstr object encoding to add immediate obj encoding.
This commit adjusts the definition of qstr encoding in all object
representations by taking a single bit from the qstr space and using it to
distinguish between qstrs and a new kind of literal object: immediate
objects.  In other words, the qstr space is divided in two pieces, one half
for qstrs and the other half for immediate objects.

There is still enough room for qstr values (29 bits in representation A on
a 32-bit architecture, and 19 bits in representation C) and the new
immediate objects can be used for things like None, False and True.
2020-01-13 01:01:45 +11:00
Damien George
ecdb30ea64 py/nativeglue: Use mp_const_X instead of &mp_const_X_obj. 2020-01-12 13:51:09 +11:00
David Lechner
339d0816c5 py/runtime: Move MICROPY_PORT_INIT_FUNC near the end of mp_init().
This moves the MICROPY_PORT_INIT_FUNC hook to the end of mp_init(), just
before MP_THREAD_GIL_ENTER(), so that everything (in particular the GIL
mutex) is intialized before the hook is called.  MICROPY_PORT_DEINIT_FUNC
is also moved to be symmetric (but there is no functional change there).

If a port needs to perform initialisation earlier than
MICROPY_PORT_INIT_FUNC then it can do it before calling mp_init().
2020-01-12 13:27:04 +11:00
Yonatan Goldschmidt
853aaa06f2 lib/mp-readline: Add word-based move/delete EMACS key sequences.
This commit adds backward-word, backward-kill-word, forward-word,
forward-kill-word sequences for the REPL, with bindings to Alt+F, Alt+B,
Alt+D and Alt+Backspace respectively.  It is disabled by default and can be
enabled via MICROPY_REPL_EMACS_WORDS_MOVE.

Further enabling MICROPY_REPL_EMACS_EXTRA_WORDS_MOVE adds extra bindings
for these new sequences: Ctrl+Right, Ctrl+Left and Ctrl+W.

The features are enabled on unix micropython-coverage and micropython-dev.
2020-01-12 13:09:27 +11:00
Yonatan Goldschmidt
df5c3bd976 py/unicode: Add unichar_isalnum(). 2020-01-12 13:03:57 +11:00
Jim Mussared
7d2ccd027f py/mkenv.mk: Move usage of 32-bit flags to py.mk.
This allows ports/variants to configure MICROPY_FORCE_32BIT after including
mkenv.mk, but before py.mk.
2020-01-12 10:34:10 +11:00
Damien George
bfbd94401d py: Make mp_obj_get_type() return a const ptr to mp_obj_type_t.
Most types are in rodata/ROM, and mp_obj_base_t.type is a constant pointer,
so enforce this const-ness throughout the code base.  If a type ever needs
to be modified (eg a user type) then a simple cast can be used.
2020-01-09 11:25:26 +11:00
Nicko van Someren
10709846f3 py/objslice: Inline fetching of slice paramters in str_subscr().
To reduce code size.
2019-12-29 00:06:02 +11:00
Nicko van Someren
4c93955b7b py/objslice: Add support for indices() method on slice objects.
Instances of the slice class are passed to __getitem__() on objects when
the user indexes them with a slice.  In practice the majority of the time
(other than passing it on untouched) is to work out what the slice means in
the context of an array dimension of a particular length.  Since Python 2.3
there has been a method on the slice class, indices(), that takes a
dimension length and returns the real start, stop and step, accounting for
missing or negative values in the slice spec.  This commit implements such
a indices() method on the slice class.

It is configurable at compile-time via MICROPY_PY_BUILTINS_SLICE_INDICES,
disabled by default, enabled on unix, stm32 and esp32 ports.

This commit also adds new tests for slice indices and for slicing unicode
strings.
2019-12-28 23:55:15 +11:00
Damien George
b23bd6433c py: Clean up commented-out code and comments about exception hierarchy.
In CPython, EnvironmentError and IOError are now aliases of OSError so no
need to have them listed in the code.  OverflowError inherits from
ArithmeticError because it's intended to be raised "when the result of an
arithmetic operation is too large to be represented" (per CPython docs),
and MicroPython aims to match the CPython exception hierarchy.
2019-12-28 01:01:36 +11:00
Yonatan Goldschmidt
61d2b40ad5 lib/utils/pyexec: Introduce MICROPY_REPL_INFO, wrap debug prints in it.
For the 3 ports that already make use of this feature (stm32, nrf and
teensy) this doesn't make any difference, it just allows to disable it from
now on.

For other ports that use pyexec, this decreases code size because the debug
printing code is dead (it can't be enabled) but the compiler can't deduce
that, so code is still emitted.
2019-12-28 00:05:39 +11:00
Damien George
d56bc6e03d py/obj.h: Use 32-bit shift in MP_OBJ_NEW_QSTR calc for obj-repr D.
The qst value is always small enough to fit in 31-bits (even less) and
using a 32-bit shift rather than a 64-bit shift reduces code size by about
300 bytes.
2019-12-27 23:33:34 +11:00
Damien George
4c0176d13f py/objstr: Don't use inline GET_STR_DATA_LEN for object-repr D.
Changing to use the helper function mp_obj_str_get_data_no_check() reduces
code size of nan-boxing builds by about 1000 bytes.
2019-12-27 23:15:52 +11:00
Damien George
1f499ad2fe py/objobject: Fix __setattr__/__delattr__ to build in nanbox mode. 2019-12-27 22:54:53 +11:00
Damien George
d97b40bdaa py: Introduce MP_ROM_FALSE/MP_ROM_TRUE for ROM to refer to bool objects.
This helps to prevent mistakes, and allows easily changing the ROM value of
False/True if needed.
2019-12-27 22:54:20 +11:00
Damien George
09376f0e47 py: Introduce MP_ROM_NONE macro for ROM to refer to None object.
This helps to prevent mistakes, and allows easily changing the ROM value of
None if needed.
2019-12-27 22:51:17 +11:00
Damien George
f5eec903fa py/objsingleton: Use mp_generic_unary_op for singleton objects.
So these types more closely match NoneType, eg they can be hashed, like in
CPython.
2019-12-27 12:53:36 +11:00
Damien George
aacd618939 py/runtime: Don't allocate iter buf for user-defined types.
A user-defined type that defines __iter__ doesn't need any memory to be
pre-allocated for its iterator (because it can't use such memory).  So
optimise for this case by not allocating the iter-buf.
2019-12-27 12:34:22 +11:00
Damien George
ab75210e33 py/asmx86: Remove unused 5th argument facility.
In commit 71a3d6ec3b mp_setup_code_state was
changed from a 5-arg function to a 4-arg function, and at that point 5-arg
calls in native code were no longer needed.  See also commit
4f9842ad80.
2019-12-27 12:30:51 +11:00
Damien George
b97fb683d0 py/asmx86: Fix stack to be 16-byte aligned for entry and sub-call. 2019-12-27 12:30:51 +11:00
Damien George
300eb65ae7 py/nlrx86: Silence possible warnings about unused nlr argument. 2019-12-23 00:07:03 +11:00
Yonatan Goldschmidt
42e45bd694 py/objobject: Add object.__delattr__ function.
Similar to object.__setattr__.
2019-12-21 00:14:22 +11:00
Yonatan Goldschmidt
07ccb5588c py/objobject: Add object.__setattr__ function.
Allows assigning attributes on class instances that implement their own
__setattr__.  Both object.__setattr__ and super(A, b).__setattr__ will work
with this commit.
2019-12-21 00:12:08 +11:00
Damien George
5e431188db py/obj.h: Remove comments about additional mp_buffer_info_t entries.
These entries are unlikely to be needed, so remove them to clean up the
struct definition.
2019-12-20 23:36:17 +11:00
Damien George
035180ca01 py: Remove commented-out debug printf's from emitbc and objlist.
Any debugging prints should use a macro like DEBUG_printf.
2019-12-20 23:34:46 +11:00
Damien George
1f37194730 all: Bump version to 1.12. 2019-12-20 16:58:17 +11:00
Damien George
073c5f3a40 py/profile: Fix debug opcode decoding of MP_BC_RAISE_xxx opcodes. 2019-12-20 14:57:44 +11:00
Damien George
95473980ef py/vm: Fix comment to refer to MP_BC_RAISE_OBJ instead of RAISE_VARARGS. 2019-12-20 14:57:06 +11:00
Damien George
0bd7d1f7f0 py/persistentcode: Move loading of rodata/bss to before obj/raw-code.
This makes the loading of viper-code-with-relocations a bit neater and
easier to understand, by treating the rodata/bss like a special object to
be loaded into the constant table (which is how it behaves).
2019-12-17 13:22:11 +11:00
Damien George
48e9262f55 py/dynruntime: Implement uint new/get, mp_obj_len and mp_obj_subscr. 2019-12-13 13:29:11 +11:00
Damien George
abc642973d py/dynruntime: Add support for float API to make/get floats.
We don't want to add a feature flag to .mpy files that indicate float
support because it will get complex and difficult to use.  Instead the .mpy
is built using whatever precision it chooses (float or double) and the
native glue API will convert between this choice and what the host runtime
actually uses.
2019-12-12 20:15:28 +11:00
Damien George
ff58961944 py/nativeglue: Add float new/get functions with both single and double. 2019-12-12 20:15:28 +11:00
Damien George
9ac949cdbd py/persistentcode: Make ARM Thumb archs support multiple sub-archs. 2019-12-12 20:15:28 +11:00
Damien George
aad79adab7 tools/mpy_ld.py: Add new mpy_ld.py tool and associated build files.
This commit adds a new tool called mpy_ld.py which is essentially a linker
that builds .mpy files directly from .o files.  A new header file
(dynruntime.h) and makefile fragment (dynruntime.mk) are also included
which allow building .mpy files from C source code.  Such .mpy files can
then be dynamically imported as though they were a normal Python module,
even though they are implemented in C.

Converting .o files directly (rather than pre-linked .elf files) allows the
resulting .mpy to be more efficient because it has more control over the
relocations; for example it can skip PLT indirection.  Doing it this way
also allows supporting more architectures, such as Xtensa which has
specific needs for position-independent code and the GOT.

The tool supports targets of x86, x86-64, ARM Thumb and Xtensa (windowed
and non-windowed).  BSS, text and rodata sections are supported, with
relocations to all internal sections and symbols, as well as relocations to
some external symbols (defined by dynruntime.h), and linking of qstrs.
2019-12-12 20:15:28 +11:00
Damien George
3690f79afc py/nativeglue: Add funcs/types to native glue table for dynamic runtime.
These allow discovery of symbols by native code that is loaded dynamically.
2019-12-12 20:15:28 +11:00
Damien George
360d972c16 py/nativeglue: Add new header file with native function table typedef. 2019-12-12 20:15:28 +11:00
Damien George
b47e155bd0 py/persistentcode: Add ability to relocate loaded native code.
Implements text, rodata and bss generalised relocations, as well as generic
qstr-object linking.  This allows importing dynamic native modules on all
supported architectures in a unified way.
2019-12-12 20:15:28 +11:00
Emil Renner Berthing
4ebbacd65e py/objenumerate: Check for valid args in enumerate constructor.
For the case where MICROPY_CPYTHON_COMPAT is disabled.  This fix makes
basics/fun_error2.py pass and not crash the interpreter.
2019-12-09 14:28:24 +11:00
Damien George
01e5802ee3 py: Remove 3 obsolete commented-out lines from header files. 2019-11-26 21:36:41 +11:00
Yonatan Goldschmidt
4318a6d755 py/objstringio: Slightly optimize stringio_copy_on_write for code size.
With the memcpy() call placed last it avoids the effects of registers
clobbering.  It's definitely effective in non-inlined functions, but even
here it is still making a small difference.  For example, on stm32, this
saves an extra `ldr` instruction to load `o->vstr` after the memcpy()
returns.
2019-11-26 14:26:24 +11:00
Léa Saviot
bc129f1b84 py/qstr: Raise exception in qstr_from_strn if str to intern is too long.
The string length being longer than the allowed qstr length can happen in
many locations, for example in the parser with very long variable names.
Without an explicit check that the length is within range (as done in this
patch) the code would exhibit crashes and strange behaviour with truncated
strings.
2019-11-26 10:51:47 +11:00
Léa Saviot
a7bc4d1a14 py/builtinimport: Raise exception on empty module name.
To prevent a crash returning MP_OBJ_NULL.  A test is added for this case.
2019-11-26 00:28:32 +11:00
Petr Viktorin
57c18fdd38 py/compile: Coalesce error message for break/continue outside loop.
To reduce code size.
2019-11-21 12:13:11 +11:00
Jim Mussared
4f96689281 py/ringbuf: Add peek16 method. 2019-11-21 12:04:53 +11:00
Andrew Leech
1e87f11d3f py/objdict: Support ujson.dump() of OrderedDict objects.
Following CPython, OrderedDict are dumped with the syntax of dict.
2019-11-13 13:51:18 +11:00
Damien George
7e374d2317 py/emitnx86: Make mp_f_n_args table match order of mp_fun_kind_t. 2019-11-07 19:43:23 +11:00
Thea Flowers
f0e4677f0d py/emitnative: Fix typo, REG_PARENT_ARG_RET should be REG_PARENT_RET. 2019-11-07 12:30:47 +11:00
Damien George
c13f9f209d all: Convert nlr_raise(mp_obj_new_exception_msg(x)) to mp_raise_msg(x).
This helper function was added a while ago and these are the remaining
cases to convert, to save a bit of code size.
2019-11-05 11:35:45 +11:00
Damien George
80df377e95 py/modsys: Report .mpy version in sys.implementation.
This commit adds a sys.implementation.mpy entry when the system supports
importing .mpy files.  This entry is a 16-bit integer which encodes two
bytes of information from the header of .mpy files that are supported by
the system being run: the second and third bytes, .mpy version, and flags
and native architecture.  This allows determining the supported .mpy file
dynamically by code, and also for the user to find it out by inspecting
this value.  It's further possible to dynamically detect if the system
supports importing .mpy files by `hasattr(sys.implementation, 'mpy')`.
2019-11-04 16:00:41 +11:00
Damien George
f4601af10a py/persistentcode: Move declarations for .mpy header from .c to .h file. 2019-11-04 16:00:05 +11:00
Jim Mussared
5578182ec9 py/objgenerator: Allow pend_throw to an unstarted generator.
Replace the is_running field with a tri-state variable to indicate
running/not-running/pending-exception.

Update tests to cover the various cases.

This allows cancellation in uasyncio even if the coroutine hasn't been
executed yet.  Fixes #5242
2019-11-04 15:51:16 +11:00
Jim Mussared
576ed89224 py/objgenerator: Remove globals from mp_obj_gen_instance_t.
This wasn't necessary as the wrapped function already has a reference to
its globals.  But it had a dual purpose of tracking whether the function
was currently running, so replace it with a bool.
2019-11-04 15:50:59 +11:00
Damien George
f2ecfe8b83 py/nativeglue: Remove unused mp_obj_new_cell from mp_fun_table.
It has been unused since 9988618e0e
2019-11-01 17:26:10 +11:00
Damien George
d3c383de79 py/stream.h: Add MP_STREAM_POLL_NVAL constant. 2019-10-31 12:54:37 +11:00
Damien George
323d47887f py/runtime: Reorder some binary ops so they don't require conditionals.
runtime0.h is part of the MicroPython ABI so it's simpler if it's
independent of config options, like MICROPY_PY_REVERSE_SPECIAL_METHODS.

What's effectively done here is to move MP_BINARY_OP_DIVMOD and
MP_BINARY_OP_CONTAINS up in the enum, then remove the #if
MICROPY_PY_REVERSE_SPECIAL_METHODS conditional.

Without this change .mpy files would need to have a feature flag for
MICROPY_PY_REVERSE_SPECIAL_METHODS (when embedding native code that uses
this enum).

This commit has no effect when MICROPY_PY_REVERSE_SPECIAL_METHODS is
disabled.  With this option enabled this commit reduces code size by about
60 bytes.
2019-10-29 23:13:51 +11:00
Damien George
a099505420 extmod: Add VFS littlefs bindings.
Both LFS1 and LFS2 are supported at the same time.
2019-10-29 14:17:29 +11:00
Damien George
e1c7b1cb43 extmod/vfs_blockdev: Factor out block device interface code. 2019-10-29 12:55:17 +11:00
Michael Neuling
079cc940a6 powerpc: Add initial port to bare metal PowerPC arch.
Runs in microwatt (GHDL and FPGA) and qemu.

Port done initially by Michael Neuling, with help from Anton Blanchard and
Jordan Niethe.
2019-10-22 22:45:33 +11:00
Jim Mussared
b02d7e612d extmod/modbluetooth: Rename module to "ubluetooth".
For consistency with "umachine". Now that weak links are enabled
by default for built-in modules, this should be a no-op, but allows
extension of the bluetooth module by user code.

Also move registration of ubluetooth to objmodule rather than
port-specific.
2019-10-22 21:58:05 +11:00
Damien George
21a60935a5 py/modarray: Rename "array" module to "uarray".
Following the other modules like ustruct, ucollections.

See issues #4370 and #4449.
2019-10-22 16:35:46 +11:00
Damien George
d2384efa80 py: Automatically provide weak links from "foo" to "ufoo" module name.
This commit implements automatic module weak links for all built-in
modules, by searching for "ufoo" in the built-in module list if "foo"
cannot be found.  This means that all modules named "ufoo" are always
available as "foo".  Also, a port can no longer add any other weak links,
which makes strict the definition of a weak link.

It saves some code size (about 100-200 bytes) on ports that previously had
lots of weak links.

Some changes from the previous behaviour:
- It doesn't intern the non-u module names (eg "foo" is not interned),
  which saves code size, but will mean that "import foo" creates a new qstr
  (namely "foo") in RAM (unless the importing module is frozen).
- help('modules') no longer lists non-u module names, only the u-variants;
  this reduces duplication in the help listing.

Weak links are effectively the same as having a set of symbolic links on
the filesystem that is searched last.  So an "import foo" will search
built-in modules first, then all paths in sys.path, then weak links last,
importing "ufoo" if it exists.  Thus a file called "foo.py" somewhere in
sys.path will still have precedence over the weak link of "foo" to "ufoo".

See issues: #1740, #4449, #5229, #5241.
2019-10-22 15:30:52 +11:00
Jim Mussared
c7ae8c5a99 py/objstr: Size-optimise failure path for mp_obj_str_get_buffer.
These fields are never looked at if the function returns non-zero.
2019-10-22 13:54:09 +11:00
Jim Mussared
7662501d5b py/mkrules.mk: Add warning/error for invalid frozen config. 2019-10-21 23:21:04 +11:00
Jim Mussared
8ba963cfa3 tools/makemanifest.py: Eval relative paths w.r.t. current manifest file.
When loading a manifest file, e.g. by include(), it will chdir first to the
directory of that manifest.  This means that all file operations within a
manifest are relative to that manifest's location.

As a consequence of this, additional environment variables are needed to
find absolute paths, so the following are added: $(MPY_LIB_DIR),
$(PORT_DIR), $(BOARD_DIR).  And rename $(MPY) to $(MPY_DIR) to be
consistent.

Existing manifests are updated to match.
2019-10-21 23:01:41 +11:00
Josh Lloyd
8f9e2e325a py/objtype: Add type.__bases__ attribute.
Enabled as part of MICROPY_CPYTHON_COMPAT.
2019-10-18 15:20:56 +11:00
Damien George
e81f538e25 tools: Add mechanism to provide a manifest of frozen files.
This introduces a new build variable FROZEN_MANIFEST which can be set to a
manifest listing (written in Python) that describes the set of files to be
frozen in to the firmware.
2019-10-15 21:34:23 +11:00
Damien George
d7a9388fe0 ports: Add new make target "submodules" which inits required modules. 2019-10-15 17:14:41 +11:00
Damien George
23f0691fdd py/persistentcode: Make .mpy more compact with qstr directly in prelude.
Instead of encoding 4 zero bytes as placeholders for the simple_name and
source_file qstrs, and storing the qstrs after the bytecode, store the
qstrs at the location of these 4 bytes.  This saves 4 bytes per bytecode
function stored in a .mpy file (for example lcd160cr.mpy drops by 232
bytes, 4x 58 functions).  And resulting code size is slightly reduced on
ports that use this feature.
2019-10-15 16:56:27 +11:00
Damien George
9adedce42e py: Add new Xtensa-Windowed arch for native emitter.
Enabled via the configuration MICROPY_EMIT_XTENSAWIN.
2019-10-05 13:44:53 +10:00
Damien George
f7ddc94166 py/asmxtensa: Add support for Xtensa with windowed registers.
Window-specific asm emit functions are added, along with a new macro option
GENERIC_ASM_API_WIN.
2019-10-05 13:44:08 +10:00
Damien George
306ec5369a py/emitnative: Add support for archs that cannot read executable data.
In which case place the native function prelude in a bytes object, linked
from the const_table of that function.  An architecture should define
N_PRELUDE_AS_BYTES_OBJ to 1 before including py/emitnative.c to emit
correct machine code, then enable MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ
so the runtime can correctly handle the prelude being in a bytes object.
2019-10-05 13:42:39 +10:00
Damien George
3504edc804 py/emitnative: Add support for using setjmp with native emitter.
To enable this feature the N_NLR_SETJMP macro should be set to 1 before
including py/emitnative.c.
2019-10-05 13:41:58 +10:00
Damien George
4107597b84 py/emitnative: Add support for archs with windowed registers.
Such that args/return regs for the parent are different to args/return regs
for child calls.  For an architecture to use this feature it should define
the REG_PARENT_xxx macros before including py/emitnative.c.
2019-10-05 13:41:14 +10:00
Damien George
809d89c794 py/runtime: Fix PEP479 behaviour throwing StopIteration into yield from.
Commit 3f6ffe059f implemented PEP479 but did
not catch the case fixed in this commit.  Found by coverage analysis, that
the VM had uncovered code.
2019-10-04 23:27:00 +10:00
Damien George
82c494a97e py/vm: Fix handling of unwind jump out of active finally.
Prior to this commit, when unwinding through an active finally the stack
was not being correctly popped/folded, which resulting in the VM crashing
for complicated unwinding of nested finallys.

This should be fixed with this commit, and more tests for return/break/
continue within a finally have been added to exercise this.
2019-10-04 23:01:29 +10:00
Jim Mussared
a09fd04758 py/makeqstrdefs.py: Remove unused blacklist.
As of 7d58a197cf, `NULL` should no longer be
here because it's allowed (MP_QSTRnull took its place).  This entry was
preventing the use of MP_QSTR_NULL to mean "NULL" (although this is not
currently used).

A blacklist should not be needed because it should be possible to intern
all strings.

Fixes issue #5140.
2019-10-04 17:18:56 +10:00
Jim Mussared
4ddd46e6cf docs/develop/qstr.rst: Add documentation for string interning. 2019-10-04 17:13:13 +10:00
Petr Viktorin
25a9bccdee py/compile: Disallow 'import *' outside module level.
This check follows CPython's behaviour, because 'import *' always populates
the globals with the imported names, not locals.

Since it's safe to do this (doesn't lead to a crash or undefined behaviour)
the check is only enabled for MICROPY_CPYTHON_COMPAT.

Fixes issue #5121.
2019-10-04 16:46:47 +10:00
Damien George
1d0423419b py/bc: Don't include mp_decode_uint funcs when not needed.
These are now only needed when persistent code is disabled.
2019-10-01 12:26:22 +10:00
Damien George
c8c0fd4ca3 py: Rework and compress second part of bytecode prelude.
This patch compresses the second part of the bytecode prelude which
contains the source file name, function name, source-line-number mapping
and cell closure information.  This part of the prelude now begins with a
single varible length unsigned integer which encodes 2 numbers, being the
byte-size of the following 2 sections in the header: the "source info
section" and the "closure section".  After decoding this variable unsigned
integer it's possible to skip over one or both of these sections very
easily.

This scheme saves about 2 bytes for most functions compared to the original
format: one in the case that there are no closure cells, and one because
padding was eliminated.
2019-10-01 12:26:22 +10:00
Damien George
b5ebfadbd6 py: Compress first part of bytecode prelude.
The start of the bytecode prelude contains 6 numbers telling the amount of
stack needed for the Python values and exceptions, and the signature of the
function.  Prior to this patch these numbers were all encoded one after the
other (2x variable unsigned integers, then 4x bytes), but using so many
bytes is unnecessary.

An entropy analysis of around 150,000 bytecode functions from the CPython
standard library showed that the optimal Shannon coding would need about
7.1 bits on average to encode these 6 numbers, compared to the existing 48
bits.

This patch attempts to get close to this optimal value by packing the 6
numbers into a single, varible-length unsigned integer via bit-wise
interleaving.  The interleaving scheme is chosen to minimise the average
number of bytes needed, and at the same time keep the scheme simple enough
so it can be implemented without too much overhead in code size or speed.
The scheme requires about 10.5 bits on average to store the 6 numbers.

As a result most functions which originally took 6 bytes to encode these 6
numbers now need only 1 byte (in 80% of cases).
2019-10-01 12:26:22 +10:00
Damien George
81d04a0200 py: Add n_state to mp_code_state_t struct.
This value is used often enough that it is better to cache it instead of
decode it each time.
2019-10-01 12:26:22 +10:00
Damien George
4c5e1a0368 py/bc: Change mp_code_state_t.exc_sp to exc_sp_idx.
Change from a pointer to an index, to make space in mp_code_state_t.
2019-10-01 12:26:22 +10:00
Damien George
1d7afcce49 py/bc: Remove comments referring to obsolete currently_in_except_block.
It was made obsolete in 6f9e3ff719
2019-10-01 12:26:22 +10:00
Jim Mussared
16f8ceeaaa extmod/modbluetooth: Add low-level Python BLE API. 2019-10-01 09:51:02 +10:00
Jim Mussared
42e9bdf19b py/ringbuf: Add helpers for put16/get16. 2019-10-01 09:51:02 +10:00
Damien George
5716c5cf65 py/persistentcode: Bump .mpy version to 5.
The bytecode opcodes have changed (there are more, and they have been
reordered).
2019-09-26 16:39:37 +10:00
Damien George
96f2a38075 py/nativeglue: Make mp_fun_table fixed size regardless of config.
So that mpy files with native code will always work correctly, and raise an
exception if a feature is used that is not supported by the runtime.
2019-09-26 16:24:06 +10:00
Josh Lloyd
7d58a197cf py: Rename MP_QSTR_NULL to MP_QSTRnull to avoid intern collisions.
Fixes #5140.
2019-09-26 16:04:56 +10:00
Damien George
b596638b9b mpy-cross: Set number of registers in nlr_buf_t based on native arch.
Fixes #5059.  Done in collaboration with Jim Mussared.
2019-09-26 15:53:05 +10:00
Damien George
74503107a7 py/emitnative: Factor sizeof/offsetof calculations to macros. 2019-09-26 15:52:19 +10:00
Damien George
a48cdb57b2 py/nlr.h: Factor out constants to specific macros. 2019-09-26 15:52:19 +10:00
Damien George
02db91a7a3 py: Split RAISE_VARARGS opcode into 3 separate ones.
From the beginning of this project the RAISE_VARARGS opcode was named and
implemented following CPython, where it has an argument (to the opcode)
counting how many args the raise takes:

    raise # 0 args (re-raise previous exception)
    raise exc # 1 arg
    raise exc from exc2 # 2 args (chained raise)

In the bytecode this operation therefore takes 2 bytes, one for
RAISE_VARARGS and one for the number of args.

This patch splits this opcode into 3, where each is now a single byte.
This reduces bytecode size by 1 byte for each use of raise.  Every byte
counts!  It also has the benefit of reducing code size (on all ports except
nanbox).
2019-09-26 15:39:50 +10:00
Damien George
870e900d02 py: Introduce and use constants for multi-opcode sizes. 2019-09-26 15:27:11 +10:00
Damien George
1f7202d122 py/bc: Replace big opcode format table with simple macro. 2019-09-26 15:27:11 +10:00
Damien George
5889cf58db py/bc0: Order opcodes into groups based on their size and format. 2019-09-26 15:27:10 +10:00
Damien George
2069c563f9 py: Add support for matmul operator @ as per PEP 465.
To make progress towards MicroPython supporting Python 3.5, adding the
matmul operator is important because it's a really "low level" part of the
language, being a new token and modifications to the grammar.

It doesn't make sense to make it configurable because 1) it would make the
grammar and lexer complicated/messy; 2) no other operators are
configurable; 3) it's not a feature that can be "dynamically plugged in"
via an import.

And matmul can be useful as a general purpose user-defined operator, it
doesn't have to be just for numpy use.

Based on work done by Jim Mussared.
2019-09-26 15:12:39 +10:00
Damien George
14e203282a py/compile: Use calculation instead of switch to convert token to op. 2019-09-26 14:37:26 +10:00
Damien George
9bf2feba63 py/parse: Use calculation instead of table to convert token to operator. 2019-09-26 14:37:26 +10:00
Damien George
6ce7c051e8 py/lexer: Reorder operator tokens to match corresponding binary ops. 2019-09-26 14:37:26 +10:00
stijn
22131a6738 py/persistentcode: Enable persistent code saving for Windows ports. 2019-09-18 22:15:36 +10:00
Damien George
6e07fde895 py/mkrules.mk: Add QSTR_GLOBAL_REQUIREMENTS variable for qstr auto-gen. 2019-09-12 18:16:16 +10:00
Damien George
ea060a42e9 py/vm: Factor cached map lookup code to inline function.
To reduce code duplication and allow to more easily modify this function.
2019-09-10 11:23:52 +10:00
Damien George
fb73bdae62 py/mkenv.mk: Add GDB variable. 2019-09-05 22:59:06 +10:00
Damien George
b29fae0c56 py/bc: Fix size calculation of UNWIND_JUMP opcode in mp_opcode_format.
Prior to this patch mp_opcode_format would calculate the incorrect size of
the MP_BC_UNWIND_JUMP opcode, missing the additional byte.  But, because
opcodes below 0x10 are unused and treated as bytes in the .mpy load/save
and freezing code, this bug did not show any symptoms, since nested unwind
jumps would rarely (if ever) reach a depth of 16 (so the extra byte of this
opcode would be between 0x01 and 0x0f and be correctly loaded/saved/frozen
simply as an undefined opcode).

This patch fixes this bug by correctly accounting for the additional byte.
        .
2019-09-02 13:30:16 +10:00
Damien George
c348e79187 py/binary: Change mp_uint_t to size_t for index, size, align args.
Reduces code size for nan-box builds, otherwise changes nothing.
2019-09-02 13:14:27 +10:00
Damien George
24c3e9b283 py/modstruct: Fix struct.pack_into with unaligned offset of native type.
Following the same fix for unpack.
2019-09-02 13:14:16 +10:00
Tom McDermott
1022f9cc35 py/modstruct: Fix struct.unpack with unaligned offset of native type.
With this patch alignment is done relative to the start of the buffer that
is being unpacked, not the raw pointer value, as per CPython.

Fixes issue #3314.
2019-09-02 13:10:55 +10:00
Jeff Epler
12f13ee634 py/objtuple: Allow compatible subclasses of tuple in mp_obj_tuple_get.
As part of this patch a private macro mp_obj_is_tuple_compatible is
introduced to encapsulate the check, which is used in two locations.

Fixes #5005.
2019-08-31 00:14:11 +10:00
Milan Rossa
b295df4b08 py/profile: Add debugging for sys.settrace feature. 2019-08-30 16:48:47 +10:00
Milan Rossa
310b3d1b81 py: Integrate sys.settrace feature into the VM and runtime.
This commit adds support for sys.settrace, allowing to install Python
handlers to trace execution of Python code.  The interface follows CPython
as closely as possible.  The feature is disabled by default and can be
enabled via MICROPY_PY_SYS_SETTRACE.
2019-08-30 16:44:12 +10:00
Milan Rossa
c96aedad46 py/profile: Add initial implementation of sys.settrace feature. 2019-08-30 16:43:56 +10:00
Damien George
dbf35d3da3 py/bc: Factor out code to get bytecode line number info into new func. 2019-08-30 16:43:46 +10:00
Damien George
c7c6703950 py/compile: Improve the line numbering precision for lambdas.
Prior to this patch the line number for a lambda would be "line 1" if the
body of the lambda contained only a simple expression (with no line number
stored in the parse node).  Now the line number is always reported
correctly.
2019-08-30 16:43:46 +10:00
Damien George
af20c2ead3 py: Add global default_emit_opt variable to make emit kind persistent.
mp_compile no longer takes an emit_opt argument, rather this setting is now
provided by the global default_emit_opt variable.

Now, when -X emit=native is passed as a command-line option, the emitter
will be set for all compiled modules (included imports), not just the
top-level script.

In the future there could be a way to also set this variable from a script.

Fixes issue #4267.
2019-08-28 12:47:58 +10:00
Damien George
08c1fe5569 py/vm: Don't add traceback info for exceptions that are re-raised.
With this patch exceptions that are re-raised have improved tracebacks
(less confusing, match CPython), and it makes re-raise slightly more
efficient (in time and RAM) because they no longer need to add a traceback.
Also general VM performance is not measurably affected.

Partially fixes issue #2928.
2019-08-28 12:31:53 +10:00
Damien George
16f6169c88 py/vm: Don't add traceback info for exc's propagated through a finally.
With this patch exception tracebacks that go through a finally are improved
(less confusing, match CPython), and it makes finally's slightly more
efficient (in time and RAM) because they no longer need to add a traceback.

Partially fixes issue #2928.
2019-08-28 12:31:49 +10:00
Damien George
3d7455a0bb py/py.mk: Remove trailing spaces at end of line. 2019-08-27 16:37:43 +10:00
Damien George
2fca0d7f18 py/vm: Shorten error message for not-implemented opcode.
It's really an opcode that's not implemented, so use "opcode" instead of
"byte code".  And remove the redundant "not implemented" text because that
is already implied by the exception type.  There's no need to have a long
error message for an exception that is almost never encountered.  Saves
about 20 bytes of code size on most ports.
2019-08-22 16:13:05 +10:00
Damien George
53527138a9 py/bc0.h: Add comment that MP_BC_MAKE_CLOSURE/_DEFARGS take extra byte. 2019-08-22 16:07:28 +10:00
Damien George
973c87d8fa py/objgenerator: Move defn of mp_const_GeneratorExit_obj here.
Because the mp_obj_exception_t type is now globally available.
2019-08-22 16:05:05 +10:00
Damien George
bc9b656f35 py/runtime: Remove obsolete comment about mp_parse_compile_execute.
mp_locals_get/set and mp_globals_get/set are now static-inline functions so
this comment is no longer correct.
2019-08-22 15:59:14 +10:00
Damien George
8e7745eb31 py/emitbc: Make all emit_write_bytecode_* funcs take a stack_adj arg.
This factoring of code gives significant code-size savings:

   bare-arm:  -456 -0.682%
minimal x86:  -844 -0.547%
   unix x64:  -472 -0.095%
unix nanbox: -1348 -0.303%
      stm32:  -472 -0.130% PYBV10
     cc3200:  -448 -0.242%
    esp8266:  -708 -0.108%
      esp32:  -400 -0.036% GENERIC
        nrf:  -520 -0.356% pca10040
       samd:  -456 -0.448% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2019-08-22 15:32:26 +10:00
Damien George
3d9bd80447 py/emitbc: Rewrite switch in load_const_tok to reduce code size. 2019-08-22 15:10:25 +10:00
Milan Rossa
ae6fe8b43c py/compile: Improve the line numbering precision for comprehensions.
The line number for comprehensions is now always reported as the correct
global location in the script, instead of just "line 1".
2019-08-19 23:50:30 +10:00
Damien George
7d851a27f1 extmod/modure: Make regex dump-code debugging feature optional.
Enabled via MICROPY_PY_URE_DEBUG, disabled by default (but enabled on unix
coverage build).  This is a rarely used feature that costs a lot of code
(500-800 bytes flash).  Debugging of regular expressions can be done
offline with other tools.
2019-08-19 16:43:00 +10:00
Damien George
11ecdf2ec6 py/nlr: Use MP_UNREACHABLE at the end of arch-specific nlr_jump funcs.
Recent versions of gcc perform optimisations which can lead to the
following code from the MP_NLR_JUMP_HEAD macro being omitted:

    top->ret_val = val; \
    MP_NLR_RESTORE_PYSTACK(top); \
    *_top_ptr = top->prev; \

This is noticeable (at least) in the unix coverage on x86-64 built with gcc
9.1.0.  This is because the nlr_jump function is marked as no-return, so
gcc deduces that the above code has no effect.

Adding MP_UNREACHABLE tells the compiler that the asm code may branch
elsewhere, and so it cannot optimise away the code.
2019-08-19 16:14:57 +10:00
Damien George
0c80cb39af py: Introduce MP_UNREACHABLE macro to annotate unreachable code.
And use it to replace the same pattern at the end of nlrthumb.c:nlr_jump.
2019-08-19 16:14:33 +10:00
stijn
af5c998f37 py/modmath: Implement math.isclose() for non-complex numbers.
As per PEP 485, this function appeared in for Python 3.5.  Configured via
MICROPY_PY_MATH_ISCLOSE which is disabled by default, but enabled for the
ports which already have MICROPY_PY_MATH_SPECIAL_FUNCTIONS enabled.
2019-08-17 23:23:17 +10:00
Damien George
acfbb9febd py/objarray: Fix amount of free space in array when doing slice assign.
Prior to this patch the amount of free space in an array (including
bytearray) was not being maintained correctly for the case of slice
assignment which changed the size of the array.  Under certain cases (as
encoded in the new test) it was possible that the array could grow beyond
its allocated memory block and corrupt the heap.

Fixes issue #4127.
2019-08-15 23:02:04 +10:00
Milan Rossa
cb3647004f py: Implement new sys.atexit feature.
This patch implements a new sys.atexit function which registers a function
that is later executed when the main script ends.  It is configurable via
MICROPY_PY_SYS_ATEXIT, disabled by default.

This is not compliant with CPython, rather it can be used to implement a
CPython compatible "atexit" module if desired (similar to how
sys.print_exception can be used to implement functionality of the
"traceback" module).
2019-08-15 17:30:50 +10:00
Milan Rossa
efdcd6baa7 py/showbc: Fix off-by-one when showing address of unknown opcode. 2019-08-06 16:08:39 +10:00
Damien George
cd35dd9d9a py: Allow to pass in read-only buffers to viper and inline-asm funcs.
Fixes #4936.
2019-08-06 15:58:23 +10:00
Paul m. p. P
f60229e261 py/modio: Call mp_import_name to do resource stream import.
So code is not duplicated and it can take advantage of __import__ being
overridden.
2019-07-31 22:37:44 +10:00
Paul m. p. P
60f1063797 py/runtime: Allow to override builtins.__import__ with Python func.
This patch adds a simple but powerful hook into the import system, in a
CPython compatible way, by allowing to override builtins.__import__.

This does introduce some overhead to all imports but it's minor:
- the dict lookup of __import__ is bypassed if there are no modifications
  to the builtins module (which is the case at start up);
- imports are not performance critical, usually done just at the start of a
  script;
- compared to how much work is done in an import, looking up a value in a
  dict is a relatively small additional piece of work.
2019-07-31 22:36:00 +10:00
Paul m. p. P
a8e3201b37 py/builtinimport: Populate __file__ when importing frozen or mpy files.
Note that bytecode already includes the source filename as a qstr so there
is no additional memory used by the interning operation here.
2019-07-31 17:00:11 +10:00
Eric Poulsen
01054f2092 py/objdict: Quote non-string types when used as keys in JSON output.
JSON requires that keys of objects be strings.  CPython will therefore
automatically quote simple types (NoneType, bool, int, float) when they are
used directly as keys in JSON output.  To prevent subtle bugs and emit
compliant JSON, MicroPython should at least test for such keys so they
aren't silently let through.  Then doing the actual quoting is a similar
cost to raising an exception, so that's what is implemented by this patch.

Fixes issue #4790.
2019-07-30 16:34:27 +10:00
Yonatan Goldschmidt
e9593d5075 py/sequence: Fix grammar in comment about equality. 2019-07-25 16:37:25 +10:00
Paul m. p. P
3b3a4749ce py/objstringio: Guard bytesio_stream_p struct w/ MICROPY_PY_IO_BYTESIO.
It's static and can lead to a compilation warning/error when
MICROPY_PY_IO_BYTESIO is disabled.
2019-07-17 16:12:19 +10:00
Jim Mussared
bc66fe9064 py/scheduler: Rename sched_stack to sched_queue.
Behaviour was changed from stack to queue in
8977c7eb58, and this updates variable names
to match.  Also updates other references (docs, error messages).
2019-07-17 16:09:32 +10:00
Paul m. p. P
154062d9cb py/makeqstrdata.py: Allow using \r\n as a qstr if a port requires it. 2019-07-12 17:10:11 +10:00
David Lechner
82dc9856b0 py/asmarm: Use __builtin___clear_cache instead of __clear_cache.
__clear_cache causes a compile error when using clang.  Instead use
__builtin___clear_cache which is available under both gcc and clang.

Also replace tabs with spaces in this section of code (introduced by a
previous commit).
2019-07-12 16:48:37 +10:00
Laurens Valk
a73859d5af py/objgenerator: Add missing #if guard for PY_GENERATOR_PEND_THROW.
Without it, gen_instance_pend_throw_obj is defined but not used when
MICROPY_PY_GENERATOR_PEND_THROW is set to 0.
2019-07-09 13:24:23 +10:00
David Lechner
f3a5b313e5 py/nlrthumb: Check __thumb2__ instead of __ARM_ARCH_6M__.
This fixes compiling for older architectures (e.g. armv5tej).

According to [1], the limit of R0-R7 for the STR and LDR instructions is
tied to the Thumb instruction set and not any specific processor
architectures.

[1]: http://www.keil.com/support/man/docs/armasm/armasm_dom1361289906890.htm
2019-07-03 01:24:22 +10:00
David Lechner
62b00dd5d8 py/asmarm: Use __clear_cache on Linux/GCC when creating new asm code.
Comes from https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/caches-and-self-modifying-code

This fixes a crash when running MicroPython using qemu-arm.
2019-07-03 01:19:41 +10:00
Paul m. p. P
2920d26af5 py/persistentcode: Ensure prelude_offset is always initialised. 2019-07-01 23:46:49 +10:00
Damien George
b7da67cdaa lib/utils/sys_stdio_mphal: Add support to poll sys.stdin and sys.stdout.
A port must provide the following function for this to work:

    uintptr_t mp_hal_stdio_poll(uintptr_t poll_flags);
2019-07-01 17:10:12 +10:00
Jun Wu
d165a401dc py/persistentcode: Fix compilation with load and save both enabled.
With both MICROPY_PERSISTENT_CODE_SAVE and MICROPY_PERSISTENT_CODE_LOAD
enabled the code fails to compile, due to undeclared 'n_obj'.  If
MICROPY_EMIT_NATIVE is disabled there are more errors due to the use of
undefined fields in mp_raw_code_t.

This patch fixes such compilation by avoiding undefined fields.

MICROPY_EMIT_NATIVE was changed to MICROPY_EMIT_MACHINE_CODE in this file
to match the mp_raw_code_t definition.
2019-06-28 13:59:45 +10:00