Commit Graph

3155 Commits

Author SHA1 Message Date
Paul Sokolovsky
4556bd2acd py/objtype: mp_obj_class_lookup: Improve debug logging.
Now traces more explicitly thru the lookup process.
2017-08-31 00:44:51 +03:00
Paul Sokolovsky
df6605eaba py/objtype: mp_obj_instance_make_new: Fix typos in comment. 2017-08-30 20:55:34 +03:00
Damien George
0102ee092b py: Change obsolete "///" comment formatting to normal comments.
This comment style is no longer used because the docs are written by hand,
not generated.
2017-08-30 21:02:00 +10:00
Paul Sokolovsky
784909ce16 py/objtype: Handle NotImplemented return from binary special methods.
NotImplemented means "try other fallbacks (like calling __rop__
instead of __op__) and if nothing works, raise TypeError". As
MicroPython doesn't implement any fallbacks, signal to raise
TypeError right away.
2017-08-30 01:39:24 +03:00
Damien George
58321dd985 all: Convert mp_uint_t to mp_unary_op_t/mp_binary_op_t where appropriate
The unary-op/binary-op enums are already defined, and there are no
arithmetic tricks used with these types, so it makes sense to use the
correct enum type for arguments that take these values.  It also reduces
code size quite a bit for nan-boxing builds.
2017-08-29 13:16:30 +10:00
Damien George
be8e5744e6 py/nlrx86,x64: Replace #define of defined() with portable macro usage.
Using gcc -Wpedantic will warn that #define of defined() is non-portable
and this patch fixes this.
2017-08-29 12:52:18 +10:00
Paul Sokolovsky
37379a2974 py/objstr: startswith, endswith: Check arg to be a string.
Otherwise, it will silently get incorrect result on other values types,
including CPython tuple form like "foo.png".endswith(("png", "jpg"))
(which MicroPython doesn't support for unbloatedness).
2017-08-29 00:06:21 +03:00
Damien George
f9ecaa132f py/asmthumb: Use existing macro to properly clear the D-cache.
This macro is provided by stmhal/mphalport.h and makes sure the addr and
size arguments are correctly aligned.
2017-08-23 11:32:27 +10:00
Damien George
f5309fc4ff py/formatfloat: Don't post-increment variable that won't be used again. 2017-08-21 22:04:23 +10:00
Damien George
103ae43f95 py/objcomplex: Remove unnecessary assignment of variable. 2017-08-21 22:03:27 +10:00
Damien George
ad6aae13a4 py/compile: Remove unused pn_colon code when compiling func params. 2017-08-21 22:00:34 +10:00
Damien George
b16a755a0b py/mkrules.mk: Use "find -path" when searching for frozen obj files.
This allows the command to succeed without error even if there is no
$(BUILD)/build directory, which is the case for mpy-cross.
2017-08-21 20:32:30 +10:00
Paul Sokolovsky
e3383e9352 py/stream: seek: Consistently handle negative offset for SEEK_SET.
Per POSIX, this is EINVAL, so raises OSError(EINVAL).
2017-08-20 22:02:41 +03:00
Paul Sokolovsky
0cd9ab7755 py/objstringio: Fix regression with handling SEEK_SET.
For SEEK_SET, offset should be treated as unsigned, to allow full-width
stream sizes (e.g. 32-bit instead of 31-bit). This is now fully documented
in stream.h. Also, seek symbolic constants are added.
2017-08-20 22:02:41 +03:00
Tom Collins
168350cd98 py/objstringio: Prevent offset wraparound for io.BytesIO objects.
Too big positive, or too big negative offset values could lead to overflow
and address space wraparound and thus access to unrelated areas of memory
(a security issue).
2017-08-20 22:00:05 +03:00
Damien George
025e5f2b33 py/binary: Change internal bytearray typecode from 0 to 1.
The value of 0 can't be used because otherwise mp_binary_get_size will let
a null byte through as the type code (intepreted as byterray).  This can
lead to invalid type-specifier strings being let through without an error
in the struct module, and even buffer overruns.
2017-08-17 16:19:35 +10:00
Stefan Naumann
ace9fb5405 py: Add verbose debug compile-time flag MICROPY_DEBUG_VERBOSE.
It enables all the DEBUG_printf outputs in the py/ source code.
2017-08-15 11:53:36 +10:00
Bas van Sisseren
a14ce77b28 py/binary.c: Fix bug when packing big-endian 'Q' values.
Without bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x10\x00\x00\x00\x00'

With bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x00\x00\x00\x00\x10'
2017-08-15 11:33:43 +10:00
Javier Candeira
35a1fea90b all: Raise exceptions via mp_raise_XXX
- Changed: ValueError, TypeError, NotImplementedError
  - OSError invocations unchanged, because the corresponding utility
    function takes ints, not strings like the long form invocation.
  - OverflowError, IndexError and RuntimeError etc. not changed for now
    until we decide whether to add new utility functions.
2017-08-13 22:52:33 +10:00
Paul Sokolovsky
bfc2092dc5 py/modsys: Initial implementation of sys.getsizeof().
Implemented as a new MP_UNARY_OP. This patch adds support lists, dicts and
instances.
2017-08-11 09:43:07 +03:00
Damien George
7d4a2f773c all: Make use of $(TOP) variable in Makefiles, instead of "..".
$(TOP) is defined in py/mkenv.mk and should be used to refer to the top
level of this repository.
2017-08-11 12:22:19 +10:00
Damien George
3d25d9c7d9 py/objstr: Raise an exception for wrong type on RHS of str binary op.
The main case to catch is invalid types for the containment operator, of
the form str.__contains__(non-str).
2017-08-09 21:25:48 +10:00
Damien George
eb2784e8a2 py/objtuple: Allow to use inplace-multiplication operator on tuples. 2017-08-09 21:20:42 +10:00
Paul Sokolovsky
4dc7c5649b py/mkrules.mk: Show frozen modules sizes together with executable size.
This works for Unix and similar ports so far.
2017-08-06 15:43:21 +03:00
Damien George
0f12082f5b py,extmod,stmhal: Use "static inline" for funcs that should be inline.
"STATIC inline" can expand to "inline" if STATIC is defined to nothing, and
this case can lead to link errors.
2017-08-02 13:42:34 +10:00
Alexander Steffen
55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Damien George
b62bb53d0e py/modsys: Use MP_ROM_INT for int values in an mp_rom_map_elem_t. 2017-07-31 12:59:39 +10:00
Paul Sokolovsky
456450437f py/modio: BufferedWriter: Convert to mp_rom_map_elem_t. 2017-07-28 21:54:59 +03:00
Damien George
04552ff71b py: Implement raising a big-int to a negative power.
Before this patch raising a big-int to a negative power would just return
0.  Now it returns a floating-point number with the correct value.
2017-07-25 11:49:22 +10:00
Damien George
4d1fb6107f py/mpz: Make mpz_is_zero() an inline function.
It's more efficient as an inline function, and saves code size.
2017-07-25 11:32:04 +10:00
Damien George
aa7be82a4d all: Don't include system errno.h when it's not needed. 2017-07-24 18:43:14 +10:00
Damien George
a559098fec py/mperrno: Allow mperrno.h to be correctly included before other hdrs.
Before this patch the mperrno.h file could be included and would silently
succeed with incorrect config settings, because mpconfig.h was not yet
included.
2017-07-24 18:41:24 +10:00
Damien George
1ed3356540 py/py.mk: Make berkeley-db C-defs apply only to relevant source files.
Otherwise they can interfere (eg redefinition of "abort") with other source
files in a given uPy port.
2017-07-24 15:50:47 +10:00
Tom Collins
bb3bddabb5 py/builtinevex: Add typechecking of globals/locals args to eval/exec. 2017-07-21 15:16:51 +10:00
Damien George
761e4c7ff6 all: Remove trailing spaces, per coding conventions. 2017-07-19 13:12:10 +10:00
Damien George
cadbd7f3e6 py/modmicropython: Cast stack_limit value so it prints correctly.
Without this cast the print will give a wrong result on nan-boxing builds.
2017-07-18 22:30:22 +10:00
Damien George
3235b95f08 py/asmx64: Support moving a 64-bit immediate to one of top 8 registers.
If constants (eg mp_const_none_obj) are placed in very high memory
locations that require 64-bits for the pointer then the assembler must be
able to emit instructions to move such pointers to one of the top 8
registers (ie r8-r15).
2017-07-18 17:30:23 +10:00
Damien George
016325dd0a py/vm: Make n_state variable local to just set-up part of VM.
It's not used anywhere else in the VM loop, and clashes with (is shadowed
by) the n_state variable that's redeclared towards the end of the
mp_execute_bytecode function.  Code size is unchanged.
2017-07-18 16:17:23 +10:00
Alexander Steffen
299bc62586 all: Unify header guard usage.
The code conventions suggest using header guards, but do not define how
those should look like and instead point to existing files. However, not
all existing files follow the same scheme, sometimes omitting header guards
altogether, sometimes using non-standard names, making it easy to
accidentally pick a "wrong" example.

This commit ensures that all header files of the MicroPython project (that
were not simply copied from somewhere else) follow the same pattern, that
was already present in the majority of files, especially in the py folder.

The rules are as follows.

Naming convention:
* start with the words MICROPY_INCLUDED
* contain the full path to the file
* replace special characters with _

In addition, there are no empty lines before #ifndef, between #ifndef and
one empty line before #endif. #endif is followed by a comment containing
the name of the guard macro.

py/grammar.h cannot use header guards by design, since it has to be
included multiple times in a single C file. Several other files also do not
need header guards as they are only used internally and guaranteed to be
included only once:
* MICROPY_MPHALPORT_H
* mpconfigboard.h
* mpconfigport.h
* mpthreadport.h
* pin_defs_*.h
* qstrdefs*.h
2017-07-18 11:57:39 +10:00
Damien George
12d4fa9b37 py/gc: Refactor assertions in gc_free function.
gc_free() expects either NULL or a valid pointer into the heap, so the
checks for a valid pointer can be turned into assertions.
2017-07-12 12:17:38 +10:00
Damien George
0291a624cb py/compile: Fix enum variable declaration. 2017-07-09 13:18:14 +10:00
Damien George
f69ab79ec8 py/objgenerator: Allow to hash generators and generator instances.
Adds nothing to the code size, since it uses existing empty slots in the
type structures.
2017-07-07 11:47:38 +10:00
Tom Collins
145796f037 py,extmod: Some casts and minor refactors to quiet compiler warnings. 2017-07-07 11:32:22 +10:00
Krzysztof Blazewicz
91a385db98 py/compile: Use switch-case to match token and operator.
Reduces code size.
2017-07-05 15:50:36 +10:00
Krzysztof Blazewicz
a040fb89e7 py/compile: Combine arith and bit-shift ops into 1 compile routine.
This refactoring saves code space.
2017-07-05 15:49:00 +10:00
Damien George
e66fd56852 py/repl: Change mp_uint_t to size_t in repl helpers. 2017-07-04 23:44:54 +10:00
Damien George
7bd10c1ffe py: Change mp_uint_t to size_t in builtins code. 2017-07-04 23:44:22 +10:00
Damien George
8b84b8ab8a py/objdict: Factorise dict accessor helper to reduce code size.
Code size change in bytes for this patch is:

   bare-arm:   -72
minimal x86:   -48
   unix x64:   -32
unix nanbox:  -120
     stmhal:   -68
     cc3200:   -64
    esp8266:   -56
2017-07-04 23:24:59 +10:00
Damien George
b51919f5b7 py/makeversionhdr.py: Update to parse new release line in docs/conf.py.
The line in docs/conf.py with the release/version number was recently
changed and this patch makes the makeversionhdr.py script work again.
2017-07-04 22:37:41 +10:00
Damien George
6b8b56f859 py/modmath: Check for zero division in log with 2 args. 2017-07-04 02:15:11 +10:00
Damien George
9ed5e80eea py/vm: Make "if" control flow more obvious in YIELD_FROM opcode. 2017-07-04 02:14:25 +10:00
Damien George
9d2c72ad4f py/objstr: Remove unnecessary "sign" variable in formatting code. 2017-07-04 02:13:27 +10:00
Damien George
2138258fea py/runtime: Mark m_malloc_fail() as NORETURN. 2017-07-04 02:12:36 +10:00
Damien George
9aeba3e41b py/binary: Add missing "break" statements. 2017-07-04 02:11:46 +10:00
Damien George
65417c5ad9 py/objstr: Move uPy function wrappers to just after the C function.
This matches the coding/layout style of all the other objects.
2017-07-02 23:35:42 +10:00
Damien George
adf22c19ae py/mpprint: Remove unreachable check for neg return of mp_format_float. 2017-06-30 12:10:50 +10:00
Damien George
265500c5c8 py/objnamedtuple: Simplify and remove use of alloca building namedtuple.
Prior to this patch there were 2 paths for creating the namedtuple, one for
when no keyword args were passed, and one when there were keyword args.
And alloca was used in the keyword-arg path to temporarily create the array
of elements for the namedtuple, which would then be copied to a
heap-allocated object (the namedtuple itself).

This patch simplifies the code by combining the no-keyword and keyword
paths, and removing the need for the alloca by constructing the namedtuple
on the heap before populating it.

Heap usage in unchanged, stack usage is reduced, use of alloca is removed,
and code size is not increased and is actually reduced by between 20-30
bytes for most ports.
2017-06-29 17:40:25 +10:00
Damien George
2f7fad66a2 py/builtinimport: Remove unreachable code for relative imports.
The while-loop that calls chop_component will guarantee that level==-1 at
the end of the loop.  Hence the code following it is unnecessary.

The check for p==this_name will catch imports that are beyond the
top-level, and also covers the case of new_mod_q==MP_QSTR_ (equivalent to
new_mod_l==0) so that check is removed.

There is also a new check at the start for level>=0 to guard against
__import__ being called with bad level values.
2017-06-28 12:21:29 +10:00
Alexander Steffen
cd0987f5b7 py/frozenmod.h: Add missing header guards 2017-06-28 00:51:45 +03:00
Damien George
c408ed9fb1 py/mpconfig.h: Remove spaces in "Micro Python" and remove blank line. 2017-06-26 12:29:20 +10:00
Damien George
d94bc675e8 py/compile: Optimise emitter label indices to save a word of heap.
Previous to this patch, a label with value "0" was used to indicate an
invalid label, but that meant a wasted word (at slot 0) in the array of
label offsets.  This patch adjusts the label indices so the first one
starts at 0, and the maximum value indicates an invalid label.
2017-06-22 15:05:58 +10:00
Damien George
4c5f108321 py/compile: Fix bug with break/continue in else of optimised for-range.
This patch fixes a bug whereby the Python stack was not correctly reset if
there was a break/continue statement in the else black of an optimised
for-range loop.

For example, in the following code the "j" variable from the inner for loop
was not being popped off the Python stack:

    for i in range(4):
        for j in range(4):
            pass
        else:
            continue

This is now fixed with this patch.
2017-06-22 13:50:33 +10:00
Damien George
e269cabe3e py/objint: In to_bytes(), allow length arg to be any int and check sign. 2017-06-15 14:21:02 +10:00
Damien George
8c5632a869 py/objint: Support "big" byte-order in int.to_bytes(). 2017-06-15 13:56:21 +10:00
Damien George
48d867b4a6 all: Make more use of mp_raise_{msg,TypeError,ValueError} helpers. 2017-06-15 11:54:41 +10:00
Damien George
1e70fda69f py/compile: Raise SyntaxError if positional args are given after */**.
In CPython 3.4 this raises a SyntaxError.  In CPython 3.5+ having a
positional after * is allowed but uPy has the wrong semantics and passes
the arguments in the incorrect order.  To prevent incorrect use of a
function going unnoticed it is important to raise the SyntaxError in uPy,
until the behaviour is fixed to follow CPython 3.5+.
2017-06-14 18:18:01 +10:00
Damien George
e374cfff80 py/modthread: Raise RuntimeError in release() if lock is not acquired. 2017-06-14 14:43:50 +10:00
Damien George
6ed4581f54 py/formatfloat: Fix number of digits and exponent sign when rounding.
This patch fixes 2 things when printing a floating-point number that
requires rounding up of the mantissa:
- retain the correct precision; eg 0.99 becomes 1.0, not 1.00
- if the exponent goes from -1 to 0 then render it as +0, not -0
2017-06-13 13:36:56 +10:00
Paul Sokolovsky
07241cd37a py/objstringio: If created from immutable object, follow copy on write policy.
Don't create copy of immutable object's contents until .write() is called
on BytesIO.
2017-06-09 17:33:01 +03:00
Damien George
b24ccfc639 py/makeqstrdefs.py: Make script run correctly with Python 2.6. 2017-06-09 13:42:13 +10:00
Damien George
a8a5d1e8c8 py: Provide mp_decode_uint_skip() to help reduce stack usage.
Taking the address of a local variable leads to increased stack usage, so
the mp_decode_uint_skip() function is added to reduce the need for taking
addresses.  The changes in this patch reduce stack usage of a Python call
by 8 bytes on ARM Thumb, by 16 bytes on non-windowing Xtensa archs, and by
16 bytes on x86-64.  Code size is also slightly reduced on most archs by
around 32 bytes.
2017-06-09 13:36:33 +10:00
Damien George
e1cda00387 py/modsys: Allow to compile with obj-repr D and PY_ATTRTUPLE disabled. 2017-06-08 00:41:27 +10:00
Damien George
326e8860ab py/objstr: Allow to compile with obj-repr D, and unicode disabled. 2017-06-08 00:40:38 +10:00
Damien George
72732fea1a py/persistentcode: Allow to compile with complex numbers disabled. 2017-06-08 00:28:28 +10:00
Paul Sokolovsky
5da8de2b66 extmod/modlwip: Fix error codes for duplicate calls to connect().
If socket is already connected, POSIX requires returning EISCONN. If
connection was requested, but not yet complete (for non-blocking
socket), error code is EALREADY.

http://pubs.opengroup.org/onlinepubs/7908799/xns/connect.html
2017-06-04 12:30:41 +03:00
Damien George
9f85c4fe48 py/objstr: Catch case of negative "maxsplit" arg to str.rsplit().
Negative values mean no limit on the number of splits so should delegate to
the .split() method.
2017-06-02 13:07:22 +10:00
Damien George
bc76302eab py/modbuiltins: Add core-provided version of input() function.
The implementation is taken from stmhal/input.c, with code added to handle
ctrl-C.  This built-in is controlled by MICROPY_PY_BUILTINS_INPUT and is
disabled by default.  It uses readline() to capture input but this can be
overridden by defining the mp_hal_readline macro.
2017-06-01 16:02:49 +10:00
Ville Skyttä
ca16c38210 various: Spelling fixes 2017-05-29 11:36:05 +03:00
Tom Collins
e26fb3ad73 py/objstringio: Catch mp_uint_t overflow of stream position in write(). 2017-05-26 13:40:08 +10:00
Damien George
85f7b0b468 py/mkrules.mk: Fix auto-qstr generation when "make -B" is used.
For make v3.81, using "make -B" can set $? to empty and in this case the
auto-qstr generation needs to pass all args (ie $^) to cpp.  The previous
fix for this (which was removed in 23a693ec2d)
used if statements in the shell command, which gave very long lines that
didn't work on certain systems (eg cygwin).

The fix in this patch is to use an $if(...) expression, which will evaluate
to $? (only newer prerequisites) if it's non empty, otherwise it will use
$^ (all prerequisites).
2017-05-26 13:12:42 +10:00
Damien George
8f064e469d py/emitbc: Fix bug with BC emitter computing Python stack size.
Previous to this patch the mp_emit_bc_adjust_stack_size function would
adjust the current stack size but would not increase the maximum stack size
if the current size went above it.  This meant that certain Python code
(eg a try-finally block with no statements inside it) would not have enough
Python stack allocated to it.

This patch fixes the problem by always checking if the current stack size
goes above the maximum, and adjusting the latter if it does.
2017-05-25 20:42:30 +10:00
Damien George
04d05db27e py/vm: Fix bug with unwind jump popping the iterator from a for loop.
This patch fixes a regression introduced by
088740ecc4
2017-05-25 20:39:08 +10:00
Damien George
68e71eacb8 py/vm: Fix bug with stackless mode and unwinding of exceptions.
This patch fixes a regression introduced by
71a3d6ec3b

Previous to this patch the n_state variable was referring to that computed
at the very start of the mp_execute_bytecode function.  This patch fixes it
so that n_state is recomputed when the code_state changes.
2017-05-25 20:35:57 +10:00
Damien George
338f0849d9 py/mkenv.mk: Use $(TOP) instead of ".." to reference tools, mpy-cross. 2017-05-19 15:53:55 +10:00
Damien George
03659c51ca py/objrange: Fix slicing of range when step of slice is negative. 2017-05-18 17:32:42 +10:00
Damien George
eb4c37f7a4 py/sequence: Fix boundary errors when slicing with a negative step. 2017-05-18 17:32:42 +10:00
Tom Collins
53461deb04 py/objstringio: Fix StringIO reads at or beyond EOF.
Existing code failed if seek() went past EOF (which is acceptable when writing).
2017-05-15 23:58:04 +03:00
Tom Collins
f06d0839bd py/modsys: update conditionals for code referencing sys.stdout
Working on a build with PY_IO enabled (for PY_UJSON support) but PY_SYS_STDFILES disabled (no filesystem).  There are multiple references to mp_sys_stdout_obj that should only be enabled if both PY_IO and PY_SYS_STDFILES are enabled.
2017-05-14 18:24:50 +03:00
Tom Collins
6f56412ec3 py/lexer: Process CR earlier to allow newlines checks on chr1.
Resolves an issue where lexer failed to accept CR after line continuation
character.  It also simplifies the code.
2017-05-12 15:14:24 +10:00
Damien George
a1f254028d py/mkrules.mk: Add dependency of .mpy files upon mpy-cross.
This ensures that mpy-cross is automatically built (and is up-to-date) for
ports that use frozen bytecode.  It also makes sure that .mpy files are
re-built if mpy-cross is changed.
2017-05-11 23:40:16 +10:00
Paul Sokolovsky
edc02bd952 unix/main: Implement -m option for packages. 2017-05-09 14:22:21 +03:00
Tom Collins
2998647c4e py/lexer: Simplify lexer startup by using dummy bytes and next_char().
Now consistently uses the EOL processing ("\r" and "\r\n" convert to "\n")
and EOF processing (ensure "\n" before EOF) provided by next_char().

In particular the lexer can now correctly handle input that starts with CR.
2017-05-09 14:43:23 +10:00
Damien George
6cfa61a4cc py/binary: Handle storing big-ints to all arrays types.
Prior to this patch only 'q' and 'Q' type arrays could store big-int
values.  With this patch any big int that is stored to an array is handled
by the big-int implementation, regardless of the typecode of the array.
This allows arrays to work with all type sizes on all architectures.
2017-05-09 10:41:00 +10:00
Paul Sokolovsky
4a4490ffcc py/modio: resource_stream: Implement "package" param handling. 2017-05-06 18:42:35 +03:00
Damien George
58bb73e010 py/objint: In int.from_bytes, only create big-int if really needed.
This patch ensures that int.from_bytes only creates a big-int if necessary,
by checking the value for a small-int overflow as it's being parsed.
2017-05-06 10:29:09 +10:00
Paul Sokolovsky
d7da2dba07 py/modio: Implement uio.resource_stream(package, resource_path).
The with semantics of this function is close to
pkg_resources.resource_stream() function from setuptools, which
is the canonical way to access non-source files belonging to a package
(resources), regardless of what medium the package uses (e.g. individual
source files vs zip archive). In the case of MicroPython, this function
allows to access resources which are frozen into the executable, besides
accessing resources in the file system.

This is initial stage of the implementation, which actually doesn't
implement "package" part of the semantics, just accesses frozen resources
from "root", or filesystem resource - from current dir.
2017-05-03 01:47:08 +03:00
stijn
2f0ce2a6f5 py: Cleanup use of global DEBUG preprocessor definition
The standard preprocessor definition to differentiate debug and non-debug
builds is NDEBUG, not DEBUG, so don't rely on the latter:
- just delete the use of it in objint_longlong.c as it has been stale code
  for years anyway (since commit [c4029e5]): SUFFIX isn't used anywhere.
- replace DEBUG with MICROPY_DEBUG_NLR in nlr.h: it is rarely used anymore
  so can be off by default
2017-04-30 14:28:37 +03:00
Damien George
f85fd79c6c py/mpz: In mpn_sub, use existing function to remove trailing zeros. 2017-04-25 12:22:04 +10:00
Damien George
c7aa86ce6f py/mpz: Strip trailing zeros from mpz value when set from bytes. 2017-04-25 12:06:10 +10:00
Damien George
dd11af209d py: Add LOAD_SUPER_METHOD bytecode to allow heap-free super meth calls.
This patch allows the following code to run without allocating on the heap:

    super().foo(...)

Before this patch such a call would allocate a super object on the heap and
then load the foo method and call it right away.  The super object is only
needed to perform the lookup of the method and not needed after that.  This
patch makes an optimisation to allocate the super object on the C stack and
discard it right after use.

Changes in code size due to this patch are:

   bare-arm: +128
    minimal: +232
   unix x64: +416
unix nanbox: +364
     stmhal: +184
    esp8266: +340
     cc3200: +128
2017-04-22 23:39:20 +10:00
Damien George
5335942b59 py/compile: Refactor handling of special super() call.
This patch refactors the handling of the special super() call within the
compiler.  It removes the need for a global (to the compiler) state variable
which keeps track of whether the subject of an expression is super.  The
handling of super() is now done entirely within one function, which makes
the compiler a bit cleaner and allows to easily add more optimisations to
super calls.

Changes to the code size are:

   bare-arm: +12
    minimal:  +0
   unix x64: +48
unix nanbox: -16
     stmhal:  +4
     cc3200:  +0
    esp8266: -56
2017-04-22 21:46:32 +10:00
Damien George
0dd6a59c89 py/compile: Don't do unnecessary check if iter parse node is a struct.
If we get to this point in the code then pn_iter is guaranteed to be a
struct.
2017-04-22 21:43:42 +10:00
Damien George
ae54fbf166 py/compile: Add COMP_RETURN_IF_EXPR option to enable return-if-else opt.
With this optimisation enabled the compiler optimises the if-else
expression within a return statement.  The optimisation reduces bytecode
size by 2 bytes for each use of such a return-if-else statement.  Since
such a statement is not often used, and costs bytes for the code, the
feature is disabled by default.

For example the following code:

    def f(x):
        return 1 if x else 2

compiles to this bytecode with the optimisation disabled (left column is
bytecode offset in bytes):

    00 LOAD_FAST 0
    01 POP_JUMP_IF_FALSE 8
    04 LOAD_CONST_SMALL_INT 1
    05 JUMP 9
    08 LOAD_CONST_SMALL_INT 2
    09 RETURN_VALUE

and to this bytecode with the optimisation enabled:

    00 LOAD_FAST 0
    01 POP_JUMP_IF_FALSE 6
    04 LOAD_CONST_SMALL_INT 1
    05 RETURN_VALUE
    06 LOAD_CONST_SMALL_INT 2
    07 RETURN_VALUE

So the JUMP to RETURN_VALUE is optimised and replaced by RETURN_VALUE,
saving 2 bytes and making the code a bit faster.
2017-04-22 14:58:01 +10:00
Damien George
40b40ffc98 py/compile: Extract parse-node kind at start of func for efficiency.
Otherwise the type of parse-node and its kind has to be re-extracted
multiple times.  This optimisation reduces code size by a bit (16 bytes on
bare-arm).
2017-04-22 14:23:47 +10:00
Damien George
fa03bbf0fd py/compile: Don't do unnecessary check if parse node is a struct.
PN_atom_expr_normal parse nodes always have structs for their second
sub-node, so simplify the check for the sub-node kind to save code size.
2017-04-22 14:13:37 +10:00
Damien George
4df013c8cc py/objtype: mp_obj_new_super doesn't need to be public, so inline it.
Saves code size (20 bytes on bare-arm) and makes it a tiny bit more
efficient.
2017-04-22 12:14:04 +10:00
Paul Sokolovsky
9e8f316392 extmod/moductypes: Fix bigint handling for 32-bit ports. 2017-04-21 16:43:21 +03:00
Damien George
7a72c0db5a py: Reduce str/repr precision of float numbers when floats are 30-bit.
With 30-bit floats there aren't enough bits to faithfully print 7 decimal
digits, so reduce the precision to 6 digits.
2017-04-21 16:21:56 +10:00
Damien George
bbb4b9822f py/modmicropython: Add micropython.kbd_intr() function.
It controls the character that's used to (asynchronously) raise a
KeyboardInterrupt exception.  Passing "-1" allows to disable the
interception of the interrupt character (as long as a port allows such a
behaviour).
2017-04-18 17:24:30 +10:00
Damien George
c7e8c6f7de py/gc: Execute finaliser code in a protected environment.
If a finaliser raises an exception then it must not propagate through the
GC sweep function.  This patch protects against such a thing by running
finaliser code via the mp_call_function_1_protected call.

This patch also adds scheduler lock/unlock calls around the finaliser
execution to further protect against any possible reentrancy issues: the
memory manager is already locked when doing a collection, but we also don't
want to allow any scheduled code to run, KeyboardInterrupts to interupt the
code, nor threads to switch.
2017-04-12 13:52:04 +10:00
Damien George
08242eed26 py/nlrsetjmp: Add check for failed NLR jump.
Also optimise the function so it only needs to call the MP_STATE_THREAD
macro once (following how other nlr code is written).
2017-04-12 13:50:31 +10:00
Damien George
a73501b1d6 py/objfloat: Add implementation of high-quality float hashing.
Disabled by default.
2017-04-12 13:38:17 +10:00
Damien George
816413e4b2 py: Optimise types for common case where type has a single parent type.
The common cases for inheritance are 0 or 1 parent types, for both built-in
types (eg built-in exceptions) as well as user defined types.  So it makes
sense to optimise the case of 1 parent type by storing just the type and
not a tuple of 1 value (that value being the single parent type).

This patch makes such an optimisation.  Even though there is a bit more
code to handle the two cases (either a single type or a tuple with 2 or
more values) it helps reduce overall code size because it eliminates the
need to create a static tuple to hold single parents (eg for the built-in
exceptions).  It also helps reduce RAM usage for user defined types that
only derive from a single parent.

Changes in code size (in bytes) due to this patch:

    bare-arm:       -16
    minimal (x86): -176
    unix (x86-64): -320
    unix nanbox:   -384
    stmhal:         -64
    cc3200:         -32
    esp8266:       -108
2017-04-12 13:22:21 +10:00
Damien George
fc710169b7 py/obj: Clean up and add comments describing mp_obj_type_t struct. 2017-04-12 13:20:26 +10:00
Damien George
e31fbd9b41 py/objint: Use unsigned arithmetic when formatting an integer.
Otherwise the edge case of the most negative integer value will not convert
correctly.
2017-04-11 15:18:35 +10:00
Damien George
f66df1efc8 py/objint: Extract small int value directly because type is known. 2017-04-11 15:16:09 +10:00
Damien George
79ce664952 py/runtime: When init'ing kbd intr exc, use tuple ptr instead of object. 2017-04-10 17:07:26 +10:00
Damien George
ee86de1f1a py: Make sure that static emg-exc-buffer is aligned to size of mp_obj_t.
This buffer is used to allocate objects temporarily, and such objects
require that their underlying memory be correctly aligned for their data
type.  Aligning for mp_obj_t should be sufficient for emergency exceptions,
but in general the memory buffer should aligned to the maximum alignment of
the machine (eg on a 32-bit machine with mp_obj_t being 4 bytes, a double
may not be correctly aligned).

This patch fixes a bug for certain nan-boxing builds, where mp_obj_t is 8
bytes and must be aligned to 8 bytes (even though the machine is 32 bit).
2017-04-10 16:02:56 +10:00
Damien George
81d2ca2b12 py/objtuple: Add support for inplace add (same as normal add). 2017-04-05 12:38:02 +10:00
Damien George
de9b53695d py: Raise a ValueError if range() step is zero.
Following CPython.  Otherwise one gets either an infinite loop (if code is
optimised by the uPy compiler) or possibly a divide-by-zero CPU exception.
2017-04-05 10:50:26 +10:00
Damien George
fc245d1ca4 py/objint: Consolidate mp_obj_new_int_from_float to one implementation.
This reduces code duplication and allows to make mp_classify_fp_as_int
static, which reduces code size.
2017-04-04 16:45:49 +10:00
Damien George
19f2e47d59 py: Add very simple but correct hashing for float and complex numbers.
Hashing of float and complex numbers that are exact (real) integers should
return the same integer hash value as hashing the corresponding integer
value.  Eg hash(1), hash(1.0) and hash(1+0j) should all be the same (this
is how Python is specified: if x==y then hash(x)==hash(y)).

This patch implements the simplest way of doing float/complex hashing by
just converting the value to int and returning that value.
2017-04-04 11:57:21 +10:00
Paul Sokolovsky
9a973977bb py/objstr: Use MICROPY_FULL_CHECKS for range checking when constructing bytes.
Split this setting from MICROPY_CPYTHON_COMPAT. The idea is to be able to
keep MICROPY_CPYTHON_COMPAT disabled, but still pass more of regression
testsuite. In particular, this fixes last failing test in basics/ for
Zephyr port.
2017-04-02 21:20:07 +03:00
Damien George
a5500a8aad py/obj.h: Make sequence grow more efficient and support overlapping.
The first memmove now copies less bytes in some cases (because len_adj <=
slice_len), and the memcpy is replaced with memmove to support the
possibility that dest and slice regions are overlapping.
2017-04-02 17:28:24 +10:00
Damien George
4c307bfba1 all: Move BYTES_PER_WORD definition from ports to py/mpconfig.h
It can still be overwritten by a port in mpconfigport.h but for almost
all cases one can use the provided default.
2017-04-01 11:39:38 +11:00
Damien George
b6c7e4b143 all: Use full path name when including mp-readline/timeutils/netutils.
This follows the pattern of how all other headers are now included, and
makes it explicit where the header file comes from.  This patch also
removes -I options from Makefile's that specify the mp-readline/timeutils/
netutils directories, which are no longer needed.
2017-03-31 22:29:39 +11:00
Damien George
4b753515c7 py/objzip: Convert mp_uint_t to size_t. 2017-03-30 22:58:08 +11:00
Paul Sokolovsky
29dd92c82a zephyr/Makefile: Rework to use modern, official build integration.
Build happens in 3 stages:

1. Zephyr config header and make vars are generated from prj.conf.
2. libmicropython is built using them.
3. Zephyr is built and final link happens.
2017-03-30 00:08:27 +03:00
Damien George
6b34107537 py: Change mp_uint_t to size_t for mp_obj_str_get_data len arg. 2017-03-29 12:56:45 +11:00
Damien George
6213ad7f46 py: Convert mp_uint_t to size_t for tuple/list accessors.
This patch changes mp_uint_t to size_t for the len argument of the
following public facing C functions:

mp_obj_tuple_get
mp_obj_list_get
mp_obj_get_array

These functions take a pointer to the len argument (to be filled in by the
function) and callers of these functions should update their code so the
type of len is changed to size_t.  For ports that don't use nan-boxing
there should be no change in generate code because the size of the type
remains the same (word sized), and in a lot of cases there won't even be a
compiler warning if the type remains as mp_uint_t.

The reason for this change is to standardise on the use of size_t for
variables that count memory (or memory related) sizes/lengths.  It helps
builds that use nan-boxing.
2017-03-29 12:56:17 +11:00
Damien George
f9b0e644e5 py/compile: Provide terse error message for invalid dict/set literals. 2017-03-29 12:44:27 +11:00
Damien George
18c059febf py: Shorten a couple of error messages. 2017-03-29 12:36:46 +11:00
Damien George
f55a059e7a py/compile: Simplify syntax-error messages for illegal assignments.
With this patch all illegal assignments are reported as "can't assign to
expression".  Before the patch there were special cases for a literal on
the LHS, and for augmented assignments (eg +=), but it seems a waste of
bytes (and there are lots of bytes used in error messages) to spend on
distinguishing such errors which a user will rarely encounter.
2017-03-29 12:28:33 +11:00
Damien George
5010d1958f py/lexer: Simplify and reduce code size for operator tokenising.
By removing the 'E' code from the operator token encoding mini-language the
tokenising can be simplified.  The 'E' code was only used for the !=
operator which is now handled as a special case; the optimisations for the
general case more than make up for the addition of this single, special
case.  Furthermore, the . and ... operators can be handled in the same way
as != which reduces the code size a little further.

This simplification also removes a "goto".

Changes in code size for this patch are (measured in bytes):

bare-arm:       -48
minimal x86:    -64
unix x86-64:   -112
unix nanbox:    -64
stmhal:         -48
cc3200:         -48
esp8266:        -76
2017-03-29 10:56:52 +11:00
Damien George
94c41bb06f py: Use mp_raise_TypeError/mp_raise_ValueError helpers where possible.
Saves 168 bytes on bare-arm.
2017-03-28 22:37:26 +11:00
Damien George
54507f78ee py/objmap: Convert mp_uint_t to size_t. 2017-03-27 12:19:24 +11:00
Damien George
40c1272e55 py/compile: When compiling super(), handle closed-over self variable.
The self variable may be closed-over in the function, and in that case the
call to super() should load the contents of the closure cell using
LOAD_DEREF (before this patch it would just load the cell directly).
2017-03-27 11:27:08 +11:00
Damien George
a0973b09ce py/vm: Fix VM opcode tracing to print correct stack pointer.
Also const_table is now moved to the code_state->fun_bc structure.
2017-03-27 10:52:04 +11:00
Damien George
3f810daeb3 py/obj: Change mp_uint_t to size_t for mp_obj_get_array_fixed_n len arg. 2017-03-26 19:20:06 +11:00
Paul Sokolovsky
99866a00a0 py/nlrx86: Better check for Zephyr (requires 1.7). 2017-03-26 00:33:23 +03:00
Damien George
ab5689bc9d py/objarray: Use mp_obj_str_get_str instead of mp_obj_str_get_data. 2017-03-25 19:53:31 +11:00
Damien George
64a4f11b2d py: Remove MP_STATE_CTX, use MP_STATE_THREAD instead (it's an alias).
MP_STATE_CTX was recently aliased to MP_STATE_THREAD and can now be
removed.
2017-03-24 18:43:28 +11:00
Damien George
707f16b05c py: Use mp_locals/mp_globals accessor funcs instead of MP_STATE_CTX.
To improve maintainability of the code.
2017-03-24 18:41:11 +11:00
Damien George
f4ee1ba9b4 py/objnamedtuple: Use size_t where appropriate, instead of mp_uint_t. 2017-03-24 17:25:25 +11:00
Damien George
d1b93ced78 py/objtype: Use size_t where appropriate, instead of mp_uint_t or uint. 2017-03-24 16:58:13 +11:00
Damien George
c236ebfea7 py/modbuiltins: Allow round() to return a big int if necessary.
Previous to this patch, if the result of the round function overflowed a
small int, or was inf or nan, then a garbage value was returned.  With
this patch the correct big-int is returned if necessary and exceptions are
raised for inf or nan.
2017-03-24 11:00:45 +11:00
Damien George
125eae1ba3 py/modbuiltins: For round() builtin use nearbyint instead of round.
The C nearbyint function has exactly the semantics that Python's round()
requires, whereas C's round() requires extra steps to handle rounding of
numbers half way between integers.  So using nearbyint reduces code size
and potentially eliminates any source of errors in the handling of half-way
numbers.

Also, bare-metal implementations of nearbyint can be more efficient than
round, so further code size is saved (and efficiency improved).

nearbyint is provided in the C99 standard so it should be available on all
supported platforms.
2017-03-24 11:00:45 +11:00
Damien George
c073519ec8 py/objint: Handle special case of -0 when classifying fp as int.
Otherwise -0.0 is classified as a bigint, which for builds without bigints
will lead unexpectedly to an overflow.
2017-03-23 23:51:35 +11:00
Damien George
febeff4af4 py/modmath: Allow trunc/ceil/floor to return a big int if necessary.
Previous to this patch, if the result of the trunc/ceil/floor functions
overflowed a small int, or was inf or nan, then a garbage value was
returned.  With this patch the correct big-int is returned if necessary,
and exceptions are raised for inf or nan.
2017-03-23 22:57:08 +11:00
Damien George
f64a3e296e py/lexer: Remove obsolete comment, since lexer can now raise exceptions. 2017-03-23 16:40:24 +11:00
Damien George
60656eaea4 py: Define and use MP_OBJ_ITER_BUF_NSLOTS to get size of stack iter buf.
It improves readability of code and reduces the chance to make a mistake.

This patch also fixes a bug with nan-boxing builds by rounding up the
calculation of the new NSLOTS variable, giving the correct number of slots
(being 4) even if mp_obj_t is larger than the native machine size.
2017-03-23 16:36:08 +11:00
Damien George
507119f4d8 py/sequence: Convert mp_uint_t to size_t where appropriate. 2017-03-23 16:23:20 +11:00
Damien George
c88cfe165b py: Use size_t as len argument and return type of mp_get_index.
These values are used to compute memory addresses and so size_t is the
more appropriate type to use.
2017-03-23 16:17:40 +11:00
Damien George
58f23def55 py/bc: Provide better error message for an unexpected keyword argument.
Now, passing a keyword argument that is not expected will correctly report
that fact.  If normal or detailed error messages are enabled then the name
of the unexpected argument will be reported.

This patch decreases the code size of bare-arm and stmhal by 12 bytes, and
cc3200 by 8 bytes.  Other ports (minimal, unix, esp8266) remain the same in
code size.  For terse error message configuration this is because the new
message is shorter than the old one.  For normal (and detailed) error
message configuration this is because the new error message already exists
in py/objnamedtuple.c so there's no extra space in ROM needed for the
string.
2017-03-22 13:40:27 +11:00
Damien George
1a5c8d1053 py/vm: Don't release the GIL if the scheduler is locked.
The scheduler being locked general means we are running a scheduled
function, and switching to another thread violates that, so don't switch in
such a case (even though we technically could).

And if we are running a scheduled function then we want to finish it ASAP,
so we shouldn't switch to another thread.

Furthermore, ports with threading enabled will lock the scheduler during a
hard IRQ, and this patch to the VM will make sure that threads are not
switched during a hard IRQ (which would crash the VM).
2017-03-20 18:42:27 +11:00
Damien George
6e74d24f30 py: Add micropython.schedule() function and associated runtime code. 2017-03-20 15:20:26 +11:00
stijn
bf29fe2e13 py/objstr: Use better msg in bad implicit str/bytes conversion exception
Instead of always reporting some object cannot be implicitly be converted
to a 'str', even when it is a 'bytes' object, adjust the logic so that
when trying to convert str to bytes it is shown like that.
This will still report bad implicit conversion from e.g. 'int to bytes'
as 'int to str' but it will not result in the confusing
'can't convert 'str' object to str implicitly' anymore for calls like
b'somestring'.count('a').
2017-03-20 15:11:45 +11:00
Damien George
5640e6dacd py: Provide mp_decode_uint_value to help optimise stack usage.
This has a noticeable improvement on x86-64 and Thumb2 archs, where stack
usage is reduced by 2 machine words in the VM.
2017-03-17 16:50:19 +11:00
Damien George
71a3d6ec3b py: Reduce size of mp_code_state_t structure.
Instead of caching data that is constant (code_info, const_table and
n_state), store just a pointer to the underlying function object from which
this data can be derived.

This helps reduce stack usage for the case when the mp_code_state_t
structure is stored on the stack, as well as heap usage when it's stored
on the heap.

The downside is that the VM becomes a little more complex because it now
needs to derive the data from the underlying function object.  But this
doesn't impact the performance by much (if at all) because most of the
decoding of data is done outside the main opcode loop.  Measurements using
pystone show that little to no performance is lost.

This patch also fixes a nasty bug whereby the bytecode can be reclaimed by
the GC during execution.  With this patch there is always a pointer to the
function object held by the VM during execution, since it's stored in the
mp_code_state_t structure.
2017-03-17 16:39:13 +11:00
Damien George
d279bcff8a py/objstr: Fix eager optimisation of str/bytes addition.
The RHS can only be returned if it is the same type as the LHS.
2017-03-16 14:30:04 +11:00
Damien George
23a693ec2d py/mkrules.mk: Remove special check for "-B" in qstr auto generation.
When make is passed "-B" it seems that everything is considered out-of-date
and so $? expands to all prerequisites.  Thus there is no need for a
special check to see if $? is emtpy.
2017-03-15 21:50:48 +11:00
Damien George
d65371538d py/mpprint: Fix int formatting so "+" is printed for 0-valued integer. 2017-03-15 17:25:12 +11:00
Damien George
8cd4911e63 py/emitnative: Remove obsolete commented out code. 2017-03-15 10:03:22 +11:00
Damien George
a5a84e1f85 py/emitnative: Use assertions and mp_not_implemented correctly.
Assertions are used to check expressions that should always be true,
and mp_not_implemented is used for code that can be reached.
2017-03-14 23:05:40 +11:00
Damien George
d1ae6ae080 py/objint: Allow to print long-long ints without using the heap.
Some stack is allocated to format ints, and when the int implementation uses
long-long there should be additional stack allocated compared with the other
cases.  This patch uses the existing "fmt_int_t" type to determine the
amount of stack to allocate.
2017-03-14 14:54:20 +11:00
Damien George
1831034be1 py: Allow lexer to raise exceptions during construction.
This patch refactors the error handling in the lexer, to simplify it (ie
reduce code size).

A long time ago, when the lexer/parser/compiler were first written, the
lexer and parser were designed so they didn't use exceptions (ie nlr) to
report errors but rather returned an error code.  Over time that has
gradually changed, the parser in particular has more and more ways of
raising exceptions.  Also, the lexer never really handled all errors without
raising, eg there were some memory errors which could raise an exception
(and in these rare cases one would get a fatal nlr-not-handled fault).

This patch accepts the fact that the lexer can raise exceptions in some
cases and allows it to raise exceptions to handle all its errors, which are
for the most part just out-of-memory errors during construction of the
lexer.  This makes the lexer a bit simpler, and also the persistent code
stuff is simplified.

What this means for users of the lexer is that calls to it must be wrapped
in a nlr handler.  But all uses of the lexer already have such an nlr
handler for the parser (and compiler) so that doesn't put any extra burden
on the callers.
2017-03-14 11:52:05 +11:00
Paul Sokolovsky
776883cb80 py/objint_longlong: Implement mp_obj_int_from_bytes_impl().
This makes int.from_bytes() work for MICROPY_LONGINT_IMPL_LONGLONG.
2017-03-10 00:22:53 +01:00
Damien George
52b6764894 py/nlrx64: Fixes to support Mac OS.
Two independent fixes:
- need to prefix symbols referenced from asm with underscore;
- need to undo the C-function prelude.
2017-03-08 22:36:02 +11:00
Paul Sokolovsky
fd49ff9917 py/nlrx86: Add workaround for Zephyr.
Actually, this removes -fno-omit-frame-pointer workaround for Zephyr.
2017-03-07 16:48:09 +01:00
Krzysztof Blazewicz
7e480e8a30 py: Use mp_obj_get_array where sequence may be a tuple or a list. 2017-03-07 16:48:16 +11:00
Krzysztof Blazewicz
1215dc47e2 py/runtime.c: Remove optimization of '*a,=b', it caused a bug.
*a, = b should always make a copy of b, instead, before this patch
if b was a list it would copy only a reference to it.
2017-03-07 16:48:16 +11:00
Paul Sokolovsky
96aa3a3102 py/modsys: Use MP_SMALL_INT_MAX for sys.maxsize in case of LONGINT_IMPL_NONE.
INT_MAX used previosly is indeed max value for int, whereas on LP64
platforms, long is used for mp_int_t. Using MP_SMALL_INT_MAX is the
correct way to do it anyway.
2017-03-06 12:15:25 +01:00
Damien George
914648ce0e py/py.mk: Force nlr files to be compiled with -Os. 2017-03-06 17:13:43 +11:00
Damien George
f0dddb688d py/nlrx86: Convert from assembler to C file with inline asm. 2017-03-06 17:13:43 +11:00
Damien George
321848470c py/nlrx64: Convert from assembler to C file with inline asm. 2017-03-06 17:13:35 +11:00
Damien George
a85755aa22 py/nlrxtensa: Convert from assembler to C file with inline asm.
nlr_jump is a little bit inefficient because it now saves a register to
the stack.
2017-03-06 17:13:16 +11:00
Damien George
be3d7f91e5 py/nlr.h: Mark nlr_jump_fail as NORETURN. 2017-03-06 17:13:16 +11:00
Damien George
05fe66f68a py: Move locals/globals dicts to the thread-specific state.
Each threads needs to have its own private references to its current
locals/globals dicts, otherwise functions running within different
contexts (eg imported from different files) can behave very strangely.
2017-03-06 17:01:56 +11:00
Damien George
9275c18270 py/map: Fix bugs with deletion of elements from OrderedDict.
There were 2 bugs, now fixed by this patch:
- after deleting an element the len of the dict did not decrease by 1
- after deleting an element searching through the dict could lead to
  a seg fault due to there being an MP_OBJ_SENTINEL in the ordered array
2017-03-03 11:21:19 +11:00
Damien George
f4a12dca58 py/objarray: Disallow slice-assignment to read-only memoryview.
Also comes with a test for this.  Fixes issue #2904.
2017-02-27 16:09:57 +11:00
Paul Sokolovsky
4b3da60324 py/runtime: mp_raise_msg(): Accept NULL argument for message.
In this case, raise an exception without a message.

This would allow to shove few code bytes comparing to currently used
mp_raise_msg(..., "") pattern. (Actual savings depend on function code
alignment used by a particular platform.)
2017-02-24 09:57:25 -05:00
Damien George
f615d82d5b py/parse: Simplify handling of errors by raising them directly.
The parser was originally written to work without raising any exceptions
and instead return an error value to the caller.  But it's now required
that a call to the parser be wrapped in an nlr handler, so we may as well
make use of that fact and simplify the parser so that it doesn't need to
keep track of any memory errors that it had.  The parser anyway explicitly
raises an exception at the end if there was an error.

This patch simplifies the parser by letting the underlying memory
allocation functions raise an exception if they fail to allocate any
memory.  And if there is an error parsing the "<id> = const(<val>)" pattern
then that also raises an exception right away instead of trying to recover
gracefully and then raise.
2017-02-24 14:56:37 +11:00
Damien George
5255255fb9 py: Create str/bytes objects in the parser, not the compiler.
Previous to this patch any non-interned str/bytes objects would create a
special parse node that held a copy of the str/bytes data.  Then in the
compiler this data would be turned into a str/bytes object.  This actually
lead to 2 copies of the data, one in the parse node and one in the object.
The parse node's copy of the data would be freed at the end of the compile
stage but nevertheless it meant that the peak memory usage of the
parse/compile stage was higher than it needed to be (by an amount equal to
the number of bytes in all the non-interned str/bytes objects).

This patch changes the behaviour so that str/bytes objects are created
directly in the parser and the object stored in a const-object parse node
(which already exists for bignum, float and complex const objects).  This
reduces peak RAM usage of the parse/compile stage, simplifies the parser
and compiler, and reduces code size by about 170 bytes on Thumb2 archs,
and by about 300 bytes on Xtensa archs.
2017-02-24 13:43:43 +11:00
Damien George
74f4d2c659 py/parse: Allow parser/compiler consts to be bignums.
This patch allows uPy consts to be bignums, eg:

    X = const(1 << 100)

The infrastructure for consts to be a bignum (rather than restricted to
small integers) has been in place for a while, ever since constant folding
was upgraded to allow bignums.  It just required a small change (in this
patch) to enable it.
2017-02-24 13:03:44 +11:00
Damien George
b1b090255c py/moduerrno: Make list of errno codes configurable.
It's configurable by defining MICROPY_PY_UERRNO_LIST.  If this is not
defined then a default is provided.
2017-02-22 12:58:11 +11:00
Damien George
f563406d2e py/moduerrno: Make uerrno.errorcode dict configurable.
It's configured by MICROPY_PY_UERRNO_ERRORCODE and enabled by default
(since that's the behaviour before this patch).

Without this dict the lookup of errno codes to strings must use the
uerrno module itself.
2017-02-22 12:58:11 +11:00
Damien George
89267886cc py/objlist: For list slice assignment, allow RHS to be a tuple or list.
Before this patch, assigning anything other than a list would lead to a
crash.  Fixes issue #2886.
2017-02-20 15:09:59 +11:00
Damien George
bdebfaa4bf py/grammar: Remove unused rule.
Since the recent changes to string/bytes literal concatenation, this rule
is no longer used.
2017-02-17 12:48:45 +11:00
Damien George
5124a94067 py/lexer: Convert mp_uint_t to size_t where appropriate. 2017-02-17 12:44:24 +11:00
Damien George
534b7c368d py: Do adjacent str/bytes literal concatenation in lexer, not compiler.
It's much more efficient in RAM and code size to do implicit literal string
concatenation in the lexer, as opposed to the compiler.

RAM usage is reduced because the concatenation can be done right away in the
tokeniser by just accumulating the string/bytes literals into the lexer's
vstr.  Prior to this patch adjacent strings/bytes would create a parse tree
(one node per string/bytes) and then in the compiler a whole new chunk of
memory was allocated to store the concatenated string, which used more than
double the memory compared to just accumulating in the lexer.

This patch also significantly reduces code size:

bare-arm: -204
minimal:  -204
unix x64: -328
stmhal:   -208
esp8266:  -284
cc3200:   -224
2017-02-17 12:12:40 +11:00
Damien George
773278ec30 py/lexer: Simplify handling of line-continuation error.
Previous to this patch there was an explicit check for errors with line
continuation (where backslash was not immediately followed by a newline).

But this check is not necessary: if there is an error then the remaining
logic of the tokeniser will reject the backslash and correctly produce a
syntax error.
2017-02-17 11:30:14 +11:00
Damien George
ae43679792 py/lexer: Use strcmp to make keyword searching more efficient.
Since the table of keywords is sorted, we can use strcmp to do the search
and stop part way through the search if the comparison is less-than.

Because all tokens that are names are subject to this search, this
optimisation will improve the overall speed of the lexer when processing
a script.

The change also decreases code size by a little bit because we now use
strcmp instead of the custom str_strn_equal function.
2017-02-17 11:10:35 +11:00
Damien George
a68c754688 py/lexer: Move check for keyword to name-tokenising block.
Keywords only needs to be searched for if the token is a MP_TOKEN_NAME, so
we can move the seach to the part of the code that does the tokenising for
MP_TOKEN_NAME.
2017-02-17 10:59:57 +11:00
Damien George
98b3072da5 py/lexer: Simplify handling of indenting of very first token. 2017-02-17 10:56:06 +11:00
Damien George
6a11048af1 py/persistentcode: Bump .mpy version due to change in bytecode. 2017-02-17 00:19:34 +11:00
Damien George
c264414746 py/lexer: Don't generate string representation for period or ellipsis.
It's not needed.
2017-02-16 20:23:41 +11:00
Damien George
71019ae4f5 py/grammar: Group no-compile grammar rules together to shrink tables.
Grammar rules have 2 variants: ones that are attached to a specific
compile function which is called to compile that grammar node, and ones
that don't have a compile function and are instead just inspected to see
what form they take.

In the compiler there is a table of all grammar rules, with each entry
having a pointer to the associated compile function.  Those rules with no
compile function have a null pointer.  There are 120 such rules, so that's
120 words of essentially wasted code space.

By grouping together the compile vs no-compile rules we can put all the
no-compile rules at the end of the list of rules, and then we don't need
to store the null pointers.  We just have a truncated table and it's
guaranteed that when indexing this table we only index the first half,
the half with populated pointers.

This patch implements such a grouping by having a specific macro for the
compile vs no-compile grammar rules (DEF_RULE vs DEF_RULE_NC).  It saves
around 460 bytes of code on 32-bit archs.
2017-02-16 19:45:06 +11:00
Damien George
e6003f466e py: De-optimise some uses of mp_getiter, so they don't use the C stack.
In these cases the heap is anyway used to create a new object so no real
need to use the C stack for iterating.  It saves a few bytes of code size.
2017-02-16 19:11:34 +11:00
Damien George
4d2bab1444 py/compile: Optimise list/dict/set comprehensions to use stack iter. 2017-02-16 18:38:07 +11:00
Damien George
cb6300697c py/runtime: Optimise case of identity iterator so it doesn't alloc RAM. 2017-02-16 18:38:06 +11:00
Damien George
30b42dd72d py: Remove unused "use_stack" argument from for_iter_end emit function. 2017-02-16 18:38:06 +11:00
Damien George
088740ecc4 py: Optimise storage of iterator so it takes only 4 slots on Py stack. 2017-02-16 18:38:06 +11:00
Damien George
6e769da0da py: Make FOR_ITER opcode pop 1+4 slots from the stack when finished.
The extra 4 slots correspond to the iterator object stored on the stack.
2017-02-16 18:38:06 +11:00
Damien George
f4df3aaa72 py: Allow bytecode/native to put iter_buf on stack for simple for loops.
So that the "for x in it: ..." statement can now work without using the
heap (so long as the iterator argument fits in an iter_buf structure).
2017-02-16 18:38:06 +11:00
Damien George
ae8d867586 py: Add iter_buf to getiter type method.
Allows to iterate over the following without allocating on the heap:
- tuple
- list
- string, bytes
- bytearray, array
- dict (not dict.keys, dict.values, dict.items)
- set, frozenset

Allows to call the following without heap memory:
- all, any, min, max, sum

TODO: still need to allocate stack memory in bytecode for iter_buf.
2017-02-16 18:38:06 +11:00
Damien George
101886f529 py/vm: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George
da36f5232d py/objint: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George
fa5a591757 py/objexcept: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George
efa629028a py/objclosure: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George
dbcdb9f8d8 py/objfun: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
ccc5254224 py/objarray: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
c0d9500eee py/objstr: Convert mp_uint_t to size_t (and use int) where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
68cd3a93f0 py/objset: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
1ea2f7a8ce py/objdict: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
58d9eeb8d9 py/objlist: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
229823942c py/objtuple: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
891dc5c62c py/persistentcode: Replace mp_uint_t with size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George
6ed77bedbd py/mpz: Change type of "base" args from mp_uint_t to unsigned int. 2017-02-16 16:51:16 +11:00
Damien George
eb90edb5c0 py/mpz: Remove obsolete declaration of mpz_as_str_size. 2017-02-16 16:51:16 +11:00
Damien George
dcdcc43dad py/mpz: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:13 +11:00
Damien George
4e3bac2e42 py/runtime: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:13 +11:00
Damien George
f6c22a0679 py/vm: Add MICROPY_PY_THREAD_GIL_VM_DIVISOR option.
This improves efficiency of GIL release within the VM, by only doing the
release after a fixed number of jump-opcodes have executed in the current
thread.
2017-02-15 11:28:15 +11:00
Damien George
234f07f16c py/modthread: Use system-provided mutexs for _thread locks.
It's more efficient using the system mutexs instead of synthetic ones with
a busy-wait loop.  The system can do proper scheduling and blocking of the
threads waiting on the mutex.
2017-02-15 11:28:02 +11:00
Damien George
adc80b8f84 py/objtype: Replace non-ASCII single-quote char with ASCII version. 2017-02-14 20:55:31 +11:00
Damien George
cc2dbdd1fe py/emitbc: Produce correct line number info for large bytecode chunks.
Previous to this patch, for large chunks of bytecode that originated from
a single source-code line, the bytecode-line mapping would generate
something like (for 42 bytecode bytes and 1 line):

  BC_SKIP=31  LINE_SKIP=1
  BC_SKIP=11  LINE_SKIP=0

This would mean that any errors in the last 11 bytecode bytes would be
reported on the following line.  This patch fixes it to generate instead:

  BC_SKIP=31  LINE_SKIP=0
  BC_SKIP=11  LINE_SKIP=1
2017-02-10 11:58:10 +11:00
dmazzella
18e6569166 py/objtype: Implement __delattr__ and __setattr__.
This patch implements support for class methods __delattr__ and __setattr__
for customising attribute access.  It is controlled by the config option
MICROPY_PY_DELATTR_SETATTR and is disabled by default.
2017-02-09 12:40:15 +11:00
Dave Hylands
aa34c553ec py/nlr: Fix execstack builds for ARM.
It seems that the gcc toolchain on the RaspberryPi
likes %progbits instead of @progbits. I verified that
%progbits also works under x86, so this should
fix #2848 and fix #2842

I verified that unix and mpy-cross both compile
on my RaspberryPi and on my x64 machine.
2017-02-08 11:12:26 +11:00
Damien George
af622eb2a6 py/map: Change mp_uint_t to size_t where appropriate.
The internal map/set functions now use size_t exclusively for computing
addresses.  size_t is enough to reach all of available memory when
computing addresses so is the right type to use.  In particular, for
nanbox builds it saves quite a bit of code size and RAM compared to the
original use of mp_uint_t (which is 64-bits on nanbox builds).
2017-02-08 11:00:15 +11:00
Damien George
a25aa2bcc3 py/asmxtensa.h: Explicitly cast args to 32-bits so left-shift is legal.
For archs that have 16-bit pointers, the asmxtensa.h file can give compiler
warnings about left-shift being greater than the width of the type (due to
the inline functions in this header file).  Explicitly casting the
constants to uint32_t stops these warnings.
2017-02-08 10:48:51 +11:00
Damien George
bd04ed3e8a py/objcomplex: Fix typo in ternary expression.
This typo actually did the correct thing, but it was very obscure (came
about from think in terms of Python's "x if cond else y" expression).
2017-02-04 00:23:56 +11:00
Damien George
90ab191b65 py/objstr: Convert some instances of mp_uint_t to size_t. 2017-02-03 13:04:56 +11:00
Damien George
a19b5a01ce py/mpconfig.h: Move PY_BUILTINS_POW3 config option to diff part of file.
With so many config options it's good to (at least try to) keep them
grouped into logical sections.
2017-02-03 12:35:48 +11:00
Damien George
7317e34383 py/objstr: Give correct behaviour when passing a dict to %-formatting.
This patch fixes two main things:
- dicts can be printed directly using '%s' % dict
- %-formatting should not crash when passed a non-dict to, eg, '%(foo)s'
2017-02-03 12:13:44 +11:00
Nicko van Someren
df0117c8ae py: Added optimised support for 3-argument calls to builtin.pow()
Updated modbuiltin.c to add conditional support for 3-arg calls to
pow() using MICROPY_PY_BUILTINS_POW3 config parameter. Added support in
objint_mpz.c for for optimised implementation.
2017-02-02 22:23:10 +03:00
Damien George
2486c4ff46 py/objset: Fix inplace binary ops so frozensets are not modified. 2017-02-03 00:27:56 +11:00
Damien George
3ed0e5e5d4 py/objcomplex: Correctly handle case of 0j to power of something.
0j to the power of negative now raises ZeroDivisionError, and 0j to the
power of positive returns 0.
2017-02-03 00:01:45 +11:00
Damien George
4b8ec5256d py/objfloat: Raise ZeroDivisionError for 0 to negative power. 2017-02-03 00:01:37 +11:00
Damien George
30cf503e97 py/objset: Make inplace binary operators actually modify the set. 2017-02-02 23:33:49 +11:00
Paul Sokolovsky
50d3a9df67 py/objstringio: Allow to specify initial capacity by passing numeric argument.
E.g. uio.BytesIO(100) will allocate buffer with 100 bytes of space.
2017-02-02 00:33:43 +03:00
Dave Hylands
aee74a1dae unix: Make stack be non-executable
This PR is to address issue #2812.
2017-02-01 23:03:10 +03:00
Damien George
1808b2e8d5 extmod: Remove MICROPY_FSUSERMOUNT and related files.
Replaced by MICROPY_VFS and the VFS sub-system.
2017-01-30 12:26:07 +11:00
Damien George
8beba7310f extmod/vfs_fat: Remove MICROPY_READER_FATFS component. 2017-01-30 12:26:07 +11:00
Paul Sokolovsky
7a7516d40d extmod/machine_signal: Implement "signal" abstraction for machine module.
A signal is like a pin, but ca also be inverted (active low). As such, it
abstracts properties of various physical devices, like LEDs, buttons,
relays, buzzers, etc. To instantiate a Signal:

pin = machine.Pin(...)
signal = machine.Signal(pin, inverted=True)

signal has the same .value() and __call__() methods as a pin.
2017-01-29 18:57:36 +03:00
Damien George
3f6b4e08e3 extmod/vfs: Expose mp_vfs_mount_t type.
It should only be used for low-level things and with caution, for example
putting mounted VFS data in ROM or the static data section.
2017-01-27 23:22:15 +11:00
Damien George
dcb9ea7215 extmod: Add generic VFS sub-system.
This provides mp_vfs_XXX functions (eg mount, open, listdir) which are
agnostic to the underlying filesystem type, and just require an object with
the relevant filesystem-like methods (eg .mount, .open, .listidr) which can
then be mounted.

These mp_vfs_XXX functions would typically be used by a port to implement
the "uos" module, and mp_vfs_open would be the builtin open function.

This feature is controlled by MICROPY_VFS, disabled by default.
2017-01-27 17:19:06 +11:00
Damien George
d4464b0050 py/py.mk: Add CFLAGS_MOD flag to set config file for FatFs. 2017-01-27 13:19:10 +11:00
Damien George
cc4c1adf6e py/showbc: Make sure to set the const_table before printing bytecode. 2017-01-27 12:34:09 +11:00
Paul Sokolovsky
e2e663291d py/objstr: Optimize string concatenation with empty string.
In this, don't allocate copy, just return non-empty string. This helps
with a standard pattern of buffering data in case of short reads:

    buf = b""
    while ...:
        s = f.read(...)
        buf += s
        ...

For a typical case when single read returns all data needed, there won't
be extra allocation. This optimization helps uasyncio.
2017-01-27 00:49:39 +03:00
Damien George
e9cb1f8077 py/objmodule: Move module init/deinit code into runtime functions.
They are one-line functions and having them inline in mp_init/mp_deinit
eliminates the overhead of a function call, and matches how other state
is initialised in mp_init.
2017-01-26 23:30:38 +11:00
Damien George
eaa77455c3 py/objint: Fix left-shift overflow in checking for large int. 2017-01-25 14:39:13 +11:00