Commit Graph

385 Commits

Author SHA1 Message Date
Damien George 1e70fda69f py/compile: Raise SyntaxError if positional args are given after */**.
In CPython 3.4 this raises a SyntaxError.  In CPython 3.5+ having a
positional after * is allowed but uPy has the wrong semantics and passes
the arguments in the incorrect order.  To prevent incorrect use of a
function going unnoticed it is important to raise the SyntaxError in uPy,
until the behaviour is fixed to follow CPython 3.5+.
2017-06-14 18:18:01 +10:00
Ville Skyttä ca16c38210 various: Spelling fixes 2017-05-29 11:36:05 +03:00
Damien George dd11af209d py: Add LOAD_SUPER_METHOD bytecode to allow heap-free super meth calls.
This patch allows the following code to run without allocating on the heap:

    super().foo(...)

Before this patch such a call would allocate a super object on the heap and
then load the foo method and call it right away.  The super object is only
needed to perform the lookup of the method and not needed after that.  This
patch makes an optimisation to allocate the super object on the C stack and
discard it right after use.

Changes in code size due to this patch are:

   bare-arm: +128
    minimal: +232
   unix x64: +416
unix nanbox: +364
     stmhal: +184
    esp8266: +340
     cc3200: +128
2017-04-22 23:39:20 +10:00
Damien George 5335942b59 py/compile: Refactor handling of special super() call.
This patch refactors the handling of the special super() call within the
compiler.  It removes the need for a global (to the compiler) state variable
which keeps track of whether the subject of an expression is super.  The
handling of super() is now done entirely within one function, which makes
the compiler a bit cleaner and allows to easily add more optimisations to
super calls.

Changes to the code size are:

   bare-arm: +12
    minimal:  +0
   unix x64: +48
unix nanbox: -16
     stmhal:  +4
     cc3200:  +0
    esp8266: -56
2017-04-22 21:46:32 +10:00
Damien George 0dd6a59c89 py/compile: Don't do unnecessary check if iter parse node is a struct.
If we get to this point in the code then pn_iter is guaranteed to be a
struct.
2017-04-22 21:43:42 +10:00
Damien George ae54fbf166 py/compile: Add COMP_RETURN_IF_EXPR option to enable return-if-else opt.
With this optimisation enabled the compiler optimises the if-else
expression within a return statement.  The optimisation reduces bytecode
size by 2 bytes for each use of such a return-if-else statement.  Since
such a statement is not often used, and costs bytes for the code, the
feature is disabled by default.

For example the following code:

    def f(x):
        return 1 if x else 2

compiles to this bytecode with the optimisation disabled (left column is
bytecode offset in bytes):

    00 LOAD_FAST 0
    01 POP_JUMP_IF_FALSE 8
    04 LOAD_CONST_SMALL_INT 1
    05 JUMP 9
    08 LOAD_CONST_SMALL_INT 2
    09 RETURN_VALUE

and to this bytecode with the optimisation enabled:

    00 LOAD_FAST 0
    01 POP_JUMP_IF_FALSE 6
    04 LOAD_CONST_SMALL_INT 1
    05 RETURN_VALUE
    06 LOAD_CONST_SMALL_INT 2
    07 RETURN_VALUE

So the JUMP to RETURN_VALUE is optimised and replaced by RETURN_VALUE,
saving 2 bytes and making the code a bit faster.
2017-04-22 14:58:01 +10:00
Damien George 40b40ffc98 py/compile: Extract parse-node kind at start of func for efficiency.
Otherwise the type of parse-node and its kind has to be re-extracted
multiple times.  This optimisation reduces code size by a bit (16 bytes on
bare-arm).
2017-04-22 14:23:47 +10:00
Damien George fa03bbf0fd py/compile: Don't do unnecessary check if parse node is a struct.
PN_atom_expr_normal parse nodes always have structs for their second
sub-node, so simplify the check for the sub-node kind to save code size.
2017-04-22 14:13:37 +10:00
Damien George de9b53695d py: Raise a ValueError if range() step is zero.
Following CPython.  Otherwise one gets either an infinite loop (if code is
optimised by the uPy compiler) or possibly a divide-by-zero CPU exception.
2017-04-05 10:50:26 +10:00
Damien George f9b0e644e5 py/compile: Provide terse error message for invalid dict/set literals. 2017-03-29 12:44:27 +11:00
Damien George 18c059febf py: Shorten a couple of error messages. 2017-03-29 12:36:46 +11:00
Damien George f55a059e7a py/compile: Simplify syntax-error messages for illegal assignments.
With this patch all illegal assignments are reported as "can't assign to
expression".  Before the patch there were special cases for a literal on
the LHS, and for augmented assignments (eg +=), but it seems a waste of
bytes (and there are lots of bytes used in error messages) to spend on
distinguishing such errors which a user will rarely encounter.
2017-03-29 12:28:33 +11:00
Damien George 40c1272e55 py/compile: When compiling super(), handle closed-over self variable.
The self variable may be closed-over in the function, and in that case the
call to super() should load the contents of the closure cell using
LOAD_DEREF (before this patch it would just load the cell directly).
2017-03-27 11:27:08 +11:00
Damien George 60656eaea4 py: Define and use MP_OBJ_ITER_BUF_NSLOTS to get size of stack iter buf.
It improves readability of code and reduces the chance to make a mistake.

This patch also fixes a bug with nan-boxing builds by rounding up the
calculation of the new NSLOTS variable, giving the correct number of slots
(being 4) even if mp_obj_t is larger than the native machine size.
2017-03-23 16:36:08 +11:00
Damien George 5255255fb9 py: Create str/bytes objects in the parser, not the compiler.
Previous to this patch any non-interned str/bytes objects would create a
special parse node that held a copy of the str/bytes data.  Then in the
compiler this data would be turned into a str/bytes object.  This actually
lead to 2 copies of the data, one in the parse node and one in the object.
The parse node's copy of the data would be freed at the end of the compile
stage but nevertheless it meant that the peak memory usage of the
parse/compile stage was higher than it needed to be (by an amount equal to
the number of bytes in all the non-interned str/bytes objects).

This patch changes the behaviour so that str/bytes objects are created
directly in the parser and the object stored in a const-object parse node
(which already exists for bignum, float and complex const objects).  This
reduces peak RAM usage of the parse/compile stage, simplifies the parser
and compiler, and reduces code size by about 170 bytes on Thumb2 archs,
and by about 300 bytes on Xtensa archs.
2017-02-24 13:43:43 +11:00
Damien George 534b7c368d py: Do adjacent str/bytes literal concatenation in lexer, not compiler.
It's much more efficient in RAM and code size to do implicit literal string
concatenation in the lexer, as opposed to the compiler.

RAM usage is reduced because the concatenation can be done right away in the
tokeniser by just accumulating the string/bytes literals into the lexer's
vstr.  Prior to this patch adjacent strings/bytes would create a parse tree
(one node per string/bytes) and then in the compiler a whole new chunk of
memory was allocated to store the concatenated string, which used more than
double the memory compared to just accumulating in the lexer.

This patch also significantly reduces code size:

bare-arm: -204
minimal:  -204
unix x64: -328
stmhal:   -208
esp8266:  -284
cc3200:   -224
2017-02-17 12:12:40 +11:00
Damien George 71019ae4f5 py/grammar: Group no-compile grammar rules together to shrink tables.
Grammar rules have 2 variants: ones that are attached to a specific
compile function which is called to compile that grammar node, and ones
that don't have a compile function and are instead just inspected to see
what form they take.

In the compiler there is a table of all grammar rules, with each entry
having a pointer to the associated compile function.  Those rules with no
compile function have a null pointer.  There are 120 such rules, so that's
120 words of essentially wasted code space.

By grouping together the compile vs no-compile rules we can put all the
no-compile rules at the end of the list of rules, and then we don't need
to store the null pointers.  We just have a truncated table and it's
guaranteed that when indexing this table we only index the first half,
the half with populated pointers.

This patch implements such a grouping by having a specific macro for the
compile vs no-compile grammar rules (DEF_RULE vs DEF_RULE_NC).  It saves
around 460 bytes of code on 32-bit archs.
2017-02-16 19:45:06 +11:00
Damien George 4d2bab1444 py/compile: Optimise list/dict/set comprehensions to use stack iter. 2017-02-16 18:38:07 +11:00
Damien George 30b42dd72d py: Remove unused "use_stack" argument from for_iter_end emit function. 2017-02-16 18:38:06 +11:00
Damien George 088740ecc4 py: Optimise storage of iterator so it takes only 4 slots on Py stack. 2017-02-16 18:38:06 +11:00
Damien George 6e769da0da py: Make FOR_ITER opcode pop 1+4 slots from the stack when finished.
The extra 4 slots correspond to the iterator object stored on the stack.
2017-02-16 18:38:06 +11:00
Damien George f4df3aaa72 py: Allow bytecode/native to put iter_buf on stack for simple for loops.
So that the "for x in it: ..." statement can now work without using the
heap (so long as the iterator argument fits in an iter_buf structure).
2017-02-16 18:38:06 +11:00
Damien George de9cd00b39 py/compile: Add an extra pass for Xtensa inline assembler.
It needs an extra pass to compute the size of the constant table for the
l32r instructions.
2016-12-19 17:42:25 +11:00
Damien George e920bab976 py/emitinline: Move common code for end of final pass to compiler.
This patch moves some common code from the individual inline assemblers to
the compiler, the code that calls the emit-glue to assign the machine code
to the functions scope.
2016-12-09 21:23:17 +11:00
Damien George dd53b12193 py/emitinline: Move inline-asm align and data methods to compiler.
These are generic methods that don't depend on the architecture and so
can be handled directly by the compiler.
2016-12-09 20:54:54 +11:00
Damien George f76b1bfa9f py: Add inline Xtensa assembler.
This patch adds the MICROPY_EMIT_INLINE_XTENSA option, which, when
enabled, allows the @micropython.asm_xtensa decorator to be used.

The following opcodes are currently supported (ax is a register, a0-a15):

    ret_n()
    callx0(ax)
    j(label)
    jx(ax)

    beqz(ax, label)
    bnez(ax, label)
    mov(ax, ay)
    movi(ax, imm) # imm can be full 32-bit, uses l32r if needed

    and_(ax, ay, az)
    or_(ax, ay, az)
    xor(ax, ay, az)
    add(ax, ay, az)
    sub(ax, ay, az)
    mull(ax, ay, az)

    l8ui(ax, ay, imm)
    l16ui(ax, ay, imm)
    l32i(ax, ay, imm)
    s8i(ax, ay, imm)
    s16i(ax, ay, imm)
    s32i(ax, ay, imm)
    l16si(ax, ay, imm)
    addi(ax, ay, imm)

    ball(ax, ay, label)
    bany(ax, ay, label)
    bbc(ax, ay, label)
    bbs(ax, ay, label)
    beq(ax, ay, label)
    bge(ax, ay, label)
    bgeu(ax, ay, label)
    blt(ax, ay, label)
    bnall(ax, ay, label)
    bne(ax, ay, label)
    bnone(ax, ay, label)

Upon entry to the assembly function the registers a0, a12, a13, a14 are
pushed to the stack and the stack pointer (a1) decreased by 16.  Upon
exit, these registers and the stack pointer are restored, and ret.n is
executed to return to the caller (caller address is in a0).

Note that the ABI for the Xtensa emitters is non-windowing.
2016-12-09 17:07:38 +11:00
Damien George ad297a1950 py: Allow inline-assembler emitter to be generic.
This patch refactors some code so that it is easier to integrate new
inline assemblers for different architectures other than ARM Thumb.
2016-12-09 17:06:21 +11:00
Damien George 8e5aced1fd py: Integrate Xtensa assembler into native emitter.
The config option MICROPY_EMIT_XTENSA can now be enabled to target the
Xtensa architecture with @micropython.native and @micropython.viper
decorators.
2016-12-09 16:51:49 +11:00
Damien George 080a78b15e py/compile: Simplify configuration of native emitter. 2016-12-07 11:17:17 +11:00
Damien George e6cf5fb2cc py/compile: Remove comment about TODO for short circuiting for if-stmt.
Short circuiting is handled correctly by c_if_cond, and constants within
short-circuit expressions are optimised by the parser.
2016-11-26 16:15:55 +11:00
Damien George ed9c93f0f1 py/parse: Make mp_parse_node_new_leaf an inline function.
It is split into 2 functions, one to make small ints and the other to make
a non-small-int leaf node.  This reduces code size by 32 bytes on
bare-arm, 64 bytes on unix (x64-64) and 144 bytes on stmhal.
2016-11-15 16:48:48 +11:00
Damien George b0cbfb0492 py/parse: Move function to check for const parse node to parse.[ch]. 2016-11-15 16:48:48 +11:00
Damien George deaa57acf3 py/compile: Remove debugging code for compiler dispatch.
It was a relic from the days of developing the compiler and is no longer
needed, and it's impossible to trigger via a test.
2016-10-12 10:20:48 +11:00
Damien George e49153fb98 py/compile: Remove unreachable code. 2016-10-11 12:29:54 +11:00
Damien George 216a711cd4 py/compile: Fix typo when checking for parse-node kind. 2016-09-30 14:48:06 +10:00
Damien George 0d10517a45 py/scope: Factor common code to find locals and close over them.
Saves 50-100 bytes of code.
2016-09-30 13:53:00 +10:00
Damien George 3dea8c9e92 py/scope: Use lookup-table to determine a scope's simple name.
Generates slightly smaller and more efficient code.
2016-09-30 12:34:05 +10:00
Damien George b32c01b748 py/compile: Fix async-for/async-with to work with simpler exc on stack.
There is now just the exception instance on the stack when an exception is
raised, not the full (type, exc, traceback).
2016-09-28 11:52:13 +10:00
Damien George f040685b0c py: Only store the exception instance on Py stack in bytecode try block.
When an exception is raised and is to be handled by the VM, it is stored
on the Python value stack so the bytecode can access it.  CPython stores
3 objects on the stack for each exception: exc type, exc instance and
traceback.  uPy followed this approach, but it turns out not to be
necessary.  Instead, it is enough to store just the exception instance on
the Python value stack.  The only place where the 3 values are needed
explicitly is for the __exit__ handler of a with-statement context, but
for these cases the 3 values can be extracted from the single exception
instance.

This patch removes the need to store 3 values on the stack, and instead
just stores the exception instance.

Code size is reduced by about 50-100 bytes, the compiler and VM are
slightly simpler, generate bytecode is smaller (by 2 bytes for each try
block), and the Python value stack is reduced in size for functions that
handle exceptions.
2016-09-27 12:37:21 +10:00
Damien George a5624bf381 py: Combine 3 comprehension emit functions (list/dict/set) into 1.
The 3 kinds of comprehensions are similar enough that merging their emit
functions reduces code size.  Decreases in code size in bytes are:
bare-arm:24, minimal:96, unix(NDEBUG,x86-64):328, stmhal:80, esp8266:76.
2016-09-19 12:23:31 +10:00
Damien George 24df30c133 py/compile: Don't compile assert statements when optimisations enabled.
As per CPython.
2016-08-26 22:28:22 +10:00
Damien George 3ff16ff52e py: Declare constant data as properly constant.
Otherwise some compilers (eg without optimisation) will put this read-only
data in RAM instead of ROM.
2016-05-20 12:46:20 +01:00
Damien George eacbd7aeba py: Fix constant folding and inline-asm to work with new async grammar. 2016-04-13 15:26:39 +01:00
pohmelie 81ebba7e02 py: add async/await/async for/async with syntax
They are sugar for marking function as generator, "yield from"
and pep492 python "semantically equivalents" respectively.

@dpgeorge was the original author of this patch, but @pohmelie made
changes to implement `async for` and `async with`.
2016-04-13 15:26:38 +01:00
Damien George 2c915e1ae6 py: Implement basic with support in native emitter. 2016-04-07 08:53:24 +01:00
Damien George ce8b4e8749 py: Combine continuous block of emit steps into with_cleanup emit call.
Because different emitters need to handle with-cleanup in different ways.
2016-04-07 08:50:38 +01:00
Damien George 3acaa28b52 py: Don't allocate an extra parse node for power exponent.
Previous to this patch, the "**b" in "a**b" had its own parse node with
just one item (the "b").  Now, the "b" is just the last element of the
power parse-node.  This saves (a tiny bit of) RAM when compiling.
2016-03-16 13:04:51 +00:00
Damien George ea23520403 py: Add MICROPY_DYNAMIC_COMPILER option to config compiler at runtime.
This new compile-time option allows to make the bytecode compiler
configurable at runtime by setting the fields in the mp_dynamic_compiler
structure.  By using this feature, the compiler can generate bytecode
that targets any MicroPython runtime/VM, regardless of the host and
target compile-time settings.

Options so far that fall under this dynamic setting are:
- maximum number of bits that a small int can hold;
- whether caching of lookups is used in the bytecode;
- whether to use unicode strings or not (lexer behaviour differs, and
  therefore generated string constants differ).
2016-02-25 10:05:46 +00:00
Damien George 8f54c08691 py/inlineasm: Add ability to specify return type of asm_thumb funcs.
Supported return types are: object, bool, int, uint.

For example:

@micropython.asm_thumb
def foo(r0, r1) -> uint:
    add(r0, r0, r1)
2016-01-27 14:27:10 +00:00
Damien George 93b3726240 py/parse: Optimise away parse node that's just parenthesis around expr.
Before this patch, (x+y)*z would be parsed to a tree that contained a
redundant identity parse node corresponding to the parenthesis.  With
this patch such nodes are optimised away, which reduces memory
requirements for expressions with parenthesis, and simplifies the
compiler because it doesn't need to handle this identity case.

A parenthesis parse node is still needed for tuples.
2016-01-07 13:07:52 +00:00
Damien George dd5353a405 py: Add MICROPY_ENABLE_COMPILER and MICROPY_PY_BUILTINS_EVAL_EXEC opts.
MICROPY_ENABLE_COMPILER can be used to enable/disable the entire compiler,
which is useful when only loading of pre-compiled bytecode is supported.
It is enabled by default.

MICROPY_PY_BUILTINS_EVAL_EXEC controls support of eval and exec builtin
functions.  By default they are only included if MICROPY_ENABLE_COMPILER
is enabled.

Disabling both options saves about 40k of code size on 32-bit x86.
2015-12-18 12:35:44 +00:00
Damien George a83124361e py/compile: Simplify compilation of comprehension iterators.
Saves 88 bytes on Thumb2, and 200 bytes on x86-64 archs.
2015-12-18 01:37:55 +00:00
Damien George 831137b807 py/compile: Use size_t or uintptr_t instead of mp_uint_t. 2015-12-17 13:13:18 +00:00
Damien George 29e9db0c58 py: Fix compiler to handle lambdas used as default arguments.
Addresses issue #1709.
2015-12-12 13:42:51 +00:00
Damien George 33ac0fd09f py: Don't try to optimise for+range when args are not simple expressions.
Addresses issue #1693.
2015-12-08 21:05:14 +00:00
Damien George b8cfb0d7b2 py: Add support for 64-bit NaN-boxing object model, on 32-bit machine.
To use, put the following in mpconfigport.h:

    #define MICROPY_OBJ_REPR (MICROPY_OBJ_REPR_D)
    #define MICROPY_FLOAT_IMPL (MICROPY_FLOAT_IMPL_DOUBLE)
    typedef int64_t mp_int_t;
    typedef uint64_t mp_uint_t;
    #define UINT_FMT "%llu"
    #define INT_FMT "%lld"

Currently does not work with native emitter enabled.
2015-11-29 14:25:36 +00:00
Damien George c3f64d9799 py: Change qstr_* functions to use size_t as the type for str len arg. 2015-11-29 14:25:04 +00:00
Damien George 9a56912ad1 py/compile: Do proper checking of * and ** in function definition.
This patch checks that there is only one *, and that ** is last in the
arg list.
2015-11-23 16:50:42 +00:00
Damien George d4dba88236 py/compile: Add mp_compile_to_raw_code() to return raw code object.
This can then be passed to mp_raw_code_save_file to save a .mpy file.
2015-11-20 12:30:37 +00:00
Damien George 2c83894257 py: Implement default and star args for lambdas. 2015-11-17 14:00:14 +00:00
Damien George cbd9ae5256 py/compile: Don't unnecessarily save state when compiling param list.
Parameter lists can't be nested so there is no need to save the global
state when compiling them.
2015-11-17 12:37:02 +00:00
Damien George 3a3db4dcf0 py: Put all bytecode state (arg count, etc) in bytecode. 2015-11-13 12:49:18 +00:00
Damien George d7e3b36a09 py/compile: Remove unnecessary label in compilation of for statement. 2015-10-14 15:51:12 +01:00
Damien George 64f2b213bb py: Move constant folding from compiler to parser.
It makes much more sense to do constant folding in the parser while the
parse tree is being built.  This eliminates the need to create parse
nodes that will just be folded away.  The code is slightly simpler and a
bit smaller as well.

Constant folding now has a configuration option,
MICROPY_COMP_CONST_FOLDING, which is enabled by default.
2015-10-12 12:58:45 +01:00
Damien George b948de36fb py: Don't generate unnecessary parse nodes for assignment or kwargs.
This patch eliminates the need for a nested parse node for assignments
and keyword arguments.  It saves a little bit of RAM when parsing.
2015-10-08 14:26:01 +01:00
Damien George 7e12a601b8 py/compile: Fix edge case when constant-folding negation of integer.
Also adds tests specifically for testing constant folding.
2015-10-08 13:02:00 +01:00
Damien George 0496de26d3 py: Allow to enable inline assembler without native emitter. 2015-10-03 17:07:54 +01:00
Damien George 58e0f4ac50 py: Allocate parse nodes in chunks to reduce fragmentation and RAM use.
With this patch parse nodes are allocated sequentially in chunks.  This
reduces fragmentation of the heap and prevents waste at the end of
individually allocated parse nodes.

Saves roughly 20% of RAM during parse stage.
2015-10-02 00:11:11 +01:00
Damien George e5635f4ab3 py: Catch all cases of integer (big and small) division by zero. 2015-10-01 22:48:48 +01:00
Damien George 9d5e5c08ab py/compile: Put compiler state on the C stack.
It's relatively small (between 44 and 56 bytes) and helps to reduce heap
pressure and fragmentation during compilation.
2015-09-24 13:15:57 +01:00
Damien George fbcaf0ea18 py: Slightly simplify compile and emit of star/double-star arguments.
Saves a few bytes of code space and eliminates need for rot_two
bytecode (hence saving RAM and execution time, by a tiny bit).
2015-09-23 11:47:01 +01:00
Delio Brignoli e6978a4e26 py: Fix call args when a stararg is followed by keyword args. 2015-09-23 11:37:00 +01:00
Damien George 558a016e2c py/compile: Refine SyntaxError for repeated use of global/nonlocal. 2015-09-07 16:55:02 +01:00
Damien George ea5b59bfe6 py/compile: Only compile function annotations if really needed.
Function annotations are only needed when the native emitter is enabled
and when the current scope is emitted in viper mode.  All other times
the annotations can be skipped completely.
2015-09-04 16:44:14 +01:00
Damien George 7f70b60f4d py: Remove unused compile scope flags, and irrelevant flag compute code. 2015-08-17 22:39:03 +01:00
Damien George 65dc960e3b unix-cpy: Remove unix-cpy. It's no longer needed.
unix-cpy was originally written to get semantic equivalent with CPython
without writing functional tests.  When writing the initial
implementation of uPy it was a long way between lexer and functional
tests, so the half-way test was to make sure that the bytecode was
correct.  The idea was that if the uPy bytecode matched CPython 1-1 then
uPy would be proper Python if the bytecodes acted correctly.  And having
matching bytecode meant that it was less likely to miss some deep
subtlety in the Python semantics that would require an architectural
change later on.

But that is all history and it no longer makes sense to retain the
ability to output CPython bytecode, because:

1. It outputs CPython 3.3 compatible bytecode.  CPython's bytecode
changes from version to version, and seems to have changed quite a bit
in 3.5.  There's no point in changing the bytecode output to match
CPython anymore.

2. uPy and CPy do different optimisations to the bytecode which makes it
harder to match.

3. The bytecode tests are not run.  They were never part of Travis and
are not run locally anymore.

4. The EMIT_CPYTHON option needs a lot of extra source code which adds
heaps of noise, especially in compile.c.

5. Now that there is an extensive test suite (which tests functionality)
there is no need to match the bytecode.  Some very subtle behaviour is
tested with the test suite and passing these tests is a much better
way to stay Python-language compliant, rather than trying to match
CPy bytecode.
2015-08-17 12:51:26 +01:00
Damien George cfc4c33801 py/compile: Give more precise line number for compile errors.
Previous to this patch there were some cases where line numbers for
errors were 0 (unknown).  Now the compiler attempts to give a better
line number where possible, in some cases giving the line number of the
closest statement, and other cases the line number of the inner-most
scope of the error (eg the line number of the start of the function).
This helps to give good (and sometimes exact) line numbers for
ViperTypeError exceptions.

This patch also makes sure that the first compile error (eg SyntaxError)
that is encountered is reported (previously it was the last one that was
reported).
2015-07-29 22:16:01 +00:00
Damien George 84d59c2873 py: For viper compile errors, add traceback with function and filename.
ViperTypeError now includes filename and function name where the error
occurred.  The line number is the line number of the start of the
function definition, which is the best that can be done without a lot
more work.

Partially addresses issue #1381.
2015-07-27 22:20:00 +01:00
Damien George 59fba2d6ea py: Remove mp_load_const_bytes and instead load precreated bytes object.
Previous to this patch each time a bytes object was referenced a new
instance (with the same data) was created.  With this patch a single
bytes object is created in the compiler and is loaded directly at execute
time as a true constant (similar to loading bignum and float objects).
This saves on allocating RAM and means that bytes objects can now be
used when the memory manager is locked (eg in interrupts).

The MP_BC_LOAD_CONST_BYTES bytecode was removed as part of this.

Generated bytecode is slightly larger due to storing a pointer to the
bytes object instead of the qstr identifier.

Code size is reduced by about 60 bytes on Thumb2 architectures.
2015-06-25 14:42:13 +00:00
Damien George c8b60f013b py: Make viper codegen raise proper exception (ViperTypeError) on error.
This fixes a long standing problem that viper code generation gave
terrible error messages, and actually no errors on pyboard where
assertions are disabled.

Now all compile-time errors are raised as proper Python exceptions, and
are of type ViperTypeError.

Addresses issue #940.
2015-04-20 13:29:31 +00:00
Damien George 55fe92bb8f py: Remove old debugging printf's in compile.c. 2015-04-11 21:52:27 +01:00
Damien George 91bc32dc16 py: Provide typedefs for function types instead of writing them inline. 2015-04-09 15:31:53 +00:00
Damien George 4dea922610 py: Adjust some spaces in code style/format, purely for consistency. 2015-04-09 15:29:54 +00:00
Damien George 4112590a60 py, compiler: When just bytecode, make explicit calls instead of table.
When just the bytecode emitter is needed there is no need to have a
dynamic method table for the emitter back-end, and we can instead
directly call the mp_emit_bc_XXX functions.  This gives a significant
reduction in code size and a very slight performance boost for the
compiler.

This patch saves 1160 bytes code on Thumb2 and 972 bytes on x86, when
native emitters are disabled.

Overall savings in code over the last 3 commits are:

bare-arm: 1664 bytes.
minimal:  2136 bytes.
stmhal:    584 bytes (it has native emitter enabled).
cc3200:   1736 bytes.
2015-03-26 16:52:45 +00:00
Damien George a210c774f9 py, compiler: Remove emit_pass1 code, using emit_bc to do its job.
First pass for the compiler is computing the scope (eg if an identifier
is local or not) and originally had an entire table of methods dedicated
to this, most of which did nothing.  With changes from previous commit,
this set of methods can be removed and the methods from the bytecode
emitter used instead, with very little modification -- this is what is
done in this commit.

This factoring has little to no impact on the speed of the compiler
(tested by compiling 3763 Python scripts and timing it).

This factoring reduces code size by about 270-300 bytes on Thumb2 archs,
and 400 bytes on x86.
2015-03-26 16:52:45 +00:00
Damien George 542bd6b4a1 py, compiler: Refactor load/store/delete_id logic to reduce code size.
Saves around 230 bytes on Thumb2 and 750 bytes on x86.
2015-03-26 16:52:45 +00:00
Damien George 44f65c0e2f py: Fix bug in compiler which allowed through illegal augmented assign.
It allowed such things as (a, b) += c.
2015-03-25 23:06:48 +00:00
Damien George aedf583af2 py: Simplify some logic in compiler; add comments about CPython compat. 2015-03-25 22:06:47 +00:00
Damien George 1ef26b35c1 py, extmod: Remove include of unnecessary system headers. 2015-03-14 23:11:25 +00:00
Damien George 42e0c59308 py: Add MICROPY_COMP_{DOUBLE,TRIPLE}_TUPLE_ASSIGN config options.
These allow to fine-tune the compiler to select whether it optimises
tuple assignments of the form a, b = c, d and a, b, c = d, e, f.
Sensible defaults are provided.
2015-03-14 13:11:35 +00:00
Damien George a77ffe66b2 py: In compiler, put macro guard around potentially unused asm vars. 2015-03-14 12:59:31 +00:00
Damien George 3665d0b2ff py: Simplify some inline-assembler error messages, but retain meaning.
Just to reduce code size.  Messages are still to the point and
unambiguous.
2015-03-03 17:11:18 +00:00
Damien George 9c5cabb502 py: Give error for duplicate label in inline assembler. 2015-03-03 17:08:02 +00:00
Damien George 62a3a287d9 py: Set compiler scope before folding constants so error messages work.
Addresses issue #1140.
2015-03-01 12:04:05 +00:00
Damien George 7711afbb4a py: Combine complie functions for or_test/and_test to reduce code size.
Saves around 60 bytes code on Thumb2 archs.
2015-02-28 15:10:18 +00:00
Damien George 63f3832e81 py: Combine emit functions for jump true/false to reduce code size.
Saves 116 bytes for stmhal and 56 bytes for cc3200 port.
2015-02-28 15:04:06 +00:00
Damien George 0b2fd91890 py: Combine logic for compiling and/or tests, to reduce code size.
Reduces code size by 72 bytes on Thumb2 archs.
2015-02-28 14:37:54 +00:00
Damien George 0bb971370b py: Transform assert logic in compiler to save code space.
Saves about 250 code bytes for Thumb2 archs.
2015-02-27 14:25:47 +00:00
Damien George 3d7bf5d4b1 py: More robust checking in inline assembler compiler. 2015-02-16 17:46:28 +00:00
Damien George dfe944c3e5 py: Expose compile.c:list_get as mp_parse_node_extract_list. 2015-02-13 02:29:46 +00:00
Damien George 8dfbd2d589 py: Make inline assembler raise proper SyntaxError exception on error.
Also gives line number of location of error.  Very useful!
2015-02-13 01:00:51 +00:00
Damien George 7d414a1b52 py: Parse big-int/float/imag constants directly in parser.
Previous to this patch, a big-int, float or imag constant was interned
(made into a qstr) and then parsed at runtime to create an object each
time it was needed.  This is wasteful in RAM and not efficient.  Now,
these constants are parsed straight away in the parser and turned into
objects.  This allows constants with large numbers of digits (so
addresses issue #1103) and takes us a step closer to #722.
2015-02-08 01:57:40 +00:00
Damien George 0bfc7638ba py: Protect mp_parse and mp_compile with nlr push/pop block.
To enable parsing constants more efficiently, mp_parse should be allowed
to raise an exception, and mp_compile can already raise a MemoryError.
So these functions need to be protected by an nlr push/pop block.

This patch adds that feature in all places.  This allows to simplify how
mp_parse and mp_compile are called: they now raise an exception if they
have an error and so explicit checking is not needed anymore.
2015-02-07 18:33:58 +00:00
Damien George 0d3cb6726d py: Change vstr so that it doesn't null terminate buffer by default.
This cleans up vstr so that it's a pure "variable buffer", and the user
can decide whether they need to add a terminating null byte.  In most
places where vstr is used, the vstr did not need to be null terminated
and so this patch saves code size, a tiny bit of RAM, and makes vstr
usage more efficient.  When null termination is needed it must be
done explicitly using vstr_null_terminate.
2015-01-28 23:43:01 +00:00
Damien George 05005f679e py: Remove mp_obj_str_builder and use vstr instead.
With this patch str/bytes construction is streamlined.  Always use a
vstr to build a str/bytes object.  If the size is known beforehand then
use vstr_init_len to allocate only required memory.  Otherwise use
vstr_init and the vstr will grow as needed.  Then use
mp_obj_new_str_from_vstr to create a str/bytes object using the vstr
memory.

Saves code ROM: 68 bytes on stmhal, 108 bytes on bare-arm, and 336 bytes
on unix x64.
2015-01-21 23:18:02 +00:00
Damien George ff8dd3f486 py, unix: Allow to compile with -Wunused-parameter.
See issue #699.
2015-01-20 12:47:20 +00:00
Damien George 50912e7f5d py, unix, stmhal: Allow to compile with -Wshadow.
See issue #699.
2015-01-20 11:55:10 +00:00
Damien George 963a5a3e82 py, unix: Allow to compile with -Wsign-compare.
See issue #699.
2015-01-16 17:47:07 +00:00
Damien George 0abb5609b0 py: Remove unnecessary id_flags argument from emitter's load_fast.
Saves 24 bytes in bare-arm.
2015-01-16 12:24:49 +00:00
Damien George d2d64f00fb py: Add "default" to switches to allow better code flow analysis.
This helps compiler produce smaller code.  Saves 124 bytes on stmhal and
bare-arm.
2015-01-14 21:32:42 +00:00
Damien George 65ef6b768c py: Only allocate strings/bytes once for load_const_obj. 2015-01-14 21:17:27 +00:00
Damien George 1e1779eacf py: Reluctantly add an extra pass to bytecode compiler.
Bytecode also needs a pass to compute the stack size.  This is because
the state size of the bytecode function is encoded as a variable uint,
so we must know the value of this uint before we encode it (otherwise
the size of the generated code changes from one pass to the next).

Having an entire pass for this seems wasteful (in time).  Alternative is
to allocate fixed space for the state size (would need 3-4 bytes to be
general, when 1 byte is usually sufficient) which uses a bit of extra
RAM per bytecode function, and makes the code less elegant in places
where this uint is encoded/decoded.

So, for now, opt for an extra pass.
2015-01-14 00:20:28 +00:00
Damien George c935d69f74 py: Make compiler not crash when default except is not last. 2015-01-13 23:33:16 +00:00
Damien George 4c81ba8015 py: Never intern data of large string/bytes object; add relevant tests.
Previously to this patch all constant string/bytes objects were
interned by the compiler, and this lead to crashes when the qstr was too
long (noticeable now that qstr length storage defaults to 1 byte).

With this patch, long string/bytes objects are never interned, and are
referenced directly as constant objects within generated code using
load_const_obj.
2015-01-13 16:21:23 +00:00
Damien George ddd1e18801 py: Add config option MICROPY_COMP_MODULE_CONST for module consts.
Compiler optimises lookup of module.CONST when enabled (an existing
feature).  Disabled by default; enabled for unix, windows, stmhal.
Costs about 100 bytes ROM on stmhal.
2015-01-10 14:07:24 +00:00
Damien George 51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Damien George 83204f3406 py: Allow to properly disable builtin slice operation.
This patch makes the MICROPY_PY_BUILTINS_SLICE compile-time option
fully disable the builtin slice operation (when set to 0).  This
includes removing the slice sytanx from the grammar.  Now, enabling
slice costs 4228 bytes on unix x64, and 1816 bytes on stmhal.
2014-12-27 17:33:30 +00:00
Damien George e37dcaafb4 py: Allow to properly disable builtin "set" object.
This patch makes MICROPY_PY_BUILTINS_SET compile-time option fully
disable the builtin set object (when set to 0).  This includes removing
set constructor/comprehension from the grammar, the compiler and the
emitters.  Now, enabling set costs 8168 bytes on unix x64, and 3576
bytes on stmhal.
2014-12-27 17:33:30 +00:00
Damien George 01039b5bd8 py: Remove last uses of printf from compile; use proper SyntaxError. 2014-12-21 17:44:27 +00:00
Damien George 584ba6762f py: Move global/nonlocal decl code to compiler for proper SyntaxError.
This patch gives proper SyntaxError exceptions for bad global/nonlocal
declarations.  It also reduces code size: 304 bytes on unix x64, 132
bytes on stmhal.
2014-12-21 17:26:45 +00:00
Damien George 6efa66f125 py: Remove unnecessary RULE_none and PN_none from parser. 2014-12-20 18:41:59 +00:00
Damien George e181c0dc07 py: Fix optimised for-loop compiler so it follows proper semantics.
You can now assign to the range end variable and the for-loop still
works correctly.  This fully addresses issue #565.

Also fixed a bug with the stack not being fully popped when breaking out
of an optimised for-loop (and it's actually impossible to write a test
for this case!).
2014-12-12 17:19:56 +00:00
Damien George c33ce606cf py: Fix a semantic issue with range optimisation.
Now you can assign to the range variable within the for loop and it will
still work.

Partially addresses issue #565.
2014-12-11 17:35:23 +00:00
Damien George 969a6b37bf py: Make functions static where appropriate. 2014-12-10 22:08:14 +00:00
Paul Sokolovsky 039887a0ac py: Fix bug with right-shifting small ints by large amounts.
Undefined behavior in C, needs explicit check.
2014-11-02 02:41:30 +02:00
Damien George 391db8669b py: Add more compiler optimisations for constant if/while conditions. 2014-10-17 17:57:33 +00:00
Damien George 235f9b33c8 py: Simplify compilation of elif blocks. 2014-10-17 17:30:16 +00:00
Damien George 090c9236e8 py: Fix compiling of nested while/for and exception handler.
Addresses issue #912.
2014-10-17 14:08:49 +00:00
Damien George a91ac2011f py: Make compiler return a proper exception on SyntaxError. 2014-10-05 19:01:34 +01:00
Damien George 00be7a849a py: Fix unix-cpy to compile with uint->mp_uint_t changes. 2014-10-03 20:05:44 +01:00
Damien George 39dc145478 py: Change [u]int to mp_[u]int_t in qstr.[ch], and some other places.
This should pretty much resolve issue #50.
2014-10-03 19:52:22 +01:00
Damien George 42f3de924b py: Convert [u]int to mp_[u]int_t where appropriate.
Addressing issue #50.
2014-10-03 17:44:14 +00:00
Damien George 52b5d76a6b py: Free non-interned strings in the parser when not needed.
mp_parse_node_free now frees the memory associated with non-interned
strings.  And the parser calls mp_parse_node_free when discarding a
non-used node (such as a doc string).

Also, the compiler now frees the parse tree explicitly just before it
exits (as opposed to relying on the caller to do this).

Addresses issue #708 as best we can.
2014-09-23 15:31:56 +00:00
Damien George 7ff996c237 py: Convert [u]int to mp_[u]int_t in emit.h and associated .c files.
Towards resolving issue #50.
2014-09-08 23:05:16 +01:00
Damien George c90f59ec3a py: Add support for emitting native x86 machine code. 2014-09-06 23:06:36 +01:00
Damien George dda46460ff Code style/whitespace cleanup; remove obsolete headers.
And move the MAP_ANON redefinition from py/asmx64.c to unix/alloc.c.
2014-09-03 22:47:23 +01:00
Damien George eb4e18f057 py: Add compiler optimisation for conditions in parenthesis.
Optimises:
    if () -> if False
    if (x,...) -> if True
    if (a and b) -> if a and b
2014-08-29 20:04:01 +01:00
Fabian Vogt fe3d16e8c2 Basic native ARM emitter 2014-08-27 18:18:50 +02:00
Damien George a5190a7dac py: Fix typing of viper locals; allow default types in annotation. 2014-08-15 22:39:08 +01:00
Damien George 2ac4af6946 py: Allow viper to have type annotations.
Viper functions can now be annotated with the type of their arguments
and return value.  Eg:

@micropython.viper
def f(x:int) -> int:
    return x + 1
2014-08-15 16:45:41 +01:00
Damien George 6be0b0a8ec py: Clean up and simplify functions in scope; add STATIC in compiler.
Some small code clean-ups that result in about 80 bytes ROM saving for
stmhal.
2014-08-15 14:30:52 +01:00
Paul Sokolovsky 8215847b4d moductypes: Foreign data interface module, roughly based on ctype ideas.
But much smaller and memory-efficient. Uses Python builtin data structures
(dict, tuple, int) to describe structure layout.
2014-07-09 19:28:24 +03:00
Damien George 40f3c02682 Rename machine_(u)int_t to mp_(u)int_t.
See discussion in issue #50.
2014-07-03 13:25:24 +01:00
Damien George b601d9574a py: Improvements to native emitter.
Native emitter can now compile try/except blocks using nlr_push/nlr_pop.
It probably only works for 1 level of exception handling.  It doesn't
work on Thumb (only x64).

Native emitter can also handle some additional op codes.

With this patch, 198 tests now pass using "-X emit=native" option to
micropython.
2014-06-30 05:17:25 +01:00
Paul Sokolovsky 59c675a64c py: Include mpconfig.h before all other includes.
It defines types used by all other headers.

Fixes #691.
2014-06-21 22:43:22 +03:00
Emmanuel Blot f6932d6506 Prefix ARRAY_SIZE with micropython prefix MP_ 2014-06-19 18:54:34 +02:00
Damien George f0778a7ccb py: Implement default keyword only args.
Should finish addressing issue #524.
2014-06-07 22:01:00 +01:00
Damien George 5b5562c1d1 py: Fix stack underflow with optimised for loop. 2014-05-31 17:59:11 +01:00
Damien George 25c84643b6 py: Fix break from within a for loop.
Needed to pop the iterator object when breaking out of a for loop.  Need
also to be careful to unwind exception handler before popping iterator.

Addresses issue #635.
2014-05-30 15:20:41 +01:00
Damien George d1e355ea8e py: Fix check of small-int overflow when parsing ints.
Also unifies use of SMALL_INT_FITS macro across parser and runtime.
2014-05-28 14:51:12 +01:00