Commit Graph

125 Commits

Author SHA1 Message Date
Damien George ea23520403 py: Add MICROPY_DYNAMIC_COMPILER option to config compiler at runtime.
This new compile-time option allows to make the bytecode compiler
configurable at runtime by setting the fields in the mp_dynamic_compiler
structure.  By using this feature, the compiler can generate bytecode
that targets any MicroPython runtime/VM, regardless of the host and
target compile-time settings.

Options so far that fall under this dynamic setting are:
- maximum number of bits that a small int can hold;
- whether caching of lookups is used in the bytecode;
- whether to use unicode strings or not (lexer behaviour differs, and
  therefore generated string constants differ).
2016-02-25 10:05:46 +00:00
Damien George dd5353a405 py: Add MICROPY_ENABLE_COMPILER and MICROPY_PY_BUILTINS_EVAL_EXEC opts.
MICROPY_ENABLE_COMPILER can be used to enable/disable the entire compiler,
which is useful when only loading of pre-compiled bytecode is supported.
It is enabled by default.

MICROPY_PY_BUILTINS_EVAL_EXEC controls support of eval and exec builtin
functions.  By default they are only included if MICROPY_ENABLE_COMPILER
is enabled.

Disabling both options saves about 40k of code size on 32-bit x86.
2015-12-18 12:35:44 +00:00
Damien George bdbe8c9ae2 py: Make UNARY_OP_NOT a first-class op, to agree with Py not semantics.
Fixes #1684 and makes "not" match Python semantics.  The code is also
simplified (the separate MP_BC_NOT opcode is removed) and the patch saves
68 bytes for bare-arm/ and 52 bytes for minimal/.

Previously "not x" was implemented as !mp_unary_op(x, MP_UNARY_OP_BOOL),
so any given object only needs to implement MP_UNARY_OP_BOOL (and the VM
had a special opcode to do the ! bit).

With this patch "not x" is implemented as mp_unary_op(x, MP_UNARY_OP_NOT),
but this operation is caught at the start of mp_unary_op and dispatched as
!mp_obj_is_true(x).  mp_obj_is_true has special logic to test for
truthness, and is the correct way to handle the not operation.
2015-12-10 22:19:48 +00:00
Damien George 999cedb90f py: Wrap all obj-ptr conversions in MP_OBJ_TO_PTR/MP_OBJ_FROM_PTR.
This allows the mp_obj_t type to be configured to something other than a
pointer-sized primitive type.

This patch also includes additional changes to allow the code to compile
when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of
mp_uint_t, and various casts.
2015-11-29 14:25:35 +00:00
Damien George 5d66b427e2 py/emit: Change type of arg of load_const_obj from void* to mp_obj_t. 2015-11-29 14:25:04 +00:00
Damien George d8c834c95d py: Add MICROPY_PERSISTENT_CODE_LOAD/SAVE to load/save bytecode.
MICROPY_PERSISTENT_CODE must be enabled, and then enabling
MICROPY_PERSISTENT_CODE_LOAD/SAVE (either or both) will allow loading
and/or saving of code (at the moment just bytecode) from/to a .mpy file.
2015-11-13 12:49:18 +00:00
Damien George c8e9c0d89a py: Add MICROPY_PERSISTENT_CODE so code can persist beyond the runtime.
Main changes when MICROPY_PERSISTENT_CODE is enabled are:

- qstrs are encoded as 2-byte fixed width in the bytecode
- all pointers are removed from bytecode and put in const_table (this
  includes const objects and raw code pointers)

Ultimately this option will enable persistence for not just bytecode but
also native code.
2015-11-13 12:49:18 +00:00
Damien George 713ea1800d py: Add constant table to bytecode.
Contains just argument names at the moment but makes it easy to add
arbitrary constants.
2015-11-13 12:49:18 +00:00
Damien George 3a3db4dcf0 py: Put all bytecode state (arg count, etc) in bytecode. 2015-11-13 12:49:18 +00:00
Damien George 9b7f583b0c py: Reorganise bytecode layout so it's more structured, easier to edit. 2015-11-13 12:49:18 +00:00
Damien George fbcaf0ea18 py: Slightly simplify compile and emit of star/double-star arguments.
Saves a few bytes of code space and eliminates need for rot_two
bytecode (hence saving RAM and execution time, by a tiny bit).
2015-09-23 11:47:01 +01:00
Damien George 3a2171e406 py: Eliminate some cases which trigger unused parameter warnings. 2015-09-04 16:53:46 +01:00
Damien George 65dc960e3b unix-cpy: Remove unix-cpy. It's no longer needed.
unix-cpy was originally written to get semantic equivalent with CPython
without writing functional tests.  When writing the initial
implementation of uPy it was a long way between lexer and functional
tests, so the half-way test was to make sure that the bytecode was
correct.  The idea was that if the uPy bytecode matched CPython 1-1 then
uPy would be proper Python if the bytecodes acted correctly.  And having
matching bytecode meant that it was less likely to miss some deep
subtlety in the Python semantics that would require an architectural
change later on.

But that is all history and it no longer makes sense to retain the
ability to output CPython bytecode, because:

1. It outputs CPython 3.3 compatible bytecode.  CPython's bytecode
changes from version to version, and seems to have changed quite a bit
in 3.5.  There's no point in changing the bytecode output to match
CPython anymore.

2. uPy and CPy do different optimisations to the bytecode which makes it
harder to match.

3. The bytecode tests are not run.  They were never part of Travis and
are not run locally anymore.

4. The EMIT_CPYTHON option needs a lot of extra source code which adds
heaps of noise, especially in compile.c.

5. Now that there is an extensive test suite (which tests functionality)
there is no need to match the bytecode.  Some very subtle behaviour is
tested with the test suite and passing these tests is a much better
way to stay Python-language compliant, rather than trying to match
CPy bytecode.
2015-08-17 12:51:26 +01:00
Damien George 59fba2d6ea py: Remove mp_load_const_bytes and instead load precreated bytes object.
Previous to this patch each time a bytes object was referenced a new
instance (with the same data) was created.  With this patch a single
bytes object is created in the compiler and is loaded directly at execute
time as a true constant (similar to loading bignum and float objects).
This saves on allocating RAM and means that bytes objects can now be
used when the memory manager is locked (eg in interrupts).

The MP_BC_LOAD_CONST_BYTES bytecode was removed as part of this.

Generated bytecode is slightly larger due to storing a pointer to the
bytes object instead of the qstr identifier.

Code size is reduced by about 60 bytes on Thumb2 architectures.
2015-06-25 14:42:13 +00:00
Damien George 9a42eb541e py: Fix naming of function arguments when function is a closure.
Addresses issue #1226.
2015-05-06 13:55:33 +01:00
Damien George 8872abcbc4 py: Remove LOAD_CONST_ELLIPSIS bytecode, use LOAD_CONST_OBJ instead.
Ellipsis constant is rarely used so no point having an extra bytecode
for it.
2015-05-05 22:15:42 +01:00
Damien George 8c1d23a0e2 py: Modify bytecode "with" behaviour so it doesn't use any heap.
Before this patch a "with" block needed to create a bound method object
on the heap for the __exit__ call.  Now it doesn't because we use
load_method instead of load_attr, and save the method+self on the stack.
2015-04-24 01:52:28 +01:00
Damien George e72cda99fd py: Convert occurrences of non-debug printf to mp_printf. 2015-04-16 14:30:16 +00:00
Damien George 91bc32dc16 py: Provide typedefs for function types instead of writing them inline. 2015-04-09 15:31:53 +00:00
Damien George 4dea922610 py: Adjust some spaces in code style/format, purely for consistency. 2015-04-09 15:29:54 +00:00
Damien George c9aa1883ed py: Simplify bytecode prelude when encoding closed over variables. 2015-04-07 00:08:17 +01:00
Damien George 4112590a60 py, compiler: When just bytecode, make explicit calls instead of table.
When just the bytecode emitter is needed there is no need to have a
dynamic method table for the emitter back-end, and we can instead
directly call the mp_emit_bc_XXX functions.  This gives a significant
reduction in code size and a very slight performance boost for the
compiler.

This patch saves 1160 bytes code on Thumb2 and 972 bytes on x86, when
native emitters are disabled.

Overall savings in code over the last 3 commits are:

bare-arm: 1664 bytes.
minimal:  2136 bytes.
stmhal:    584 bytes (it has native emitter enabled).
cc3200:   1736 bytes.
2015-03-26 16:52:45 +00:00
Damien George a210c774f9 py, compiler: Remove emit_pass1 code, using emit_bc to do its job.
First pass for the compiler is computing the scope (eg if an identifier
is local or not) and originally had an entire table of methods dedicated
to this, most of which did nothing.  With changes from previous commit,
this set of methods can be removed and the methods from the bytecode
emitter used instead, with very little modification -- this is what is
done in this commit.

This factoring has little to no impact on the speed of the compiler
(tested by compiling 3763 Python scripts and timing it).

This factoring reduces code size by about 270-300 bytes on Thumb2 archs,
and 400 bytes on x86.
2015-03-26 16:52:45 +00:00
Damien George 542bd6b4a1 py, compiler: Refactor load/store/delete_id logic to reduce code size.
Saves around 230 bytes on Thumb2 and 750 bytes on x86.
2015-03-26 16:52:45 +00:00
Damien George 63f3832e81 py: Combine emit functions for jump true/false to reduce code size.
Saves 116 bytes for stmhal and 56 bytes for cc3200 port.
2015-02-28 15:04:06 +00:00
Damien George 7d414a1b52 py: Parse big-int/float/imag constants directly in parser.
Previous to this patch, a big-int, float or imag constant was interned
(made into a qstr) and then parsed at runtime to create an object each
time it was needed.  This is wasteful in RAM and not efficient.  Now,
these constants are parsed straight away in the parser and turned into
objects.  This allows constants with large numbers of digits (so
addresses issue #1103) and takes us a step closer to #722.
2015-02-08 01:57:40 +00:00
Damien George ff8dd3f486 py, unix: Allow to compile with -Wunused-parameter.
See issue #699.
2015-01-20 12:47:20 +00:00
Damien George 963a5a3e82 py, unix: Allow to compile with -Wsign-compare.
See issue #699.
2015-01-16 17:47:07 +00:00
Damien George 0abb5609b0 py: Remove unnecessary id_flags argument from emitter's load_fast.
Saves 24 bytes in bare-arm.
2015-01-16 12:24:49 +00:00
Damien George d2d64f00fb py: Add "default" to switches to allow better code flow analysis.
This helps compiler produce smaller code.  Saves 124 bytes on stmhal and
bare-arm.
2015-01-14 21:32:42 +00:00
Damien George dab1385177 py: Add load_const_obj to emitter, add LOAD_CONST_OBJ to bytecode.
This allows to directly load a Python object to the Python stack.  See
issue #722 for background.
2015-01-13 15:55:54 +00:00
Damien George 7ee91cf861 py: Add option to cache map lookup results in bytecode.
This is a simple optimisation inspired by JITing technology: we cache in
the bytecode (using 1 byte) the offset of the last successful lookup in
a map. This allows us next time round to check in that location in the
hash table (mp_map_t) for the desired entry, and if it's there use that
entry straight away.  Otherwise fallback to a normal map lookup.

Works for LOAD_NAME, LOAD_GLOBAL, LOAD_ATTR and STORE_ATTR opcodes.

On a few tests it gives >90% cache hit and greatly improves speed of
code.

Disabled by default.  Enabled for unix and stmhal ports.
2015-01-07 21:07:23 +00:00
Damien George b4b10fd350 py: Put all global state together in state structures.
This patch consolidates all global variables in py/ core into one place,
in a global structure.  Root pointers are all located together to make
GC tracing easier and more efficient.
2015-01-07 20:33:00 +00:00
Damien George 51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Damien George 83204f3406 py: Allow to properly disable builtin slice operation.
This patch makes the MICROPY_PY_BUILTINS_SLICE compile-time option
fully disable the builtin slice operation (when set to 0).  This
includes removing the slice sytanx from the grammar.  Now, enabling
slice costs 4228 bytes on unix x64, and 1816 bytes on stmhal.
2014-12-27 17:33:30 +00:00
Damien George e37dcaafb4 py: Allow to properly disable builtin "set" object.
This patch makes MICROPY_PY_BUILTINS_SET compile-time option fully
disable the builtin set object (when set to 0).  This includes removing
set constructor/comprehension from the grammar, the compiler and the
emitters.  Now, enabling set costs 8168 bytes on unix x64, and 3576
bytes on stmhal.
2014-12-27 17:33:30 +00:00
Damien George 8456cc017b py: Compress load-int, load-fast, store-fast, unop, binop bytecodes.
There is a lot potential in compress bytecodes and make more use of the
coding space.  This patch introduces "multi" bytecodes which have their
argument included in the bytecode (by addition).

UNARY_OP and BINARY_OP now no longer take a 1 byte argument for the
opcode.  Rather, the opcode is included in the first byte itself.

LOAD_FAST_[0,1,2] and STORE_FAST_[0,1,2] are removed in favour of their
multi versions, which can take an argument between 0 and 15 inclusive.
The majority of LOAD_FAST/STORE_FAST codes fit in this range and so this
saves a byte for each of these.

LOAD_CONST_SMALL_INT_MULTI is used to load small ints between -16 and 47
inclusive.  Such ints are quite common and now only need 1 byte to
store, and now have much faster decoding.

In all this patch saves about 2% RAM for typically bytecode (1.8% on
64-bit test, 2.5% on pyboard test).  It also reduces the binary size
(because bytecodes are simplified) and doesn't harm performance.
2014-10-25 20:23:13 +01:00
Damien George 1084b0f9c2 py: Store bytecode arg names in bytecode (were in own array).
This saves a lot of RAM for 2 reasons:

1. For functions that don't have default values, var args or var kw
args (which is a large number of functions in the general case), the
mp_obj_fun_bc_t type now fits in 1 GC block (previously needed 2 because
of the extra pointer to point to the arg_names array).  So this saves 16
bytes per function (32 bytes on 64-bit machines).

2. Combining separate memory regions generally saves RAM because the
unused bytes at the end of the GC block are saved for 1 of the blocks
(since that block doesn't exist on its own anymore).  So generally this
saves 8 bytes per function.

Tested by importing lots of modules:

- 64-bit Linux gave about an 8% RAM saving for 86k of used RAM.
- pyboard gave about a 6% RAM saving for 31k of used RAM.
2014-10-25 20:23:13 +01:00
Damien George 7ff996c237 py: Convert [u]int to mp_[u]int_t in emit.h and associated .c files.
Towards resolving issue #50.
2014-09-08 23:05:16 +01:00
Damien George b534e1b9f1 py: Use variable length encoded uints in more places in bytecode.
Code-info size, block name, source name, n_state and n_exc_stack now use
variable length encoded uints.  This saves 7-9 bytes per bytecode
function for most functions.
2014-09-04 14:44:01 +01:00
Damien George 2ac4af6946 py: Allow viper to have type annotations.
Viper functions can now be annotated with the type of their arguments
and return value.  Eg:

@micropython.viper
def f(x:int) -> int:
    return x + 1
2014-08-15 16:45:41 +01:00
Damien George 4747becc64 py: Improve encoding scheme for line-number to bytecode map.
Reduces by about a factor of 10 on average the amount of RAM needed to
store the line-number to bytecode map in the bytecode prelude.

Using CPython3.4's stdlib for statistics: previously, an average of
13 bytes were used per (bytecode offset, line-number offset) pair, and
now with this improvement, that's down to 1.3 bytes on average.

Large RAM usage before was due to some very large steps in line numbers,
both from the start of the first line in a function way down in the
file, and also functions that have big comments and/or big strings in
them (both cases were significant).

Although the savings are large on average for the CPython stdlib, it
won't have such a big effect for small scripts used in embedded
programming.

Addresses issue #648.
2014-07-31 16:12:01 +00:00
Paul Sokolovsky 58c9586c34 emitbc: Fix structure field alignment issue.
dummy_data field is accessed as uint value (e.g.
in emit_write_bytecode_byte_ptr), but is not aligned as such, which causes
bus errors or incorrect behavior on any arch requiring strictly aligned
data (ARM pre-v7, MIPS, etc, etc).
2014-07-12 15:57:28 +03:00
Damien George 40f3c02682 Rename machine_(u)int_t to mp_(u)int_t.
See discussion in issue #50.
2014-07-03 13:25:24 +01:00
Damien George b601d9574a py: Improvements to native emitter.
Native emitter can now compile try/except blocks using nlr_push/nlr_pop.
It probably only works for 1 level of exception handling.  It doesn't
work on Thumb (only x64).

Native emitter can also handle some additional op codes.

With this patch, 198 tests now pass using "-X emit=native" option to
micropython.
2014-06-30 05:17:25 +01:00
Paul Sokolovsky 59c675a64c py: Include mpconfig.h before all other includes.
It defines types used by all other headers.

Fixes #691.
2014-06-21 22:43:22 +03:00
Paul Sokolovsky b8f117dd32 py: For optimization level -O3 and higher, remove lineno info from bytecode. 2014-06-03 12:32:59 +03:00
Damien George 25c84643b6 py: Fix break from within a for loop.
Needed to pop the iterator object when breaking out of a for loop.  Need
also to be careful to unwind exception handler before popping iterator.

Addresses issue #635.
2014-05-30 15:20:41 +01:00
Damien George 0fb80c303a py: Compress a little the bytecode emitter structure. 2014-05-10 18:16:21 +01:00
Damien George 9597771fe4 py, emitters: Fix dummy_data size for bytecode and thumb.
Thumb uses a bit less RAM, bytecode uses a tiny bit more, to avoid
overflow of the dummy buffer in certain cases.

Addresses issue #599.
2014-05-10 18:07:08 +01:00