Commit Graph

3148 Commits

Author SHA1 Message Date
Damien George d3fbfa491f py/parse: Update debugging code to compile on 64-bit arch. 2017-12-29 00:13:36 +11:00
Damien George 0016a45368 py/parse: Compress rule pointer table to table of offsets.
This is the sixth and final patch in a series of patches to the parser that
aims to reduce code size by compressing the data corresponding to the rules
of the grammar.

Prior to this set of patches the rules were stored as rule_t structs with
rule_id, act and arg members.  And then there was a big table of pointers
which allowed to lookup the address of a rule_t struct given the id of that
rule.

The changes that have been made are:
- Breaking up of the rule_t struct into individual components, with each
  component in a separate array.
- Removal of the rule_id part of the struct because it's not needed.
- Put all the rule arg data in a big array.
- Change the table of pointers to rules to a table of offsets within the
  array of rule arg data.

The last point is what is done in this patch here and brings about the
biggest decreases in code size, because an array of pointers is now an
array of bytes.

Code size changes for the six patches combined is:

   bare-arm:  -644
minimal x86: -1856
   unix x64: -5408
unix nanbox: -2080
      stm32:  -720
    esp8266:  -812
     cc3200:  -712

For the change in parser performance: it was measured on pyboard that these
six patches combined gave an increase in script parse time of about 0.4%.
This is due to the slightly more complicated way of looking up the data for
a rule (since the 9th bit of the offset into the rule arg data table is
calculated with an if statement).  This is an acceptable increase in parse
time considering that parsing is only done once per script (if compiled on
the target).
2017-12-29 00:13:36 +11:00
Damien George c2c92ceefc py/parse: Remove rule_t struct because it's no longer needed. 2017-12-28 23:15:36 +11:00
Damien George 66d8885d85 py/parse: Pass rule_id to push_result_token, instead of passing rule_t*. 2017-12-28 23:12:10 +11:00
Damien George 815a8cd1ae py/parse: Pass rule_id to push_result_rule, instead of passing rule_t*.
Reduces code size by eliminating quite a few pointer dereferences.
2017-12-28 23:11:43 +11:00
Damien George 845511af25 py/parse: Break rule data into separate act and arg arrays.
Instead of each rule being stored in ROM as a struct with rule_id, act and
arg, the act and arg parts are now in separate arrays and the rule_id part
is removed because it's not needed.  This reduces code size, by roughly one
byte per grammar rule, around 150 bytes.
2017-12-28 23:09:49 +11:00
Damien George 1039c5e699 py/parse: Split out rule name from rule struct into separate array.
The rule name is only used for debugging, and this patch makes things a bit
cleaner by completely separating out the rule name from the rest of the
rule data.
2017-12-28 23:08:00 +11:00
Damien George b25f92160b py/nlr: Factor out common NLR code to macro and generic funcs in nlr.c.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c, along with a helper macro in nlr.h.  This eliminates duplicated
code.
2017-12-28 16:46:30 +11:00
Damien George 5bf8e85fc8 py/nlr: Clean up selection and config of NLR implementation.
If MICROPY_NLR_SETJMP is not enabled and the machine is auto-detected then
nlr.h now defines some convenience macros for the individual NLR
implementations to use (eg MICROPY_NLR_THUMB).  This keeps nlr.h and the
implementation in sync, and also makes the nlr_buf_t struct easier to read.
2017-12-28 16:18:39 +11:00
Damien George 97cc485538 py/nlrthumb: Fix use of naked funcs, must only contain basic asm code.
A function with a naked attribute must only contain basic inline asm
statements and no C code.

For nlr_push this means removing the "return 0" statement.  But for some
gcc versions this induces a compiler warning so the __builtin_unreachable()
line needs to be added.

For nlr_jump, this function contains a combination of C code and inline asm
so cannot be naked.
2017-12-28 15:59:09 +11:00
Paul Sokolovsky 096e967aad Revert "py/nlr: Factor out common NLR code to generic functions."
This reverts commit 6a3a742a6c.

The above commit has number of faults starting from the motivation down
to the actual implementation.

1. Faulty implementation.

The original code contained functions like:

NORETURN void nlr_jump(void *val) {
    nlr_buf_t **top_ptr = &MP_STATE_THREAD(nlr_top);
    nlr_buf_t *top = *top_ptr;
...
     __asm volatile (
    "mov    %0, %%edx           \n" // %edx points to nlr_buf
    "mov    28(%%edx), %%esi    \n" // load saved %esi
    "mov    24(%%edx), %%edi    \n" // load saved %edi
    "mov    20(%%edx), %%ebx    \n" // load saved %ebx
    "mov    16(%%edx), %%esp    \n" // load saved %esp
    "mov    12(%%edx), %%ebp    \n" // load saved %ebp
    "mov    8(%%edx), %%eax     \n" // load saved %eip
    "mov    %%eax, (%%esp)      \n" // store saved %eip to stack
    "xor    %%eax, %%eax        \n" // clear return register
    "inc    %%al                \n" // increase to make 1, non-local return
     "ret                        \n" // return
    :                               // output operands
    : "r"(top)                      // input operands
    :                               // clobbered registers
     );
}

Which clearly stated that C-level variable should be a parameter of the
assembly, whcih then moved it into correct register.

Whereas now it's:

NORETURN void nlr_jump_tail(nlr_buf_t *top) {
    (void)top;

    __asm volatile (
    "mov    28(%edx), %esi      \n" // load saved %esi
    "mov    24(%edx), %edi      \n" // load saved %edi
    "mov    20(%edx), %ebx      \n" // load saved %ebx
    "mov    16(%edx), %esp      \n" // load saved %esp
    "mov    12(%edx), %ebp      \n" // load saved %ebp
    "mov    8(%edx), %eax       \n" // load saved %eip
    "mov    %eax, (%esp)        \n" // store saved %eip to stack
    "xor    %eax, %eax          \n" // clear return register
    "inc    %al                 \n" // increase to make 1, non-local return
    "ret                        \n" // return
    );

    for (;;); // needed to silence compiler warning
}

Which just tries to perform operations on a completely random register (edx
in this case). The outcome is the expected: saving the pure random luck of
the compiler putting the right value in the random register above, there's
a crash.

2. Non-critical assessment.

The original commit message says "There is a small overhead introduced
(typically 1 machine instruction)". That machine instruction is a call
if a compiler doesn't perform tail optimization (happens regularly), and
it's 1 instruction only with the broken code shown above, fixing it
requires adding more. With inefficiencies already presented in the NLR
code, the overhead becomes "considerable" (several times more than 1%),
not "small".

The commit message also says "This eliminates duplicated code.". An
obvious way to eliminate duplication would be to factor out common code
to macros, not introduce overhead and breakage like above.

3. Faulty motivation.

All this started with a report of warnings/errors happening for a niche
compiler. It could have been solved in one the direct ways: a) fixing it
just for affected compiler(s); b) rewriting it in proper assembly (like
it was before BTW); c) by not doing anything at all, MICROPY_NLR_SETJMP
exists exactly to address minor-impact cases like thar (where a) or b) are
not applicable). Instead, a backwards "solution" was put forward, leading
to all the issues above.

The best action thus appears to be revert and rework, not trying to work
around what went haywire in the first place.
2017-12-26 19:27:58 +02:00
Damien George 26d4a6fa45 py/malloc: Remove unneeded code checking m_malloc return value.
m_malloc already checks for a failed allocation so there's no need to check
for it in m_malloc0.
2017-12-20 16:55:42 +11:00
Damien George 6a3a742a6c py/nlr: Factor out common NLR code to generic functions.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c.  This eliminates duplicated code.

The factoring also allows to make the machine-specific NLR code pure
assembler code, thus allowing nlrthumb.c to use naked function attributes
in the correct way (naked functions can only have basic inline assembler
code in them).

There is a small overhead introduced (typically 1 machine instruction)
because now the generic nlr_jump() must call nlr_jump_tail() rather than
them being one combined function.
2017-12-20 15:42:06 +11:00
Damien George 304a3bcc1c py/modio: Use correct config macro to enable resource_stream function. 2017-12-19 16:59:08 +11:00
Damien George ae1be76d40 py/mpz: Apply a small code-size optimisation. 2017-12-19 15:45:56 +11:00
Damien George 374eaf5271 py/mpz: Fix pow3 function so it handles the case when 3rd arg is 1.
In this case the result should always be 0, even if 2nd arg is 0.
2017-12-19 15:42:58 +11:00
Damien George 7db79d8b03 py/objset: Remove unneeded check from set_equal.
set_equal is called only from set_binary_op, and this guarantees that the
second arg to set_equal is always a set or frozenset.  So there is no need
to do a further check.
2017-12-19 14:01:19 +11:00
Damien George 136cb7f27c py/map: Don't include ordered-dict mutating code when not needed. 2017-12-19 13:37:15 +11:00
Damien George f5fb68e94f py/runtime: Remove unnecessary break statements from switch. 2017-12-19 13:13:21 +11:00
Paul Sokolovsky 6364401666 py/objgenerator: Allow to pend an exception for next execution.
This implements .pend_throw(exc) method, which sets up an exception to be
triggered on the next call to generator's .__next__() or .send() method.
This is unlike .throw(), which immediately starts to execute the generator
to process the exception. This effectively adds Future-like capabilities
to generator protocol (exception will be raised in the future).

The need for such a method arised to implement uasyncio wait_for() function
efficiently (its behavior is clearly "Future" like, and normally would
require to introduce an expensive Future wrapper around all native
couroutines, like upstream asyncio does).

py/objgenerator: pend_throw: Return previous pended value.

This effectively allows to store an additional value (not necessary an
exception) in a coroutine while it's not being executed. uasyncio has
exactly this usecase: to mark a coro waiting in I/O queue (and thus
not executed in the normal scheduling queue), for the purpose of
implementing wait_for() function (cancellation of such waiting coro
by a timeout).
2017-12-15 20:20:36 +02:00
Damien George cf8e8c29e7 py/emitglue: Change type of bit-field to explicitly unsigned mp_uint_t.
Some compilers can treat enum types as signed, in which case 3 bits is not
enough to encode all mp_raw_code_kind_t values.  So change the type to
mp_uint_t.
2017-12-15 10:21:10 +11:00
Damien George f1c9e7760d py/builtinimport: Call __init__ for modules imported via a weak link.
This is a bit of a clumsy way of doing it but solves the issue of __init__
not running when a module is imported via its weak-link name.  Ideally a
better solution would be found.
2017-12-13 14:48:53 +11:00
Damien George c78ef92d78 py/objtype: Refactor object's handling of __new__ to not create 2 objs.
Before this patch, if a user defined the __new__() function for a class
then two instances of that class would be created: once before __new__ is
called and once during the __new__ call (assuming the user creates some
instance, eg using super().__new__, which is most of the time).  The first
one was then discarded.  This refactor makes it so that a new instance is
only created if the user __new__ function doesn't exist.
2017-12-12 16:53:44 +11:00
Damien George d32d22dfd7 py/objtype: Implement better support for overriding native's __init__.
This patch cleans up and generalises part of the code which handles
overriding and calling a native base-class's __init__ method.  It defers
the call to the native make_new() function until after the user (Python)
__init__() method has run.  That user method now has the chance to call the
native __init__/make_new and pass it different arguments.  If the user
doesn't call the super().__init__ method then it will be called
automatically after the user code finishes, to finalise construction of the
instance.
2017-12-12 16:43:16 +11:00
Damien George d3f82bc425 py/mpstate.h: Remove obsolete comment about nlr_top being coded in asm. 2017-12-11 22:51:52 +11:00
Damien George 2759bec858 py: Extend nan-boxing config to have 47-bit small integers.
The nan-boxing representation has an extra 16-bits of space to store
small-int values, and making use of it allows to create and manipulate full
32-bit positive integers (ie up to 0xffffffff) without using the heap.
2017-12-11 22:39:12 +11:00
Damien George 9c02707356 py/objexcept: Use INT_FMT when printing errno value. 2017-12-11 22:38:30 +11:00
Damien George 30fd8484eb py/runtime: Use the Python stack when building *arg and **kwarg state.
With MICROPY_ENABLE_PYSTACK enabled the following language constructs no
longer allocate on the heap: f(*arg), f(**kwarg).
2017-12-11 13:49:09 +11:00
Damien George 1e5a33df41 py: Convert all uses of alloca() to use new scoped allocation API. 2017-12-11 13:49:09 +11:00
Damien George 02d830c035 py: Introduce a Python stack for scoped allocation.
This patch introduces the MICROPY_ENABLE_PYSTACK option (disabled by
default) which enables a "Python stack" that allows to allocate and free
memory in a scoped, or Last-In-First-Out (LIFO) way, similar to alloca().

A new memory allocation API is introduced along with this Py-stack.  It
includes both "local" and "nonlocal" LIFO allocation.  Local allocation is
intended to be equivalent to using alloca(), whereby the same function must
free the memory.  Nonlocal allocation is where another function may free
the memory, so long as it's still LIFO.

Follow-up patches will convert all uses of alloca() and VLA to the new
scoped allocation API.  The old behaviour (using alloca()) will still be
available, but when MICROPY_ENABLE_PYSTACK is enabled then alloca() is no
longer required or used.

The benefits of enabling this option are (or will be once subsequent
patches are made to convert alloca()/VLA):
- Toolchains without alloca() can use this feature to obtain correct and
  efficient scoped memory allocation (compared to using the heap instead
  of alloca(), which is slower).
- Even if alloca() is available, enabling the Py-stack gives slightly more
  efficient use of stack space when calling nested Python functions, due to
  the way that compilers implement alloca().
- Enabling the Py-stack with the stackless mode allows for even more
  efficient stack usage, as well as retaining high performance (because the
  heap is no longer used to build and destroy stackless code states).
- With Py-stack and stackless enabled, Python-calling-Python is no longer
  recursive in the C mp_execute_bytecode function.

The micropython.pystack_use() function is included to measure usage of the
Python stack.
2017-12-11 13:49:09 +11:00
Damien George 5b8998da6d py/runtime: Move mp_exc_recursion_depth to runtime and rename to raise.
For consistency this helper function is renamed to match the other
exception helpers, and moved to their location in runtime.c.
2017-12-11 13:49:09 +11:00
Paul Sokolovsky e7fc765880 unix/mpconfigport: Disable uio.resource_stream().
This function was implemented as an experiment, and was enabled only in
unix port. To remind, it allows to access arbitrary files frozen as
source modules (vs bytecode).

However, further experimentation showed that the same functionality can
be implemented with frozen bytecode. The process requires more steps, but
with suitable toolset it doesn't matter patch. This process is:

1. Convert binary files into "Python resource module" with
tools/mpy_bin2res.py.
2. Freeze as the bytecode.
3. Use micropython-lib's pkg_resources.resource_stream() to access it.

In other words, the extra step is using tools/mpy_bin2res.py (because
there would be wrapper for uio.resource_stream() anyway).

Going frozen bytecode route allows more flexibility, and same/additional
efficiency:

1. Frozen source support can be disabled altogether for additional code
savings.
2. Resources could be also accessed as a buffer, not just as a stream.

There're few caveats too:

1. It wasn't actually profiled the overhead of storing a resource in
"Python resource module" vs storing it directly, but it's assumed that
overhead is small.
2. The "efficiency" claim above applies to the case when resource
file is frozen as the bytecode. If it's not, it actually will take a
lot of RAM on loading. But in this case, the resource file should not
be used (i.e. generated) in the first place, and micropython-lib's
pkg_resources.resource_stream() implementation has the appropriate
fallback to read the raw files instead. This still poses some distribution
issues, e.g. to deployable to baremetal ports (which almost certainly
would require freezeing as the bytecode), a distribution package should
include the resource module. But for non-freezing deployment, presense
of resource module will lead to memory inefficiency.

All the discussion above reminds why uio.resource_stream() was implemented
in the first place - to address some of the issues above. However, since
then, frozen bytecode approach seems to prevail, so, while there're still
some issues to address with it, this change is being made.

This change saves 488 bytes for the unix x86_64 port.
2017-12-10 02:38:23 +02:00
Paul Sokolovsky d21d029d55 py/mkrules.mk: Add "clean-frozen" target to clean frozen script/modules dir.
This target removes any stray files (i.e. something not committed to git)
from scripts/ and modules/ dirs (or whatever FROZEN_DIR and FROZEN_MPY_DIR
is set to).

The expected workflow is:

1. make clean-frozen
2. micropython -m upip -p modules <packages_to_freeze>
3. make

As it can be expected that people may drop random thing in those dirs which
they can miss later, the content is actually backed up before cleaning.
2017-12-10 01:05:29 +02:00
Paul Sokolovsky a35d923cdf py/map: Allow to trace rehashing operations. 2017-12-09 17:32:16 +02:00
Paul Sokolovsky 2b00181592 py/objfun: Factor out macro for initializing codestate.
This is second part of fun_bc_call() vs mp_obj_fun_bc_prepare_codestate()
common code refactor. This factors out code to initialize codestate
object. After this patch, mp_obj_fun_bc_prepare_codestate() is effectively
DECODE_CODESTATE_SIZE() followed by allocation followed by
INIT_CODESTATE(), and fun_bc_call() starts with that too.
2017-12-09 12:49:00 +02:00
Paul Sokolovsky d72370def7 py/objfun, vm: Add comments on codestate allocation in stackless mode. 2017-12-09 11:01:34 +02:00
Paul Sokolovsky fca1d1aa62 py/objfun: Factor out macro for decoding codestate size.
fun_bc_call() starts with almost the same code as
mp_obj_fun_bc_prepare_codestate(), the only difference is a way to
allocate the codestate object (heap vs stack with heap fallback).
Still, would be nice to avoid code duplication to make further
refactoring easier.

So, this commit factors out the common code before the allocation -
decoding and calculating codestate size. It produces two values,
so structured as a macro which writes to 2 variables passed as
arguments.
2017-12-09 09:19:34 +02:00
Paul Sokolovsky dea3fb93c7 py/gc: In sweep debug output, print pointer as a pointer.
Or it will be truncated on a 64-bit platform.
2017-12-09 01:54:01 +02:00
Paul Sokolovsky 5453d88d5d py/gc: Factor out a macro to trace GC mark operations.
To allow easier override it for custom tracing.
2017-12-09 01:48:26 +02:00
Paul Sokolovsky 39dd89fe31 py/runtime: When tracing unary/binary ops, output op (method) name.
E.g.:

    >>> 1+1
    binary 26 __add__ 3 3

Output is similar to bytecode dump (numeric code, then op name).
2017-12-09 01:28:16 +02:00
Paul Sokolovsky c0877cbb0d py/objint_longlong: Check for zero division/modulo. 2017-12-08 20:40:55 +02:00
Damien George 53e111800f py/asmbase: Revert removal of clearing of label offsets for native emit.
The assembler back-end for most architectures needs to know if a jump is
backwards in order to emit optimised machine code, and they do this by
checking if the destination label has been set or not.  So always reset
label offsets to -1 (this reverts partially the previous commit, with some
minor optimisation for the if-logic with the pass variable).
2017-12-08 19:07:00 +11:00
Damien George f935bce3c5 py/{emitbc,asmbase}: Only clear emit labels to -1 when in debug mode.
Clearing the labels to -1 is purely a debugging measure.  For release
builds there is no need to do it as the label offset table should always
have the correct value assigned.
2017-12-08 18:23:23 +11:00
Paul Sokolovsky 9ef4be8b41 py/gc: Add CLEAR_ON_SWEEP option to debug mis-traced objects.
Accessing them will crash immediately instead still working for some time,
until overwritten by some other data, leading to much less deterministic
crashes.
2017-12-08 00:10:44 +02:00
Paul Sokolovsky 9ebc037eee py/malloc: Allow to use debug logging if !MICROPY_MALLOC_USES_ALLOCATED_SIZE.
This is mostly a workaround for forceful rebuilding of mpy-cross on every
codebase change. If this file has debug logging enabled (by patching),
mpy-cross build failed.
2017-12-07 18:01:40 +02:00
Paul Sokolovsky 88a8043a27 py/malloc: MICROPY_MEM_STATS requires MICROPY_MALLOC_USES_ALLOCATED_SIZE.
Error out if they're set incompatibly.
2017-12-07 10:52:40 +02:00
Paul Sokolovsky f5e097021c py/mpprint: Fix "%x" vs "%X" regression introduced in previous commit. 2017-12-07 10:31:14 +02:00
Paul Sokolovsky 5a10e63543 py/mpprint: Support "%lx" format on 64-bit systems.
Before that, the output was truncated to 32 bits. Only "%x" format is
handled, because a typical use is for addresses.

This refactor actually decreased x86_64 code size by 30 bytes.
2017-12-07 10:07:18 +02:00
Paul Sokolovsky 5f8ad284f8 py/mpprint: Make "%p" format work properly on 64-bit systems.
Before, the output was truncated to 32 bits.
2017-12-07 09:06:07 +02:00
Damien George 58f00d7c0e py/modbuiltins: Use standard arg-parsing helper func for builtin print.
This allows the function to raise an exception when unknown keyword args
are passed in.  This patch also reduces code size by (in bytes):

   bare-arm:   -24
minimal x86:   -76
   unix x64:   -56
unix nanbox:   -84
      stm32:   -40
    esp8266:   -68
     cc3200:   -48

Furthermore, this patch adds space (" ") to the set of ROM qstrs which
means it doesn't need to be put in RAM if it's ever used.
2017-12-05 12:14:57 +11:00
Paul Sokolovsky 62b96147e6 py: mp_call_function_*_protected(): Pass-thru return value if possible.
Return the result of called function. If exception happened, return
MP_OBJ_NULL. Allows to use mp_call_function_*_protected() with callbacks
returning values, etc.
2017-12-05 00:38:41 +02:00
Paul Sokolovsky 75d3c046da py/misc.h: Add m_new_obj_var_with_finaliser().
Similar to existing m_new_obj_with_finaliser().
2017-12-04 11:05:49 +02:00
Damien George 64f11470be py/objgenerator: Remove unreachable code for STOP_ITERATION case.
This commit essentially reverts aa9dbb1b03
where this if-condition was added.  It seems that even when that commit
was made the code was never reached by any tests, nor reachable by
analysis (see below).  The same is true with the code as it currently
stands: no test triggers this if-condition, nor any uasyncio examples.
Analysing the flow of the program also shows that it's not reachable:

==START==
-> to trigger this if condition mp_execute_bytecode() must return
   MP_VM_RETURN_YIELD with *sp==MP_OBJ_STOP_ITERATION

   -> mp_execute_bytecode() can only return MP_VM_RETURN_YIELD from the
      MP_BC_YIELD_VALUE bytecode, which can happen in 2 ways:

      -> 1) from a "yield <x>" in bytecode, but <x> must always be a proper
         object, never MP_OBJ_STOP_ITERATION; ==END1==

      -> 2) via yield from, via mp_resume() which must return
         MP_VM_RETURN_YIELD with ret_value==MP_OBJ_STOP_ITERATION, which
         can happen in 3 ways:

         -> 1) it delegates to mp_obj_gen_resume(); go back to ==START==

         -> 2) it returns MP_VM_RETURN_YIELD directly but with a guard that
            ret_val!=MP_OBJ_STOP_ITERATION; ==END2==

         -> 3) it returns MP_VM_RETURN_YIELD with ret_val set from
            mp_call_method_n_kw(), but mp_call_method_n_kw() must return a
            proper object, never MP_OBJ_STOP_ITERATION; ==END3==

The above shows there is no way to trigger the if-condition and it can be
removed.
2017-11-30 12:06:41 +11:00
Damien George 74fad3536b py/gc: In gc_realloc, convert pointer sanity checks to assertions.
These checks are assumed to be true in all cases where gc_realloc is
called with a valid pointer, so no need to waste code space and time
checking them in a non-debug build.
2017-11-29 17:17:08 +11:00
Damien George 8e323b8fa8 py/qstr: Rewrite find_qstr to make manifest that it returns a valid ptr.
So long as the input qstr identifier is valid (below the maximum number of
qstrs) the function will always return a valid pointer.  This patch
eliminates the "return 0" dead-code.
2017-11-29 17:01:39 +11:00
Damien George 3990a52c0f py: Annotate func defs with NORETURN when their corresp decls have it. 2017-11-29 15:43:40 +11:00
Damien George 2161d6b603 py/objdict: Reuse dict-view key iterator for standard dict iterator.
It has equivalent behaviour and reusing it saves some code bytes.
2017-11-27 23:40:31 +11:00
Damien George 84895f1a21 py/parsenum: Improve parsing of floating point numbers.
This patch improves parsing of floating point numbers by converting all the
digits (integer and fractional) together into a number 1 or greater, and
then applying the correct power of 10 at the very end.  In particular the
multiple "multiply by 0.1" operations to build a fraction are now combined
together and applied at the same time as the exponent, at the very end.

This helps to retain precision during parsing of floats, and also includes
a check that the number doesn't overflow during the parsing.  One benefit
is that a float will have the same value no matter where the decimal point
is located, eg 1.23 == 123e-2.
2017-11-27 12:51:52 +11:00
Damien George 5e34a113ea py/runtime: Add MP_BINARY_OP_CONTAINS as reverse of MP_BINARY_OP_IN.
Before this patch MP_BINARY_OP_IN had two meanings: coming from bytecode it
meant that the args needed to be swapped, but coming from within the
runtime meant that the args were already in the correct order.  This lead
to some confusion in the code and comments stating how args were reversed.
It also lead to 2 bugs: 1) containment for a subclass of a native type
didn't work; 2) the expression "{True} in True" would illegally succeed and
return True.  In both of these cases it was because the args to
MP_BINARY_OP_IN ended up being reversed twice.

To fix these things this patch introduces MP_BINARY_OP_CONTAINS which
corresponds exactly to the __contains__ special method, and this is the
operator that built-in types should implement.  MP_BINARY_OP_IN is now only
emitted by the compiler and is converted to MP_BINARY_OP_CONTAINS by
swapping the arguments.
2017-11-24 14:48:23 +11:00
Damien George 5b2f62aff3 py/opmethods: Include the correct header for binary op enums.
By directly including runtime0.h the mpconfig.h settings are not included
and so the enums in runtime0.h can be incorrect.
2017-11-24 12:16:21 +11:00
Damien George 9783ac282e py/runtime: Simplify handling of containment binary operator.
In mp_binary_op, there is no need to explicitly check for type->getiter
being non-null and raising an exception because this is handled exactly by
mp_getiter().  So just call the latter unconditionally.
2017-11-24 12:07:12 +11:00
Damien George d5cf5f70fd py/modbuiltins: Slightly simplify code in builtin round(). 2017-11-22 15:51:51 +11:00
Damien George a07fc5b640 py/objfloat: Allow float() to parse anything with the buffer protocol.
This generalises and simplifies the code and follows CPython behaviour.
2017-11-21 15:01:38 +11:00
Paul Sokolovsky 8667a5f053 py/objnamedtuple: Allow to reuse namedtuple basic functionality.
By declaring interface in objnamedtuple.h and introducing a helper
allocation function.
2017-11-20 09:30:06 +02:00
Damien George da154fdaf9 py: Add config option to disable multiple inheritance.
This patch introduces a new compile-time config option to disable multiple
inheritance at the Python level: MICROPY_MULTIPLE_INHERITANCE.  It is
enabled by default.

Disabling multiple inheritance eliminates a lot of recursion in the call
graph (which is important for some embedded systems), and can be used to
reduce code size for ports that are really constrained (by around 200 bytes
for Thumb2 archs).

With multiple inheritance disabled all tests in the test-suite pass except
those that explicitly test for multiple inheritance.
2017-11-20 16:18:50 +11:00
Damien George 8d956c26d1 py/objstr: When constructing str from bytes, check for existing qstr.
This patch uses existing qstr data where possible when constructing a str
from a bytes object.
2017-11-16 14:02:28 +11:00
Damien George 1f1d5194d7 py/objstr: Make mp_obj_new_str_of_type check for existing interned qstr.
The function mp_obj_new_str_of_type is a general str object constructor
used in many places in the code to create either a str or bytes object.
When creating a str it should first check if the string data already exists
as an interned qstr, and if so then return the qstr object.  This patch
makes the function have such behaviour, which helps to reduce heap usage by
reusing existing interned data where possible.

The old behaviour of mp_obj_new_str_of_type (which didn't check for
existing interned data) is made available through the function
mp_obj_new_str_copy, but should only be used in very special cases.

One consequence of this patch is that the following expression is now True:

    'abc' is ' abc '.split()[0]
2017-11-16 13:53:04 +11:00
Damien George 4601759bf5 py/objstr: Remove "make_qstr_if_not_already" arg from mp_obj_new_str.
This patch simplifies the str creation API to favour the common case of
creating a str object that is not forced to be interned.  To force
interning of a new str the new mp_obj_new_str_via_qstr function is added,
and should only be used if warranted.

Apart from simplifying the mp_obj_new_str function (and making it have the
same signature as mp_obj_new_bytes), this patch also reduces code size by a
bit (-16 bytes for bare-arm and roughly -40 bytes on the bare-metal archs).
2017-11-16 13:17:51 +11:00
Christopher Arndt 1871a924c9 py/mkenv.mk: Use $(PYTHON) consistently when calling Python tools.
Rationale:

* Calling Python build tool scripts from makefiles should be done
  consistently using `python </path/to/script>`, instead of relying on the
  correct she-bang line in the script [1] and the executable bit on the
  script being set. This is more platform-independent.
* The name/path of the Python executable should always be used via the
  makefile variable `PYTHON` set in `py/mkenv.mk`. This way it can be
  easily overwritten by the user with `make PYTHON=/path/to/my/python`.
* The Python executable name should be part of the value of the makefile
  variable, which stands for the build tool command (e.g. `MAKE_FROZEN` and
  `MPY_TOOL`), not part of the command line where it is used. If a Python
  tool is substituted by another (non-python) program, no change to the
  Makefiles is necessary, except in `py/mkenv.mk`.
* This also solves #3369 and #1616.

[1] There are systems, where even the assumption that `/usr/bin/env` always
exists, doesn't hold true, for example on Android (where otherwise the unix
port compiles perfectly well).
2017-11-15 11:56:58 +11:00
Damien George 564a95cb04 py/emitnative: Clean up asm macro names so they have dest as first arg.
All the asm macro names that convert a particular architecture to a generic
interface now follow the convention whereby the "destination" (usually a
register) is specified first.
2017-11-15 11:46:49 +11:00
stijn 79ed58f87b py/objnamedtuple: Add _asdict function if OrderedDict is supported 2017-11-12 14:16:54 +02:00
Paul Sokolovsky cada971113 py/objtype: mp_obj_new_type: Name base types related vars more clearly.
As vars contains array of base types and its length, name them as such,
avoid generic "items" and "len" names.
2017-11-11 00:11:24 +02:00
Paul Sokolovsky 1b146e9de9 py/mpconfig: Introduce reusable MP_HTOBE32(), etc. macros.
Macros to convert big-endian values to host byte order and vice-versa.
These were defined in adhoc way for some ports (e.g. esp8266), allow
reuse, provide default implementations, while allow ports to override.
2017-11-08 19:47:37 +02:00
Damien George 487dbdb267 py/compile: Use alloca instead of qstr_build when compiling import name.
The technique of using alloca is how dotted import names are composed in
mp_import_from and mp_builtin___import__, so use the same technique in the
compiler.  This puts less pressure on the heap (only the stack is used if
the qstr already exists, and if it doesn't exist then the standard qstr
block memory is used for the new qstr rather than a separate chunk of the
heap) and reduces overall code size.
2017-11-01 13:16:16 +11:00
Damien George 02b4b23319 Revert "py/{mkenv.mk,mkrules.mk}: Append .exe for Windows executable files."
This reverts commit 3289b9b7a7.
The commit broke building on MINGW because the filename became
micropython.exe.exe.  A proper solution to support more Windows build
environments requires more thought and testing.
2017-10-31 22:01:56 +11:00
Eric Poulsen 74ec52d857 extmod/modussl: Add finaliser support for ussl objects.
Per the comment found here
https://github.com/micropython/micropython-esp32/issues/209#issuecomment-339855157,
this patch adds finaliser code to prevent memory leaks from ussl objects,
which is especially useful when memory for a ussl context is allocated
outside the uPy heap.  This patch is in-line with the finaliser code found
in many modsocket implementations for various ports.

This feature is configured via MICROPY_PY_USSL_FINALISER and is disabled by
default because there may be issues using it when the ussl state *is*
allocated on the uPy heap, rather than externally.
2017-10-30 15:25:32 +11:00
Paul Sokolovsky 0e80f345f8 py/objtype: Introduce MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS.
This allows to configure support for inplace special methods separately,
similar to "normal" and reverse special methods. This is useful, because
inplace methods are "the most optional" ones, for example, if inplace
methods aren't defined, the operation will be executed using normal
methods instead.

As a caveat, __iadd__ and __isub__ are implemented even if
MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS isn't defined. This is similar
to the state of affairs before binary operations refactor, and allows
to run existing tests even if MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS
isn't defined.
2017-10-27 22:29:15 +03:00
Paul Sokolovsky 9b9dbc5815 py/objtype: Define all special methods if requested.
If MICROPY_PY_ALL_SPECIAL_METHODS is defined, actually define all special
methods (still subject to gating by e.g. MICROPY_PY_REVERSE_SPECIAL_METHODS).

This adds quite a number of qstr's, so should be used sparingly.
2017-10-27 20:06:35 +03:00
Damien George f4059dcc0c all: Use NULL instead of "" when calling mp_raise exception helpers.
This is the established way of doing it and reduces code size by a little
bit.
2017-10-24 22:39:36 +11:00
Paul Sokolovsky 9956fd0710 py/objtype: Fit qstrs for special methods in byte type.
Update makeqstrdata.py to sort strings starting with "__" to the beginning
of qstr list, so they get low qstr id's, guaranteedly fitting in 8 bits.
Then use this property to further compact op_id => qstr mapping arrays.
2017-10-21 11:06:32 +03:00
Paul Sokolovsky f2baa9ec24 py/objtype: Use CPython compatible method name for sizeof.
Per https://docs.python.org/3/library/sys.html#sys.getsizeof:

getsizeof() calls the object’s __sizeof__ method. Previously, "getsizeof"
was used mostly to save on new qstr, as we don't really support calling
this method on arbitrary objects (so it was used only for reporting).
However, normalize it all now.
2017-10-19 12:44:53 +03:00
Damien George 93ce125abe py/argcheck: Remove #if guard around terse error message helper func.
Not all compilers/analysers are smart enough to realise that this function
is never called if MICROPY_ERROR_REPORTING is not TERSE, because the logic
in the code uses if statements rather than #if to select whether to call
this function or not (MSC in debug mode is an example of this, but there
are others).  So just unconditionally compile this helper function.  The
code-base anyway relies on the linker to remove unused functions.
2017-10-19 18:57:26 +11:00
Damien George 37282f8fc1 extmod/uos_dupterm: Update uos.dupterm() and helper funcs to have index.
The uos.dupterm() signature and behaviour is updated to reflect the latest
enhancements in the docs.  It has minor backwards incompatibility in that
it no longer accepts zero arguments.

The dupterm_rx helper function is moved from esp8266 to extmod and
generalised to support multiple dupterm slots.

A port can specify multiple slots by defining the MICROPY_PY_OS_DUPTERM
config macro to an integer, being the number of slots it wants to have;
0 means to disable the dupterm feature altogether.

The unix and esp8266 ports are updated to work with the new interface and
are otherwise unchanged with respect to functionality.
2017-10-13 20:01:57 +11:00
Damien George c59fc1419d py/emitnative: Simplify binary op emitter, no need to check inplace ops. 2017-10-12 12:26:49 +11:00
Damien George a3afa8cfc4 py/emitnative: Implement floor-division and modulo for viper emitter. 2017-10-11 18:54:34 +11:00
Damien George 69da74e538 py/modbuiltins: Use existing utf8_get_char helper in builtin ord func. 2017-10-11 11:25:20 +11:00
Damien George dc948e4d54 py/formatfloat: Use standard isinf, isnan funcs instead of custom ones.
Reduces code size by a tiny bit.
2017-10-10 16:27:54 +11:00
Damien George 08a196697c py/formatfloat: Don't print the negative sign of a NaN value.
NaN may have the sign bit set but it has no meaning, so don't print it out.
2017-10-10 16:01:13 +11:00
Damien George d8d4e4dfbe py/modmath: Convert log2 macro into a function.
So that a pointer to it can be passed as a pointer to math_generic_1.  This
patch also makes the function work for single and double precision floating
point.
2017-10-10 16:01:04 +11:00
Damien George 25e140652b py/modmath: Add full checks for math domain errors.
This patch changes how most of the plain math functions are implemented:
there are now two generic math wrapper functions that take a pointer to a
math function (like sin, cos) and perform the necessary conversion to and
from MicroPython types.  This helps to reduce code size.  The generic
functions can also check for math domain errors in a generic way, by
testing if the result is NaN or infinity combined with finite inputs.

The result is that, with this patch, all math functions now have full
domain error checking (even gamma and lgamma) and code size has decreased
for most ports.  Code size changes in bytes for those with the math module
are:

   unix x64:  -432
unix nanbox:  -792
      stm32:   -88
    esp8266:   +12

Tests are also added to check domain errors are handled correctly.
2017-10-10 15:57:45 +11:00
Damien George 933eab46fc py/bc: Update opcode_format_table to match the bytecode. 2017-10-10 10:37:38 +11:00
Damien George ff93fd4f50 py/persistentcode: Bump .mpy version number to version 3.
The binary and unary ops have changed bytecode encoding.
2017-10-05 10:49:44 +11:00
Damien George 36f7952f76 py/objtype: Clean up unary- and binary-op enum-to-qstr mapping tables. 2017-10-05 10:49:44 +11:00
Damien George 0864a6957f py: Clean up unary and binary enum list to keep groups together.
2 non-bytecode binary ops (NOT_IN and IN_NOT) are moved out of the
bytecode group, so this change will change the bytecode format.
2017-10-05 10:49:44 +11:00
Damien George 23faf88cab py/mpprint: Only check for null string printing when NDEBUG not defined.
Printing "(null)" when a NULL string pointer is passed to %s is a debugging
feature and not a feature that's relied upon by the code.  So it only needs
to be compiled in when debugging (such as assert) is enabled, and saves
roughy 30 bytes of code when disabled.

This patch also fixes this NULL check to not do the check if the precision
is specified as zero.
2017-10-04 18:07:32 +11:00
Damien George dfa563c71f py/objstr: Make empty bytes object have a null-terminating byte.
Because a lot of string processing functions assume there is a null
terminating byte, so they can work in an efficient way.

Fixes issue #3334.
2017-10-04 17:59:22 +11:00
Damien George a3dc1b1957 all: Remove inclusion of internal py header files.
Header files that are considered internal to the py core and should not
normally be included directly are:
    py/nlr.h - internal nlr configuration and declarations
    py/bc0.h - contains bytecode macro definitions
    py/runtime0.h - contains basic runtime enums

Instead, the top-level header files to include are one of:
    py/obj.h - includes runtime0.h and defines everything to use the
        mp_obj_t type
    py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
        and defines everything to use the general runtime support functions

Additional, specific headers (eg py/objlist.h) can be included if needed.
2017-10-04 12:37:50 +11:00
Damien George 6c82cfc089 py/objtype: Change type of enum-to-qstr table to uint16_t to save space.
Qstr values fit in 16-bits (and this fact is used elsewhere in the code) so
no need to use more than that for the large lookup tables.  The compiler
will anyway give a warning if the qstr values don't fit in 16 bits.  Saves
around 80 bytes of code space for Thumb2 archs.
2017-10-04 11:31:05 +11:00
chrismas9 3289b9b7a7 py/{mkenv.mk,mkrules.mk}: Append .exe for Windows executable files.
Building mpy-cross: this patch adds .exe to the PROG name when building
executables for host (eg mpy-cross) on Windows.  make clean now removes
mpy-cross.exe under Windows.

Building MicroPython: this patch sets MPY_CROSS to mpy-cross.exe or
mpy-cross so they can coexist and use cygwin or WSL without rebuilding
mpy-cross.  The dependency in the mpy rule now uses mpy-cross.exe for
Windows and mpy-cross for Linux.
2017-10-04 00:21:05 +11:00
Damien George 1394258f37 py/objset: Include the failed key in a KeyError raised from set.remove. 2017-10-03 18:03:06 +11:00