Commit Graph

3758 Commits

Author SHA1 Message Date
Damien George 323b5f7270 py/modsys: Don't compile getsizeof function if feature is disabled. 2018-04-04 14:23:25 +10:00
Damien George bc36521386 py/vm: Optimise handling of stackless mode when pystack is enabled.
When pystack is enabled mp_obj_fun_bc_prepare_codestate() will always
return a valid pointer, and if there is no more pystack available then it
will raise an exception (a RuntimeError).  So having pystack enabled with
stackless enabled automatically gives strict stackless mode.  There is
therefore no need to have code for strict stackless mode when pystack is
enabled, and this patch optimises the VM for such a case.
2018-04-04 00:51:10 +10:00
Damien George c7f880eda3 py/vm: Don't do unnecessary updates of ip and sp variables.
Neither the ip nor sp variables are used again after the execution of the
RAISE_VARARGS opcode, so they don't need to be updated.
2018-04-04 00:46:31 +10:00
Damien George f50b64cab5 py/runtime: Be sure that non-intercepted thrown object is an exception.
The VM expects that, if mp_resume() returns MP_VM_RETURN_EXCEPTION, then
the returned value is an exception instance (eg to add a traceback to it).
It's possible that a value passed to a generator's throw() is not an
exception so must be explicitly checked for if the thrown value is not
intercepted by the generator.

Thanks to @jepler for finding the bug.
2018-03-30 12:43:38 +11:00
Damien George 3280788195 py/runtime: Check that keys in dicts passed as ** args are strings.
Prior to this patch the code would crash if a key in a ** dict was anything
other than a str or qstr.  This is because mp_setup_code_state() assumes
that keys in kwargs are qstrs (for efficiency).

Thanks to @jepler for finding the bug.
2018-03-30 11:13:32 +11:00
Damien George 5edce4539b py/objexcept: Make MP_DEFINE_EXCEPTION public so ports can define excs. 2018-03-17 00:31:40 +11:00
Damien George f6a1f18603 py/makeqstrdefs.py: Optimise by using compiled re's so it runs faster.
By using pre-compiled regexs, using startswith(), and explicitly checking
for empty lines (of which around 30% of the input lines are), automatic
qstr extraction is speed up by about 10%.
2018-03-16 23:54:06 +11:00
Damien George e0bc438e4b py/obj.h: Move declaration of mp_obj_list_init to objlist.h.
If this function is used then objlist.h is already included to get the
definition of mp_obj_list_t.
2018-03-13 14:03:15 +11:00
Damien George 9f811e9096 py/obj.h: Clean up by removing commented-out inline versions of macros. 2018-03-13 14:01:55 +11:00
Damien George d4b55eff44 py/misc.h: Remove unused count_lead_ones() inline function.
This function was never used for unicode/utf8 handling code, or anything
else, so remove it to keep things clean.
2018-03-13 13:23:30 +11:00
Damien George 9884a2c712 py/objint: Remove unreachable code checking for int type in format func.
All callers of mp_obj_int_formatted() are expected to pass in a valid int
object, and they do:

- mp_obj_int_print() should always pass through an int object because it is
  the print special method for int instances.

- mp_print_mp_int() checks that the argument is an int, and if not converts
  it to a small int.

This patch saves around 20-50 bytes of code space.
2018-03-02 11:01:24 +11:00
Damien George 955ee6477f py/formatfloat: Fix case where floats could render with negative digits.
Prior to this patch, some architectures (eg unix x86) could render floats
with "negative" digits, like ")".  For example, '%.23e' % 1e-80 would come
out as "1.0000000000000000/)/(,*0e-80".  This patch fixes the known cases.
2018-03-01 17:00:02 +11:00
Damien George 7b050fa76c py/formatfloat: Fix case where floats could render with a ":" character.
Prior to this patch, some architectures (eg unix x86) could render floats
with a ":" character in them, eg 1e+39 would come out as ":e+38" (":" is
just after "9" in ASCII so this is like 10e+38).  This patch fixes some of
these cases.
2018-03-01 16:02:59 +11:00
Damien George bc12eca461 py/formatfloat: Fix rounding of %f format with edge-case FP values.
Prior to this patch the %f formatting of some FP values could be off by up
to 1, eg '%.0f' % 123 would return "122" (unix x64).  Depending on the FP
precision (single vs double) certain numbers would format correctly, but
others wolud not.  This patch should fix all cases of rounding for %f.
2018-03-01 15:51:03 +11:00
Damien George a9f6d49218 py/vm: Simplify handling of special-case STOP_ITERATION in yield from.
There's no need to have MP_OBJ_NULL a special case, the code can re-use
the MP_OBJ_STOP_ITERATION value to signal the special case and the VM can
detect this with only one check (for MP_OBJ_STOP_ITERATION).
2018-02-27 15:48:09 +11:00
Damien George 22ade2f5c4 py/vm: Fix case of handling raised StopIteration within yield from.
This patch concerns the handling of an NLR-raised StopIteration, raised
during a call to mp_resume() which is handling the yield from opcode.

Previously, commit 6738c1dded introduced code
to handle this case, along with a test.  It seems that it was lucky that
the test worked because the code did not correctly handle the stack pointer
(sp).

Furthermore, commit 79d996a57b improved the
way mp_resume() propagated certain exceptions: it changed raising an NLR
value to returning MP_VM_RETURN_EXCEPTION.  This change meant that the
test introduced in gen_yield_from_ducktype.py was no longer hitting the
code introduced in 6738c1dded.

The patch here does two things:

1. Fixes the handling of sp in the VM for the case that yield from is
   interrupted by a StopIteration raised via NLR.

2. Introduces a new test to check this handling of sp and re-covers the
   code in the VM.
2018-02-27 15:39:31 +11:00
Damien George 9d8347a9aa py/mpstate.h: Add repl_line state for MICROPY_REPL_EVENT_DRIVEN. 2018-02-26 16:08:58 +11:00
Damien George f75c7ad1a9 py/mpz: In mpz_clone, remove unused check for NULL dig.
This path for src->deg==NULL is never used because mpz_clone() is always
called with an argument that has a non-zero integer value, and hence has
some digits allocated to it (mpz_clone() is a static function private to
mpz.c all callers of this function first check if the integer value is zero
and if so take a special-case path, bypassing the call to mpz_clone()).

There is some unused and commented-out functions that may actually pass a
zero-valued mpz to mpz_clone(), so some TODOs are added to these function
in case they are needed in the future.
2018-02-25 22:59:19 +11:00
Damien George c0bcf00ed1 py/asm*.c: Remove unnecessary check for num_locals<0 in asm entry func.
All callers of the asm entry function guarantee that num_locals>=0, so no
need to add an explicit check for it.  Use an assertion instead.

Also, the signature of asm_x86_entry is changed to match the other asm
entry functions.
2018-02-24 23:10:20 +11:00
Damien George 7dfa56e40e py/compile: Adjust c_assign_atom_expr() to use return instead of goto.
Makes the flow of the function a little more obvious, and allows to reach
100% coverage of compile.c when using gcov.
2018-02-24 23:03:17 +11:00
Damien George 638b860066 extmod/vfs_fat: Merge remaining vfs_fat_misc.c code into vfs_fat.c.
The only function left in vfs_fat_misc.c is fat_vfs_import_stat() which
can logically go into vfs_fat.c, allowing to remove vfs_fat_misc.c.
2018-02-23 17:24:57 +11:00
Damien George 6af4515969 py: Use "GEN" consistently for describing files generated in the build. 2018-02-22 12:48:51 +11:00
Damien George 65ef59a9b5 py/py.mk: Remove .. path component from list of extmod files.
This just makes it a bit cleaner in the output of the build process:
instead of "CC ../../py/../extmod/" there is now "CC ../../extmod/".
2018-02-22 12:48:51 +11:00
Damien George 8ca469cae2 py/py.mk: Split list of uPy sources into core and extmod files.
If a port only needs the core files then it can now use the $(PY_CORE_O)
variable instead of $(PY_O).  $(PY_EXTMOD_O) contains the list of extmod
files (including some files from lib/). $(PY_O) retains its original
definition as the list of all object file (including those for frozen code)
and is a convenience variable for ports that want everything.
2018-02-22 12:48:15 +11:00
Damien George 6e675c1baa py/objdeque: Use m_new0 when allocating items to avoid need to clear.
Saves a few bytes of code space, and is more efficient because with
MICROPY_GC_CONSERVATIVE_CLEAR enabled by default all memory is already
cleared when allocated.
2018-02-21 23:36:46 +11:00
Damien George 160d670868 py/objdeque: Protect against negative maxlen in deque constructor.
Otherwise passing -1 as maxlen will lead to a zero allocation and
subsequent unbound buffer overflow in deque.append() because i_put is
allowed to grow without bound.
2018-02-21 23:34:17 +11:00
Damien George 6c3faf6c17 py/objdeque: Allow to compile without warnings by disabling deque_clear. 2018-02-21 22:52:58 +11:00
Paul Sokolovsky 970eedce8f py/objdeque: Implement ucollections.deque type with fixed size.
So far, implements just append() and popleft() methods, required for
a normal queue. Constructor doesn't accept an arbitarry sequence to
initialize from (am empty deque is always created), so an empty tuple
must be passed as such. Only fixed-size deques are supported, so 2nd
argument (size) is required.

There's also an extension to CPython - if True is passed as 3rd argument,
append(), instead of silently overwriting the oldest item on queue
overflow, will throw IndexError. This behavior is desired in many
cases, where queues should store information reliably, instead of
silently losing some items.
2018-02-21 22:39:25 +11:00
Damien George fe3e17b026 py/objint: Use MP_OBJ_IS_STR_OR_BYTES macro instead of 2 separate ones. 2018-02-21 00:20:46 +11:00
Damien George 8769049e93 py/objstr: Remove unnecessary check for positive splits variable.
At this point in the code the variable "splits" is guaranteed to be
positive due to the check for "splits == 0" above it.
2018-02-20 19:19:02 +11:00
Damien George 7e2a48858c py/modmicropython: Allow to have stack_use() func without mem_info().
The micropython.stack_use() function is useful to query the current C stack
usage, and it's inclusion in the micropython module doesn't need to be tied
to the inclusion of mem_info()/qstr_info() because it doesn't rely on any
of the code from these functions.  So this patch introduces the config
option MICROPY_PY_MICROPYTHON_STACK_USE which can be used to independently
control the inclusion of stack_use().  By default it is enabled if
MICROPY_PY_MICROPYTHON_MEM_INFO is enabled (thus not changing any of the
existing ports).
2018-02-20 18:30:22 +11:00
Damien George 209936880d py/builtinimport: Add compile-time option to disable external imports.
The new option is MICROPY_ENABLE_EXTERNAL_IMPORT and is enabled by default
so that the default behaviour is the same as before.  With it disabled
import is only supported for built-in modules, not for external files nor
frozen modules.  This allows to support targets that have no filesystem of
any kind and that only have access to pre-supplied built-in modules
implemented natively.
2018-02-20 18:00:44 +11:00
Damien George 6e7819ee2e py/objmodule: Factor common code for calling __init__ on builtin module. 2018-02-20 17:56:58 +11:00
Damien George 4e469085c1 py/objstr: Protect against creating bytes(n) with n negative.
Prior to this patch uPy (on a 32-bit arch) would have severe issues when
calling bytes(-1): such a call would call vstr_init_len(vstr, -1) which
would then +1 on the len and call vstr_init(vstr, 0), which would then
round this up and allocate a small amount of memory for the vstr.  The
bytes constructor would then attempt to zero out all this memory, thinking
it had allocated 2^32-1 bytes.
2018-02-19 16:25:30 +11:00
Damien George 165aab12a3 py/repl: Generalise REPL autocomplete to use qstr probing.
This patch changes the way REPL autocomplete finds matches.  It now probes
the target object for all qstrs via mp_load_method_maybe to look for a
match with the given input string.  Similar to how the builtin dir()
function works, this new algorithm now find all methods and instances of
user-defined classes including attributes of their parent classes.  This
helps a lot at the REPL prompt for user-discovery and to autocomplete names
even for classes that are derived.

The downside is that this new algorithm is slower than the previous one,
and in particular will be slower the more qstrs there are in the system.
But because REPL autocomplete is primarily used in an interactive way it is
not that important to make it fast, as long as it is "fast enough" compared
to human reaction.

On a slow microcontroller (CPU running at 16MHz) the autocomplete time for
a list of 35 names in the outer namespace (pressing tab at a bare prompt)
takes about 160ms with this algorithm, compared to about 40ms for the
previous implementation (this time includes the actual printing of the
names as well).  This time of 160ms is very reasonable especially given the
new functionality of listing all the names.

This patch also decreases code size by:

   bare-arm:    +0
minimal x86:  -128
   unix x64:  -128
unix nanbox:  -224
      stm32:   -88
     cc3200:   -80
    esp8266:   -92
      esp32:   -84
2018-02-19 16:12:44 +11:00
Damien George 98647e83c7 py/modbuiltins: Simplify and generalise dir() by probing qstrs.
This patch improves the builtin dir() function by probing the target object
with all possible qstrs via mp_load_method_maybe.  This is very simple (in
terms of implementation), doesn't require recursion, and allows to list all
methods of user-defined classes (without duplicates) even if they have
multiple inheritance with a common parent.  The downside is that it can be
slow because it has to iterate through all the qstrs in the system, but
the "dir()" function is anyway mostly used for testing frameworks and user
introspection of types, so speed is not considered a priority.

In addition to providing a more complete implementation of dir(), this
patch is simpler than the previous implementation and saves some code
space:

   bare-arm:   -80
minimal x86:   -80
   unix x64:   -56
unix nanbox:   -48
      stm32:   -80
     cc3200:   -80
    esp8266:  -104
      esp32:   -64
2018-02-19 16:12:44 +11:00
Damien George a8775aaeb0 py/qstr: Add QSTR_TOTAL() macro to get number of qstrs. 2018-02-19 16:12:44 +11:00
Damien George 2a0cbc0d38 py/gc: Update comment now that gc_drain_stack is called gc_mark_subtree. 2018-02-19 16:08:20 +11:00
Ayke van Laethem 736faef223 py/gc: Make GC stack pointer a local variable.
This saves a bit in code size, and saves some precious .bss RAM:

                 .text  .bss
minimal CROSS=1: -28    -4
unix (64-bit):   -64    -8
2018-02-19 16:05:46 +11:00
Ayke van Laethem 5c9e5618e0 py/gc: Rename gc_drain_stack to gc_mark_subtree and pass it first block.
This saves a bit in code size:

minimal CROSS=1: -44
unix:            -96
2018-02-19 16:00:59 +11:00
Ayke van Laethem ea7cf2b738 py/gc: Reduce code size by specialising VERIFY_MARK_AND_PUSH macro.
This macro is written out explicitly in the two locations that it is used
and then the code is optimised, opening possibilities for further
optimisations and reducing code size:

unix:            -48
minimal CROSS=1: -32
stm32:           -32
2018-02-19 15:58:49 +11:00
Mike Wadsten a3e01d3642 py/objdict: Disallow possible modifications to fixed dicts. 2018-02-18 21:51:04 -06:00
Damien George 7b2a9b059a py/pystack: Use "pystack exhausted" as error msg for out of pystack mem.
Using the message "maximum recursion depth exceeded" for when the pystack
runs out of memory can be misleading because the pystack can run out for
reasons other than deep recursion (although in most cases pystack
exhaustion is probably indirectly related to deep recursion).  And it's
important to give the user more precise feedback as to the reason for the
error: if they know precisely that the pystack was exhausted then they have
a chance to increase the amount of memory available to the pystack (as
opposed to not knowing if it was the C stack or pystack that ran out).

Also, C stack exhaustion is more serious than pystack exhaustion because it
could have been that the C stack overflowed and overwrote/corrupted some
data and so the system must be restarted.  The pystack can never corrupt
data in this way so pystack exhaustion does not require a system restart.
Knowing the difference between these two cases is therefore important.

The actual exception type for pystack exhaustion remains as RuntimeError so
that programatically it behaves the same as a C stack exhaustion.
2018-02-19 00:26:14 +11:00
Ayke van Laethem 5591bd237a py/nlrthumb: Do not mark nlr_push as not returning anything.
By adding __builtin_unreachable() at the end of nlr_push, we're
essentially telling the compiler that this function will never return.
When GCC LTO is in use, this means that any time nlr_push() is called
(which is often), the compiler thinks this function will never return
and thus eliminates all code following the call.

Note: I've added a 'return 0' for older GCC versions like 4.6 which
complain about not returning anything (which doesn't make sense in a
naked function). Newer GCC versions (tested 4.8, 5.4 and some others)
don't complain about this.
2018-02-18 01:35:27 +01:00
Damien George 73d1d20b46 py/objexcept: Remove long-obsolete mp_const_MemoryError_obj.
This constant exception instance was once used by m_malloc_fail() to raise
a MemoryError without allocating memory, but it was made obsolete long ago
by 3556e45711.  The functionality is now
replaced by the use of mp_emergency_exception_obj which lives in the global
uPy state, and which can handle any exception type, not just MemoryError.
2018-02-15 16:50:02 +11:00
Damien George d77da83d55 py/objrange: Implement (in)equality comparison between range objects.
This feature is not often used so is guarded by the config option
MICROPY_PY_BUILTINS_RANGE_BINOP which is disabled by default.  With this
option disabled MicroPython will always return false when comparing two
range objects for equality (unless they are exactly the same object
instance).  This does not match CPython so if (in)equality between range
objects is needed then this option should be enabled.

Enabling this option costs between 100 and 200 bytes of code space
depending on the machine architecture.
2018-02-14 23:17:06 +11:00
Damien George 5604b710c2 py/emitglue: When assigning bytecode only pass bytecode len if needed.
Most embedded targets will have this bit of the code disabled, saving a
small amount of code space.
2018-02-14 18:41:17 +11:00
Damien George e98ff40604 py/modbuiltins: Simplify casts from char to byte ptr in builtin ord. 2018-02-14 18:27:14 +11:00
Damien George 19aee9438a py/unicode: Clean up utf8 funcs and provide non-utf8 inline versions.
This patch provides inline versions of the utf8 helper functions for the
case when unicode is disabled (MICROPY_PY_BUILTINS_STR_UNICODE set to 0).
This saves code size.

The unichar_charlen function is also renamed to utf8_charlen to match the
other utf8 helper functions, and the signature of this function is adjusted
for consistency (const char* -> const byte*, mp_uint_t -> size_t).
2018-02-14 18:19:22 +11:00
Damien George bbb08431f3 py/objfloat: Fix case of raising 0 to -infinity.
It was raising an exception but it should return infinity.
2018-02-08 14:35:43 +11:00
Damien George b75cb8392b py/parsenum: Fix parsing of floats that are close to subnormal.
Prior to this patch, a float literal that was close to subnormal would
have a loss of precision when parsed.  The worst case was something like
float('10000000000000000000e-326') which returned 0.0.
2018-02-08 14:02:50 +11:00
Damien George 0c650d4276 py/vm: Simplify stack sentinel values for unwind return and jump.
This patch simplifies how sentinel values are stored on the stack when
doing an unwind return or jump.  Instead of storing two values on the stack
for an unwind jump it now stores only one: a negative small integer means
unwind-return and a non-negative small integer means unwind-jump with the
value being the number of exceptions to unwind.  The savings in code size
are:

   bare-arm:   -56
minimal x86:   -68
   unix x64:   -80
unix nanbox:    -4
      stm32:   -56
     cc3200:   -64
    esp8266:   -76
      esp32:  -156
2018-02-08 13:30:33 +11:00
Damien George 771dfb0826 py/modbuiltins: For builtin_chr, use uint8_t instead of char for array.
The array should be of type unsigned byte because that is the type of the
values being stored.  And changing to uint8_t helps to prevent warnings
from some static analysers.
2018-02-07 16:13:02 +11:00
Damien George b45c8c17f0 py/objtype: Check and prevent delete/store on a fixed locals map.
Note that the check for elem!=NULL is removed for the
MP_MAP_LOOKUP_ADD_IF_NOT_FOUND case because mp_map_lookup will always
return non-NULL for such a case.
2018-02-07 15:44:29 +11:00
Damien George 253f2bd7be py/compile: Combine compiler-opt of 2 and 3 tuple-to-tuple assignment.
This patch combines the compiler optimisation code for double and triple
tuple-to-tuple assignment, taking it from two separate if-blocks to one
combined if-block.  This can be done because the code for both of these
optimisations has a lot in common.  Combining them together reduces code
size for ports that have the triple-tuple optimisation enabled (and doesn't
change code size for ports that have it disabled).
2018-02-04 13:35:21 +11:00
stijn 42c4dd09a1 py/nlr: Fix missing trailing characters in comments in nlr.c 2017-12-29 22:24:53 +11:00
stijn b184b6ae53 py/nlr: Fix nlr functions for 64bit ports built with gcc on Windows
The number of registers used should be 10, not 12, to match the assembly
code in nlrx64.c. With this change the 64bit mingw builds don't need to
use the setjmp implementation, and this fixes miscellaneous crashes and
assertion failures as reported in #1751 for instance.

To avoid mistakes in the future where something gcc-related for Windows
only gets fixed for one particular compiler/environment combination,
make use of a MICROPY_NLR_OS_WINDOWS macro.

To make sure everything nlr-related is now ok when built with gcc this
has been verified with:
- unix port built with gcc on Cygwin (i686-pc-cygwin-gcc and
  x86_64-pc-cygwin-gcc, version 6.4.0)
- windows port built with mingw-w64's gcc from Cygwin
 (i686-w64-mingw32-gcc and x86_64-w64-mingw32-gcc, version 6.4.0)
 and MSYS2 (like the ones on Cygwin but version 7.2.0)
2017-12-29 22:24:46 +11:00
Damien George e784274430 py/mpz: In mpz_as_str_inpl, convert always-false checks to assertions.
There are two checks that are always false so can be converted to (negated)
assertions to save code space and execution time.  They are:

1. The check of the str parameter, which is required to be non-NULL as per
   the original comment that it has enough space in it as calculated by
   mp_int_format_size.  And for all uses of this function str is indeed
   non-NULL.

2. The check of the base parameter, which is already required to be between
   2 and 16 (inclusive) via the assertion in mp_int_format_size.
2017-12-29 14:17:55 +11:00
Damien George 9766fddcdc py/mpz: Simplify handling of borrow and quo adjustment in mpn_div.
The motivation behind this patch is to remove unreachable code in mpn_div.
This unreachable code was added some time ago in
9a21d2e070, when a loop in mpn_div was copied
and adjusted to work when mpz_dig_t was exactly half of the size of
mpz_dbl_dig_t (a common case).  The loop was copied correctly but it wasn't
noticed at the time that the final part of the calculation of num-quo*den
could be optimised, and hence unreachable code was left for a case that
never occurred.

The observation for the optimisation is that the initial value of quo in
mpn_div is either exact or too large (never too small), and therefore the
subtraction of quo*den from num may subtract exactly enough or too much
(but never too little).  Using this observation the part of the algorithm
that handles the borrow value can be simplified, and most importantly this
eliminates the unreachable code.

The new code has been tested with DIG_SIZE=3 and DIG_SIZE=4 by dividing all
possible combinations of non-negative integers with between 0 and 3
(inclusive) mpz digits.
2017-12-29 14:05:48 +11:00
Damien George c7cb1dfcb9 py/parse: Fix macro evaluation by avoiding empty __VA_ARGS__.
Empty __VA_ARGS__ are not allowed in the C preprocessor so adjust the rule
arg offset calculation to not use them.  Also, some compilers (eg MSVC)
require an extra layer of macro expansion.
2017-12-29 13:44:26 +11:00
Damien George d3fbfa491f py/parse: Update debugging code to compile on 64-bit arch. 2017-12-29 00:13:36 +11:00
Damien George 0016a45368 py/parse: Compress rule pointer table to table of offsets.
This is the sixth and final patch in a series of patches to the parser that
aims to reduce code size by compressing the data corresponding to the rules
of the grammar.

Prior to this set of patches the rules were stored as rule_t structs with
rule_id, act and arg members.  And then there was a big table of pointers
which allowed to lookup the address of a rule_t struct given the id of that
rule.

The changes that have been made are:
- Breaking up of the rule_t struct into individual components, with each
  component in a separate array.
- Removal of the rule_id part of the struct because it's not needed.
- Put all the rule arg data in a big array.
- Change the table of pointers to rules to a table of offsets within the
  array of rule arg data.

The last point is what is done in this patch here and brings about the
biggest decreases in code size, because an array of pointers is now an
array of bytes.

Code size changes for the six patches combined is:

   bare-arm:  -644
minimal x86: -1856
   unix x64: -5408
unix nanbox: -2080
      stm32:  -720
    esp8266:  -812
     cc3200:  -712

For the change in parser performance: it was measured on pyboard that these
six patches combined gave an increase in script parse time of about 0.4%.
This is due to the slightly more complicated way of looking up the data for
a rule (since the 9th bit of the offset into the rule arg data table is
calculated with an if statement).  This is an acceptable increase in parse
time considering that parsing is only done once per script (if compiled on
the target).
2017-12-29 00:13:36 +11:00
Damien George c2c92ceefc py/parse: Remove rule_t struct because it's no longer needed. 2017-12-28 23:15:36 +11:00
Damien George 66d8885d85 py/parse: Pass rule_id to push_result_token, instead of passing rule_t*. 2017-12-28 23:12:10 +11:00
Damien George 815a8cd1ae py/parse: Pass rule_id to push_result_rule, instead of passing rule_t*.
Reduces code size by eliminating quite a few pointer dereferences.
2017-12-28 23:11:43 +11:00
Damien George 845511af25 py/parse: Break rule data into separate act and arg arrays.
Instead of each rule being stored in ROM as a struct with rule_id, act and
arg, the act and arg parts are now in separate arrays and the rule_id part
is removed because it's not needed.  This reduces code size, by roughly one
byte per grammar rule, around 150 bytes.
2017-12-28 23:09:49 +11:00
Damien George 1039c5e699 py/parse: Split out rule name from rule struct into separate array.
The rule name is only used for debugging, and this patch makes things a bit
cleaner by completely separating out the rule name from the rest of the
rule data.
2017-12-28 23:08:00 +11:00
Damien George b25f92160b py/nlr: Factor out common NLR code to macro and generic funcs in nlr.c.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c, along with a helper macro in nlr.h.  This eliminates duplicated
code.
2017-12-28 16:46:30 +11:00
Damien George 5bf8e85fc8 py/nlr: Clean up selection and config of NLR implementation.
If MICROPY_NLR_SETJMP is not enabled and the machine is auto-detected then
nlr.h now defines some convenience macros for the individual NLR
implementations to use (eg MICROPY_NLR_THUMB).  This keeps nlr.h and the
implementation in sync, and also makes the nlr_buf_t struct easier to read.
2017-12-28 16:18:39 +11:00
Damien George 97cc485538 py/nlrthumb: Fix use of naked funcs, must only contain basic asm code.
A function with a naked attribute must only contain basic inline asm
statements and no C code.

For nlr_push this means removing the "return 0" statement.  But for some
gcc versions this induces a compiler warning so the __builtin_unreachable()
line needs to be added.

For nlr_jump, this function contains a combination of C code and inline asm
so cannot be naked.
2017-12-28 15:59:09 +11:00
Paul Sokolovsky 096e967aad Revert "py/nlr: Factor out common NLR code to generic functions."
This reverts commit 6a3a742a6c.

The above commit has number of faults starting from the motivation down
to the actual implementation.

1. Faulty implementation.

The original code contained functions like:

NORETURN void nlr_jump(void *val) {
    nlr_buf_t **top_ptr = &MP_STATE_THREAD(nlr_top);
    nlr_buf_t *top = *top_ptr;
...
     __asm volatile (
    "mov    %0, %%edx           \n" // %edx points to nlr_buf
    "mov    28(%%edx), %%esi    \n" // load saved %esi
    "mov    24(%%edx), %%edi    \n" // load saved %edi
    "mov    20(%%edx), %%ebx    \n" // load saved %ebx
    "mov    16(%%edx), %%esp    \n" // load saved %esp
    "mov    12(%%edx), %%ebp    \n" // load saved %ebp
    "mov    8(%%edx), %%eax     \n" // load saved %eip
    "mov    %%eax, (%%esp)      \n" // store saved %eip to stack
    "xor    %%eax, %%eax        \n" // clear return register
    "inc    %%al                \n" // increase to make 1, non-local return
     "ret                        \n" // return
    :                               // output operands
    : "r"(top)                      // input operands
    :                               // clobbered registers
     );
}

Which clearly stated that C-level variable should be a parameter of the
assembly, whcih then moved it into correct register.

Whereas now it's:

NORETURN void nlr_jump_tail(nlr_buf_t *top) {
    (void)top;

    __asm volatile (
    "mov    28(%edx), %esi      \n" // load saved %esi
    "mov    24(%edx), %edi      \n" // load saved %edi
    "mov    20(%edx), %ebx      \n" // load saved %ebx
    "mov    16(%edx), %esp      \n" // load saved %esp
    "mov    12(%edx), %ebp      \n" // load saved %ebp
    "mov    8(%edx), %eax       \n" // load saved %eip
    "mov    %eax, (%esp)        \n" // store saved %eip to stack
    "xor    %eax, %eax          \n" // clear return register
    "inc    %al                 \n" // increase to make 1, non-local return
    "ret                        \n" // return
    );

    for (;;); // needed to silence compiler warning
}

Which just tries to perform operations on a completely random register (edx
in this case). The outcome is the expected: saving the pure random luck of
the compiler putting the right value in the random register above, there's
a crash.

2. Non-critical assessment.

The original commit message says "There is a small overhead introduced
(typically 1 machine instruction)". That machine instruction is a call
if a compiler doesn't perform tail optimization (happens regularly), and
it's 1 instruction only with the broken code shown above, fixing it
requires adding more. With inefficiencies already presented in the NLR
code, the overhead becomes "considerable" (several times more than 1%),
not "small".

The commit message also says "This eliminates duplicated code.". An
obvious way to eliminate duplication would be to factor out common code
to macros, not introduce overhead and breakage like above.

3. Faulty motivation.

All this started with a report of warnings/errors happening for a niche
compiler. It could have been solved in one the direct ways: a) fixing it
just for affected compiler(s); b) rewriting it in proper assembly (like
it was before BTW); c) by not doing anything at all, MICROPY_NLR_SETJMP
exists exactly to address minor-impact cases like thar (where a) or b) are
not applicable). Instead, a backwards "solution" was put forward, leading
to all the issues above.

The best action thus appears to be revert and rework, not trying to work
around what went haywire in the first place.
2017-12-26 19:27:58 +02:00
Damien George 26d4a6fa45 py/malloc: Remove unneeded code checking m_malloc return value.
m_malloc already checks for a failed allocation so there's no need to check
for it in m_malloc0.
2017-12-20 16:55:42 +11:00
Damien George 6a3a742a6c py/nlr: Factor out common NLR code to generic functions.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c.  This eliminates duplicated code.

The factoring also allows to make the machine-specific NLR code pure
assembler code, thus allowing nlrthumb.c to use naked function attributes
in the correct way (naked functions can only have basic inline assembler
code in them).

There is a small overhead introduced (typically 1 machine instruction)
because now the generic nlr_jump() must call nlr_jump_tail() rather than
them being one combined function.
2017-12-20 15:42:06 +11:00
Damien George 304a3bcc1c py/modio: Use correct config macro to enable resource_stream function. 2017-12-19 16:59:08 +11:00
Damien George ae1be76d40 py/mpz: Apply a small code-size optimisation. 2017-12-19 15:45:56 +11:00
Damien George 374eaf5271 py/mpz: Fix pow3 function so it handles the case when 3rd arg is 1.
In this case the result should always be 0, even if 2nd arg is 0.
2017-12-19 15:42:58 +11:00
Damien George 7db79d8b03 py/objset: Remove unneeded check from set_equal.
set_equal is called only from set_binary_op, and this guarantees that the
second arg to set_equal is always a set or frozenset.  So there is no need
to do a further check.
2017-12-19 14:01:19 +11:00
Damien George 136cb7f27c py/map: Don't include ordered-dict mutating code when not needed. 2017-12-19 13:37:15 +11:00
Damien George f5fb68e94f py/runtime: Remove unnecessary break statements from switch. 2017-12-19 13:13:21 +11:00
Paul Sokolovsky 6364401666 py/objgenerator: Allow to pend an exception for next execution.
This implements .pend_throw(exc) method, which sets up an exception to be
triggered on the next call to generator's .__next__() or .send() method.
This is unlike .throw(), which immediately starts to execute the generator
to process the exception. This effectively adds Future-like capabilities
to generator protocol (exception will be raised in the future).

The need for such a method arised to implement uasyncio wait_for() function
efficiently (its behavior is clearly "Future" like, and normally would
require to introduce an expensive Future wrapper around all native
couroutines, like upstream asyncio does).

py/objgenerator: pend_throw: Return previous pended value.

This effectively allows to store an additional value (not necessary an
exception) in a coroutine while it's not being executed. uasyncio has
exactly this usecase: to mark a coro waiting in I/O queue (and thus
not executed in the normal scheduling queue), for the purpose of
implementing wait_for() function (cancellation of such waiting coro
by a timeout).
2017-12-15 20:20:36 +02:00
Damien George cf8e8c29e7 py/emitglue: Change type of bit-field to explicitly unsigned mp_uint_t.
Some compilers can treat enum types as signed, in which case 3 bits is not
enough to encode all mp_raw_code_kind_t values.  So change the type to
mp_uint_t.
2017-12-15 10:21:10 +11:00
Damien George f1c9e7760d py/builtinimport: Call __init__ for modules imported via a weak link.
This is a bit of a clumsy way of doing it but solves the issue of __init__
not running when a module is imported via its weak-link name.  Ideally a
better solution would be found.
2017-12-13 14:48:53 +11:00
Damien George c78ef92d78 py/objtype: Refactor object's handling of __new__ to not create 2 objs.
Before this patch, if a user defined the __new__() function for a class
then two instances of that class would be created: once before __new__ is
called and once during the __new__ call (assuming the user creates some
instance, eg using super().__new__, which is most of the time).  The first
one was then discarded.  This refactor makes it so that a new instance is
only created if the user __new__ function doesn't exist.
2017-12-12 16:53:44 +11:00
Damien George d32d22dfd7 py/objtype: Implement better support for overriding native's __init__.
This patch cleans up and generalises part of the code which handles
overriding and calling a native base-class's __init__ method.  It defers
the call to the native make_new() function until after the user (Python)
__init__() method has run.  That user method now has the chance to call the
native __init__/make_new and pass it different arguments.  If the user
doesn't call the super().__init__ method then it will be called
automatically after the user code finishes, to finalise construction of the
instance.
2017-12-12 16:43:16 +11:00
Damien George d3f82bc425 py/mpstate.h: Remove obsolete comment about nlr_top being coded in asm. 2017-12-11 22:51:52 +11:00
Damien George 2759bec858 py: Extend nan-boxing config to have 47-bit small integers.
The nan-boxing representation has an extra 16-bits of space to store
small-int values, and making use of it allows to create and manipulate full
32-bit positive integers (ie up to 0xffffffff) without using the heap.
2017-12-11 22:39:12 +11:00
Damien George 9c02707356 py/objexcept: Use INT_FMT when printing errno value. 2017-12-11 22:38:30 +11:00
Damien George 30fd8484eb py/runtime: Use the Python stack when building *arg and **kwarg state.
With MICROPY_ENABLE_PYSTACK enabled the following language constructs no
longer allocate on the heap: f(*arg), f(**kwarg).
2017-12-11 13:49:09 +11:00
Damien George 1e5a33df41 py: Convert all uses of alloca() to use new scoped allocation API. 2017-12-11 13:49:09 +11:00
Damien George 02d830c035 py: Introduce a Python stack for scoped allocation.
This patch introduces the MICROPY_ENABLE_PYSTACK option (disabled by
default) which enables a "Python stack" that allows to allocate and free
memory in a scoped, or Last-In-First-Out (LIFO) way, similar to alloca().

A new memory allocation API is introduced along with this Py-stack.  It
includes both "local" and "nonlocal" LIFO allocation.  Local allocation is
intended to be equivalent to using alloca(), whereby the same function must
free the memory.  Nonlocal allocation is where another function may free
the memory, so long as it's still LIFO.

Follow-up patches will convert all uses of alloca() and VLA to the new
scoped allocation API.  The old behaviour (using alloca()) will still be
available, but when MICROPY_ENABLE_PYSTACK is enabled then alloca() is no
longer required or used.

The benefits of enabling this option are (or will be once subsequent
patches are made to convert alloca()/VLA):
- Toolchains without alloca() can use this feature to obtain correct and
  efficient scoped memory allocation (compared to using the heap instead
  of alloca(), which is slower).
- Even if alloca() is available, enabling the Py-stack gives slightly more
  efficient use of stack space when calling nested Python functions, due to
  the way that compilers implement alloca().
- Enabling the Py-stack with the stackless mode allows for even more
  efficient stack usage, as well as retaining high performance (because the
  heap is no longer used to build and destroy stackless code states).
- With Py-stack and stackless enabled, Python-calling-Python is no longer
  recursive in the C mp_execute_bytecode function.

The micropython.pystack_use() function is included to measure usage of the
Python stack.
2017-12-11 13:49:09 +11:00
Damien George 5b8998da6d py/runtime: Move mp_exc_recursion_depth to runtime and rename to raise.
For consistency this helper function is renamed to match the other
exception helpers, and moved to their location in runtime.c.
2017-12-11 13:49:09 +11:00
Paul Sokolovsky e7fc765880 unix/mpconfigport: Disable uio.resource_stream().
This function was implemented as an experiment, and was enabled only in
unix port. To remind, it allows to access arbitrary files frozen as
source modules (vs bytecode).

However, further experimentation showed that the same functionality can
be implemented with frozen bytecode. The process requires more steps, but
with suitable toolset it doesn't matter patch. This process is:

1. Convert binary files into "Python resource module" with
tools/mpy_bin2res.py.
2. Freeze as the bytecode.
3. Use micropython-lib's pkg_resources.resource_stream() to access it.

In other words, the extra step is using tools/mpy_bin2res.py (because
there would be wrapper for uio.resource_stream() anyway).

Going frozen bytecode route allows more flexibility, and same/additional
efficiency:

1. Frozen source support can be disabled altogether for additional code
savings.
2. Resources could be also accessed as a buffer, not just as a stream.

There're few caveats too:

1. It wasn't actually profiled the overhead of storing a resource in
"Python resource module" vs storing it directly, but it's assumed that
overhead is small.
2. The "efficiency" claim above applies to the case when resource
file is frozen as the bytecode. If it's not, it actually will take a
lot of RAM on loading. But in this case, the resource file should not
be used (i.e. generated) in the first place, and micropython-lib's
pkg_resources.resource_stream() implementation has the appropriate
fallback to read the raw files instead. This still poses some distribution
issues, e.g. to deployable to baremetal ports (which almost certainly
would require freezeing as the bytecode), a distribution package should
include the resource module. But for non-freezing deployment, presense
of resource module will lead to memory inefficiency.

All the discussion above reminds why uio.resource_stream() was implemented
in the first place - to address some of the issues above. However, since
then, frozen bytecode approach seems to prevail, so, while there're still
some issues to address with it, this change is being made.

This change saves 488 bytes for the unix x86_64 port.
2017-12-10 02:38:23 +02:00
Paul Sokolovsky d21d029d55 py/mkrules.mk: Add "clean-frozen" target to clean frozen script/modules dir.
This target removes any stray files (i.e. something not committed to git)
from scripts/ and modules/ dirs (or whatever FROZEN_DIR and FROZEN_MPY_DIR
is set to).

The expected workflow is:

1. make clean-frozen
2. micropython -m upip -p modules <packages_to_freeze>
3. make

As it can be expected that people may drop random thing in those dirs which
they can miss later, the content is actually backed up before cleaning.
2017-12-10 01:05:29 +02:00
Paul Sokolovsky a35d923cdf py/map: Allow to trace rehashing operations. 2017-12-09 17:32:16 +02:00
Paul Sokolovsky 2b00181592 py/objfun: Factor out macro for initializing codestate.
This is second part of fun_bc_call() vs mp_obj_fun_bc_prepare_codestate()
common code refactor. This factors out code to initialize codestate
object. After this patch, mp_obj_fun_bc_prepare_codestate() is effectively
DECODE_CODESTATE_SIZE() followed by allocation followed by
INIT_CODESTATE(), and fun_bc_call() starts with that too.
2017-12-09 12:49:00 +02:00
Paul Sokolovsky d72370def7 py/objfun, vm: Add comments on codestate allocation in stackless mode. 2017-12-09 11:01:34 +02:00
Paul Sokolovsky fca1d1aa62 py/objfun: Factor out macro for decoding codestate size.
fun_bc_call() starts with almost the same code as
mp_obj_fun_bc_prepare_codestate(), the only difference is a way to
allocate the codestate object (heap vs stack with heap fallback).
Still, would be nice to avoid code duplication to make further
refactoring easier.

So, this commit factors out the common code before the allocation -
decoding and calculating codestate size. It produces two values,
so structured as a macro which writes to 2 variables passed as
arguments.
2017-12-09 09:19:34 +02:00
Paul Sokolovsky dea3fb93c7 py/gc: In sweep debug output, print pointer as a pointer.
Or it will be truncated on a 64-bit platform.
2017-12-09 01:54:01 +02:00
Paul Sokolovsky 5453d88d5d py/gc: Factor out a macro to trace GC mark operations.
To allow easier override it for custom tracing.
2017-12-09 01:48:26 +02:00
Paul Sokolovsky 39dd89fe31 py/runtime: When tracing unary/binary ops, output op (method) name.
E.g.:

    >>> 1+1
    binary 26 __add__ 3 3

Output is similar to bytecode dump (numeric code, then op name).
2017-12-09 01:28:16 +02:00
Paul Sokolovsky c0877cbb0d py/objint_longlong: Check for zero division/modulo. 2017-12-08 20:40:55 +02:00
Damien George 53e111800f py/asmbase: Revert removal of clearing of label offsets for native emit.
The assembler back-end for most architectures needs to know if a jump is
backwards in order to emit optimised machine code, and they do this by
checking if the destination label has been set or not.  So always reset
label offsets to -1 (this reverts partially the previous commit, with some
minor optimisation for the if-logic with the pass variable).
2017-12-08 19:07:00 +11:00
Damien George f935bce3c5 py/{emitbc,asmbase}: Only clear emit labels to -1 when in debug mode.
Clearing the labels to -1 is purely a debugging measure.  For release
builds there is no need to do it as the label offset table should always
have the correct value assigned.
2017-12-08 18:23:23 +11:00
Paul Sokolovsky 9ef4be8b41 py/gc: Add CLEAR_ON_SWEEP option to debug mis-traced objects.
Accessing them will crash immediately instead still working for some time,
until overwritten by some other data, leading to much less deterministic
crashes.
2017-12-08 00:10:44 +02:00
Paul Sokolovsky 9ebc037eee py/malloc: Allow to use debug logging if !MICROPY_MALLOC_USES_ALLOCATED_SIZE.
This is mostly a workaround for forceful rebuilding of mpy-cross on every
codebase change. If this file has debug logging enabled (by patching),
mpy-cross build failed.
2017-12-07 18:01:40 +02:00
Paul Sokolovsky 88a8043a27 py/malloc: MICROPY_MEM_STATS requires MICROPY_MALLOC_USES_ALLOCATED_SIZE.
Error out if they're set incompatibly.
2017-12-07 10:52:40 +02:00
Paul Sokolovsky f5e097021c py/mpprint: Fix "%x" vs "%X" regression introduced in previous commit. 2017-12-07 10:31:14 +02:00
Paul Sokolovsky 5a10e63543 py/mpprint: Support "%lx" format on 64-bit systems.
Before that, the output was truncated to 32 bits. Only "%x" format is
handled, because a typical use is for addresses.

This refactor actually decreased x86_64 code size by 30 bytes.
2017-12-07 10:07:18 +02:00
Paul Sokolovsky 5f8ad284f8 py/mpprint: Make "%p" format work properly on 64-bit systems.
Before, the output was truncated to 32 bits.
2017-12-07 09:06:07 +02:00
Damien George 58f00d7c0e py/modbuiltins: Use standard arg-parsing helper func for builtin print.
This allows the function to raise an exception when unknown keyword args
are passed in.  This patch also reduces code size by (in bytes):

   bare-arm:   -24
minimal x86:   -76
   unix x64:   -56
unix nanbox:   -84
      stm32:   -40
    esp8266:   -68
     cc3200:   -48

Furthermore, this patch adds space (" ") to the set of ROM qstrs which
means it doesn't need to be put in RAM if it's ever used.
2017-12-05 12:14:57 +11:00
Paul Sokolovsky 62b96147e6 py: mp_call_function_*_protected(): Pass-thru return value if possible.
Return the result of called function. If exception happened, return
MP_OBJ_NULL. Allows to use mp_call_function_*_protected() with callbacks
returning values, etc.
2017-12-05 00:38:41 +02:00
Paul Sokolovsky 75d3c046da py/misc.h: Add m_new_obj_var_with_finaliser().
Similar to existing m_new_obj_with_finaliser().
2017-12-04 11:05:49 +02:00
Damien George 64f11470be py/objgenerator: Remove unreachable code for STOP_ITERATION case.
This commit essentially reverts aa9dbb1b03
where this if-condition was added.  It seems that even when that commit
was made the code was never reached by any tests, nor reachable by
analysis (see below).  The same is true with the code as it currently
stands: no test triggers this if-condition, nor any uasyncio examples.
Analysing the flow of the program also shows that it's not reachable:

==START==
-> to trigger this if condition mp_execute_bytecode() must return
   MP_VM_RETURN_YIELD with *sp==MP_OBJ_STOP_ITERATION

   -> mp_execute_bytecode() can only return MP_VM_RETURN_YIELD from the
      MP_BC_YIELD_VALUE bytecode, which can happen in 2 ways:

      -> 1) from a "yield <x>" in bytecode, but <x> must always be a proper
         object, never MP_OBJ_STOP_ITERATION; ==END1==

      -> 2) via yield from, via mp_resume() which must return
         MP_VM_RETURN_YIELD with ret_value==MP_OBJ_STOP_ITERATION, which
         can happen in 3 ways:

         -> 1) it delegates to mp_obj_gen_resume(); go back to ==START==

         -> 2) it returns MP_VM_RETURN_YIELD directly but with a guard that
            ret_val!=MP_OBJ_STOP_ITERATION; ==END2==

         -> 3) it returns MP_VM_RETURN_YIELD with ret_val set from
            mp_call_method_n_kw(), but mp_call_method_n_kw() must return a
            proper object, never MP_OBJ_STOP_ITERATION; ==END3==

The above shows there is no way to trigger the if-condition and it can be
removed.
2017-11-30 12:06:41 +11:00
Damien George 74fad3536b py/gc: In gc_realloc, convert pointer sanity checks to assertions.
These checks are assumed to be true in all cases where gc_realloc is
called with a valid pointer, so no need to waste code space and time
checking them in a non-debug build.
2017-11-29 17:17:08 +11:00
Damien George 8e323b8fa8 py/qstr: Rewrite find_qstr to make manifest that it returns a valid ptr.
So long as the input qstr identifier is valid (below the maximum number of
qstrs) the function will always return a valid pointer.  This patch
eliminates the "return 0" dead-code.
2017-11-29 17:01:39 +11:00
Damien George 3990a52c0f py: Annotate func defs with NORETURN when their corresp decls have it. 2017-11-29 15:43:40 +11:00
Damien George 2161d6b603 py/objdict: Reuse dict-view key iterator for standard dict iterator.
It has equivalent behaviour and reusing it saves some code bytes.
2017-11-27 23:40:31 +11:00
Damien George 84895f1a21 py/parsenum: Improve parsing of floating point numbers.
This patch improves parsing of floating point numbers by converting all the
digits (integer and fractional) together into a number 1 or greater, and
then applying the correct power of 10 at the very end.  In particular the
multiple "multiply by 0.1" operations to build a fraction are now combined
together and applied at the same time as the exponent, at the very end.

This helps to retain precision during parsing of floats, and also includes
a check that the number doesn't overflow during the parsing.  One benefit
is that a float will have the same value no matter where the decimal point
is located, eg 1.23 == 123e-2.
2017-11-27 12:51:52 +11:00
Damien George 5e34a113ea py/runtime: Add MP_BINARY_OP_CONTAINS as reverse of MP_BINARY_OP_IN.
Before this patch MP_BINARY_OP_IN had two meanings: coming from bytecode it
meant that the args needed to be swapped, but coming from within the
runtime meant that the args were already in the correct order.  This lead
to some confusion in the code and comments stating how args were reversed.
It also lead to 2 bugs: 1) containment for a subclass of a native type
didn't work; 2) the expression "{True} in True" would illegally succeed and
return True.  In both of these cases it was because the args to
MP_BINARY_OP_IN ended up being reversed twice.

To fix these things this patch introduces MP_BINARY_OP_CONTAINS which
corresponds exactly to the __contains__ special method, and this is the
operator that built-in types should implement.  MP_BINARY_OP_IN is now only
emitted by the compiler and is converted to MP_BINARY_OP_CONTAINS by
swapping the arguments.
2017-11-24 14:48:23 +11:00
Damien George 5b2f62aff3 py/opmethods: Include the correct header for binary op enums.
By directly including runtime0.h the mpconfig.h settings are not included
and so the enums in runtime0.h can be incorrect.
2017-11-24 12:16:21 +11:00
Damien George 9783ac282e py/runtime: Simplify handling of containment binary operator.
In mp_binary_op, there is no need to explicitly check for type->getiter
being non-null and raising an exception because this is handled exactly by
mp_getiter().  So just call the latter unconditionally.
2017-11-24 12:07:12 +11:00
Damien George d5cf5f70fd py/modbuiltins: Slightly simplify code in builtin round(). 2017-11-22 15:51:51 +11:00
Damien George a07fc5b640 py/objfloat: Allow float() to parse anything with the buffer protocol.
This generalises and simplifies the code and follows CPython behaviour.
2017-11-21 15:01:38 +11:00
Paul Sokolovsky 8667a5f053 py/objnamedtuple: Allow to reuse namedtuple basic functionality.
By declaring interface in objnamedtuple.h and introducing a helper
allocation function.
2017-11-20 09:30:06 +02:00
Damien George da154fdaf9 py: Add config option to disable multiple inheritance.
This patch introduces a new compile-time config option to disable multiple
inheritance at the Python level: MICROPY_MULTIPLE_INHERITANCE.  It is
enabled by default.

Disabling multiple inheritance eliminates a lot of recursion in the call
graph (which is important for some embedded systems), and can be used to
reduce code size for ports that are really constrained (by around 200 bytes
for Thumb2 archs).

With multiple inheritance disabled all tests in the test-suite pass except
those that explicitly test for multiple inheritance.
2017-11-20 16:18:50 +11:00
Damien George 8d956c26d1 py/objstr: When constructing str from bytes, check for existing qstr.
This patch uses existing qstr data where possible when constructing a str
from a bytes object.
2017-11-16 14:02:28 +11:00
Damien George 1f1d5194d7 py/objstr: Make mp_obj_new_str_of_type check for existing interned qstr.
The function mp_obj_new_str_of_type is a general str object constructor
used in many places in the code to create either a str or bytes object.
When creating a str it should first check if the string data already exists
as an interned qstr, and if so then return the qstr object.  This patch
makes the function have such behaviour, which helps to reduce heap usage by
reusing existing interned data where possible.

The old behaviour of mp_obj_new_str_of_type (which didn't check for
existing interned data) is made available through the function
mp_obj_new_str_copy, but should only be used in very special cases.

One consequence of this patch is that the following expression is now True:

    'abc' is ' abc '.split()[0]
2017-11-16 13:53:04 +11:00
Damien George 4601759bf5 py/objstr: Remove "make_qstr_if_not_already" arg from mp_obj_new_str.
This patch simplifies the str creation API to favour the common case of
creating a str object that is not forced to be interned.  To force
interning of a new str the new mp_obj_new_str_via_qstr function is added,
and should only be used if warranted.

Apart from simplifying the mp_obj_new_str function (and making it have the
same signature as mp_obj_new_bytes), this patch also reduces code size by a
bit (-16 bytes for bare-arm and roughly -40 bytes on the bare-metal archs).
2017-11-16 13:17:51 +11:00
Christopher Arndt 1871a924c9 py/mkenv.mk: Use $(PYTHON) consistently when calling Python tools.
Rationale:

* Calling Python build tool scripts from makefiles should be done
  consistently using `python </path/to/script>`, instead of relying on the
  correct she-bang line in the script [1] and the executable bit on the
  script being set. This is more platform-independent.
* The name/path of the Python executable should always be used via the
  makefile variable `PYTHON` set in `py/mkenv.mk`. This way it can be
  easily overwritten by the user with `make PYTHON=/path/to/my/python`.
* The Python executable name should be part of the value of the makefile
  variable, which stands for the build tool command (e.g. `MAKE_FROZEN` and
  `MPY_TOOL`), not part of the command line where it is used. If a Python
  tool is substituted by another (non-python) program, no change to the
  Makefiles is necessary, except in `py/mkenv.mk`.
* This also solves #3369 and #1616.

[1] There are systems, where even the assumption that `/usr/bin/env` always
exists, doesn't hold true, for example on Android (where otherwise the unix
port compiles perfectly well).
2017-11-15 11:56:58 +11:00
Damien George 564a95cb04 py/emitnative: Clean up asm macro names so they have dest as first arg.
All the asm macro names that convert a particular architecture to a generic
interface now follow the convention whereby the "destination" (usually a
register) is specified first.
2017-11-15 11:46:49 +11:00
stijn 79ed58f87b py/objnamedtuple: Add _asdict function if OrderedDict is supported 2017-11-12 14:16:54 +02:00
Paul Sokolovsky cada971113 py/objtype: mp_obj_new_type: Name base types related vars more clearly.
As vars contains array of base types and its length, name them as such,
avoid generic "items" and "len" names.
2017-11-11 00:11:24 +02:00
Paul Sokolovsky 1b146e9de9 py/mpconfig: Introduce reusable MP_HTOBE32(), etc. macros.
Macros to convert big-endian values to host byte order and vice-versa.
These were defined in adhoc way for some ports (e.g. esp8266), allow
reuse, provide default implementations, while allow ports to override.
2017-11-08 19:47:37 +02:00
Damien George 487dbdb267 py/compile: Use alloca instead of qstr_build when compiling import name.
The technique of using alloca is how dotted import names are composed in
mp_import_from and mp_builtin___import__, so use the same technique in the
compiler.  This puts less pressure on the heap (only the stack is used if
the qstr already exists, and if it doesn't exist then the standard qstr
block memory is used for the new qstr rather than a separate chunk of the
heap) and reduces overall code size.
2017-11-01 13:16:16 +11:00
Damien George 02b4b23319 Revert "py/{mkenv.mk,mkrules.mk}: Append .exe for Windows executable files."
This reverts commit 3289b9b7a7.
The commit broke building on MINGW because the filename became
micropython.exe.exe.  A proper solution to support more Windows build
environments requires more thought and testing.
2017-10-31 22:01:56 +11:00
Eric Poulsen 74ec52d857 extmod/modussl: Add finaliser support for ussl objects.
Per the comment found here
https://github.com/micropython/micropython-esp32/issues/209#issuecomment-339855157,
this patch adds finaliser code to prevent memory leaks from ussl objects,
which is especially useful when memory for a ussl context is allocated
outside the uPy heap.  This patch is in-line with the finaliser code found
in many modsocket implementations for various ports.

This feature is configured via MICROPY_PY_USSL_FINALISER and is disabled by
default because there may be issues using it when the ussl state *is*
allocated on the uPy heap, rather than externally.
2017-10-30 15:25:32 +11:00
Paul Sokolovsky 0e80f345f8 py/objtype: Introduce MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS.
This allows to configure support for inplace special methods separately,
similar to "normal" and reverse special methods. This is useful, because
inplace methods are "the most optional" ones, for example, if inplace
methods aren't defined, the operation will be executed using normal
methods instead.

As a caveat, __iadd__ and __isub__ are implemented even if
MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS isn't defined. This is similar
to the state of affairs before binary operations refactor, and allows
to run existing tests even if MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS
isn't defined.
2017-10-27 22:29:15 +03:00
Paul Sokolovsky 9b9dbc5815 py/objtype: Define all special methods if requested.
If MICROPY_PY_ALL_SPECIAL_METHODS is defined, actually define all special
methods (still subject to gating by e.g. MICROPY_PY_REVERSE_SPECIAL_METHODS).

This adds quite a number of qstr's, so should be used sparingly.
2017-10-27 20:06:35 +03:00
Damien George f4059dcc0c all: Use NULL instead of "" when calling mp_raise exception helpers.
This is the established way of doing it and reduces code size by a little
bit.
2017-10-24 22:39:36 +11:00
Paul Sokolovsky 9956fd0710 py/objtype: Fit qstrs for special methods in byte type.
Update makeqstrdata.py to sort strings starting with "__" to the beginning
of qstr list, so they get low qstr id's, guaranteedly fitting in 8 bits.
Then use this property to further compact op_id => qstr mapping arrays.
2017-10-21 11:06:32 +03:00
Paul Sokolovsky f2baa9ec24 py/objtype: Use CPython compatible method name for sizeof.
Per https://docs.python.org/3/library/sys.html#sys.getsizeof:

getsizeof() calls the object’s __sizeof__ method. Previously, "getsizeof"
was used mostly to save on new qstr, as we don't really support calling
this method on arbitrary objects (so it was used only for reporting).
However, normalize it all now.
2017-10-19 12:44:53 +03:00
Damien George 93ce125abe py/argcheck: Remove #if guard around terse error message helper func.
Not all compilers/analysers are smart enough to realise that this function
is never called if MICROPY_ERROR_REPORTING is not TERSE, because the logic
in the code uses if statements rather than #if to select whether to call
this function or not (MSC in debug mode is an example of this, but there
are others).  So just unconditionally compile this helper function.  The
code-base anyway relies on the linker to remove unused functions.
2017-10-19 18:57:26 +11:00
Damien George 37282f8fc1 extmod/uos_dupterm: Update uos.dupterm() and helper funcs to have index.
The uos.dupterm() signature and behaviour is updated to reflect the latest
enhancements in the docs.  It has minor backwards incompatibility in that
it no longer accepts zero arguments.

The dupterm_rx helper function is moved from esp8266 to extmod and
generalised to support multiple dupterm slots.

A port can specify multiple slots by defining the MICROPY_PY_OS_DUPTERM
config macro to an integer, being the number of slots it wants to have;
0 means to disable the dupterm feature altogether.

The unix and esp8266 ports are updated to work with the new interface and
are otherwise unchanged with respect to functionality.
2017-10-13 20:01:57 +11:00
Damien George c59fc1419d py/emitnative: Simplify binary op emitter, no need to check inplace ops. 2017-10-12 12:26:49 +11:00
Damien George a3afa8cfc4 py/emitnative: Implement floor-division and modulo for viper emitter. 2017-10-11 18:54:34 +11:00
Damien George 69da74e538 py/modbuiltins: Use existing utf8_get_char helper in builtin ord func. 2017-10-11 11:25:20 +11:00
Damien George dc948e4d54 py/formatfloat: Use standard isinf, isnan funcs instead of custom ones.
Reduces code size by a tiny bit.
2017-10-10 16:27:54 +11:00
Damien George 08a196697c py/formatfloat: Don't print the negative sign of a NaN value.
NaN may have the sign bit set but it has no meaning, so don't print it out.
2017-10-10 16:01:13 +11:00
Damien George d8d4e4dfbe py/modmath: Convert log2 macro into a function.
So that a pointer to it can be passed as a pointer to math_generic_1.  This
patch also makes the function work for single and double precision floating
point.
2017-10-10 16:01:04 +11:00
Damien George 25e140652b py/modmath: Add full checks for math domain errors.
This patch changes how most of the plain math functions are implemented:
there are now two generic math wrapper functions that take a pointer to a
math function (like sin, cos) and perform the necessary conversion to and
from MicroPython types.  This helps to reduce code size.  The generic
functions can also check for math domain errors in a generic way, by
testing if the result is NaN or infinity combined with finite inputs.

The result is that, with this patch, all math functions now have full
domain error checking (even gamma and lgamma) and code size has decreased
for most ports.  Code size changes in bytes for those with the math module
are:

   unix x64:  -432
unix nanbox:  -792
      stm32:   -88
    esp8266:   +12

Tests are also added to check domain errors are handled correctly.
2017-10-10 15:57:45 +11:00
Damien George 933eab46fc py/bc: Update opcode_format_table to match the bytecode. 2017-10-10 10:37:38 +11:00
Damien George ff93fd4f50 py/persistentcode: Bump .mpy version number to version 3.
The binary and unary ops have changed bytecode encoding.
2017-10-05 10:49:44 +11:00
Damien George 36f7952f76 py/objtype: Clean up unary- and binary-op enum-to-qstr mapping tables. 2017-10-05 10:49:44 +11:00
Damien George 0864a6957f py: Clean up unary and binary enum list to keep groups together.
2 non-bytecode binary ops (NOT_IN and IN_NOT) are moved out of the
bytecode group, so this change will change the bytecode format.
2017-10-05 10:49:44 +11:00
Damien George 23faf88cab py/mpprint: Only check for null string printing when NDEBUG not defined.
Printing "(null)" when a NULL string pointer is passed to %s is a debugging
feature and not a feature that's relied upon by the code.  So it only needs
to be compiled in when debugging (such as assert) is enabled, and saves
roughy 30 bytes of code when disabled.

This patch also fixes this NULL check to not do the check if the precision
is specified as zero.
2017-10-04 18:07:32 +11:00
Damien George dfa563c71f py/objstr: Make empty bytes object have a null-terminating byte.
Because a lot of string processing functions assume there is a null
terminating byte, so they can work in an efficient way.

Fixes issue #3334.
2017-10-04 17:59:22 +11:00
Damien George a3dc1b1957 all: Remove inclusion of internal py header files.
Header files that are considered internal to the py core and should not
normally be included directly are:
    py/nlr.h - internal nlr configuration and declarations
    py/bc0.h - contains bytecode macro definitions
    py/runtime0.h - contains basic runtime enums

Instead, the top-level header files to include are one of:
    py/obj.h - includes runtime0.h and defines everything to use the
        mp_obj_t type
    py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
        and defines everything to use the general runtime support functions

Additional, specific headers (eg py/objlist.h) can be included if needed.
2017-10-04 12:37:50 +11:00
Damien George 6c82cfc089 py/objtype: Change type of enum-to-qstr table to uint16_t to save space.
Qstr values fit in 16-bits (and this fact is used elsewhere in the code) so
no need to use more than that for the large lookup tables.  The compiler
will anyway give a warning if the qstr values don't fit in 16 bits.  Saves
around 80 bytes of code space for Thumb2 archs.
2017-10-04 11:31:05 +11:00
chrismas9 3289b9b7a7 py/{mkenv.mk,mkrules.mk}: Append .exe for Windows executable files.
Building mpy-cross: this patch adds .exe to the PROG name when building
executables for host (eg mpy-cross) on Windows.  make clean now removes
mpy-cross.exe under Windows.

Building MicroPython: this patch sets MPY_CROSS to mpy-cross.exe or
mpy-cross so they can coexist and use cygwin or WSL without rebuilding
mpy-cross.  The dependency in the mpy rule now uses mpy-cross.exe for
Windows and mpy-cross for Linux.
2017-10-04 00:21:05 +11:00
Damien George 1394258f37 py/objset: Include the failed key in a KeyError raised from set.remove. 2017-10-03 18:03:06 +11:00
Damien George 2ac1364688 py/objset: Check that RHS of a binary op is a set/frozenset.
CPython docs explicitly state that the RHS of a set/frozenset binary op
must be a set to prevent user errors.  It also preserves commutativity of
the ops, eg: "abc" & set() is a TypeError, and so should be set() & "abc".

This change actually decreases unix (x64) code by 160 bytes; it increases
stm32 by 4 bytes and esp8266 by 28 bytes (but previous patch already
introduced a much large saving).
2017-10-03 17:56:27 +11:00
Damien George 01978648fd py/objset: Simplify set and frozenset by separating their locals dicts.
A lot of set's methods (the mutable ones) are not allowed to operate on a
frozenset, and giving frozenset a separate locals dict with only the
methods that it supports allows to simplify the logic that verifies if
args are a set or a frozenset.  Even though the new frozenset locals dict
is relatively large (88 bytes on 32-bit archs) there is a much bigger
saving coming from the removal of a const string for an error message,
along with the removal of some checks for set or frozenset type.

Changes in code size due to this patch are (for ports that changed at all):

   unix x64:   -56
unix nanbox:  -304
      stm32:   -64
    esp8266:  -124
     cc3200:   -40

Apart from the reduced code, frozenset now has better tab-completion
because it only lists the valid methods.  And the error message for
accessing an invalid method is now more detailed (it includes the
method name that wasn't found).
2017-10-03 17:55:53 +11:00
Damien George bdc6e86e07 py/objfloat: Support raising a negative number to a fractional power.
This returns a complex number, following CPython behaviour.  For ports that
don't have complex numbers enabled this will raise a ValueError which gives
a fail-safe for scripts that were written assuming complex numbers exist.
2017-09-26 12:57:51 +10:00
David Lechner 62849b7010 py: Add config option to print warnings/errors to stderr.
This adds a new configuration option to print runtime warnings and errors to
stderr. On Unix, CPython prints warnings and unhandled exceptions to stderr,
so the unix port here is configured to use this option.

The unix port already printed unhandled exceptions on the main thread to
stderr. This patch fixes unhandled exceptions on other threads and warnings
(issue #2838) not printing on stderr.

Additionally, a couple tests needed to be fixed to handle this new behavior.
This is done by also capturing stderr when running tests.
2017-09-26 11:59:11 +10:00
Paul Sokolovsky 9d836fedbd py: Clarify which mp_unary_op_t's may appear in the bytecode.
Not all can, so we don't need to reserve bytecodes for them, and can
use free slots for something else later.
2017-09-25 16:35:19 -07:00
Anton Patrushev f008263022 py/persistentcode: Define mp_raw_code_save_file() for any unix target.
A unix target should provide POSIX open/write/close functions regardless of
its machine architecture.  Fixes issue #3325.
2017-09-25 17:09:05 +10:00
Damien George 8edc2e4b14 py/runtime0: Add comments about unary/binary-op enums used in bytecode. 2017-09-22 11:54:08 +10:00
Damien George e2ba45c35f py/vm: Use lowercase letter at start of exception message.
For consistency with all the other exception messages.
2017-09-22 11:28:45 +10:00
Damien George ede8a0235b py/vstr: Raise a RuntimeError if fixed vstr buffer overflows.
Current users of fixed vstr buffers (building file paths) assume that there
is no overflow and do not check for overflow after building the vstr.  This
has the potential to lead to NULL pointer dereferences
(when vstr_null_terminated_str returns NULL because it can't allocate RAM
for the terminating byte) and stat'ing and loading invalid path names (due
to the path being truncated).  The safest and simplest thing to do in these
cases is just raise an exception if a write goes beyond the end of a fixed
vstr buffer, which is what this patch does.  It also simplifies the vstr
code.
2017-09-21 20:29:41 +10:00
Damien George 7885a425d7 py/stream: Remove unnecessary checks for NULL return from vstr_add_len.
The vstr argument to the calls to vstr_add_len are dynamically allocated
(ie fixed_buf=false) and so vstr_add_len will never return NULL.  So
there's no need to check for it.  Any out-of-memory errors are raised by
the call to m_renew in vstr_ensure_extra.
2017-09-21 18:22:55 +10:00
Damien George 96fd80db13 py/objexcept: Prevent infinite recursion when allocating exceptions.
The aim of this patch is to rewrite the functions that create exception
instances (mp_obj_exception_make_new and mp_obj_new_exception_msg_varg) so
that they do not call any functions that may raise an exception.  Otherwise
it's possible to create infinite recursion with an exception being raised
while trying to create an exception object.

The two main things that are done to accomplish this are:
1. Change mp_obj_new_exception_msg_varg to just format the string, then
   call mp_obj_exception_make_new to actually create the exception object.
2. In mp_obj_exception_make_new and mp_obj_new_exception_msg_varg try to
   allocate all memory first using functions that don't raise exceptions
   If any of the memory allocations fail (return NULL) then degrade
   gracefully by trying other options for memory allocation, eg using the
   emergency exception buffer.
3. Use a custom printer backend to conservatively format strings: if it
   can't allocate memory then it just truncates the string.

As part of this rewrite, raising an exception without a message, like
KeyError(123), will now use the emergency buffer to store the arg and
traceback data if there is no heap memory available.

Memory use with this patch is unchanged.  Code size is increased by:

   bare-arm:  +136
minimal x86:  +124
   unix x64:   +72
unix nanbox:   +96
      stm32:   +88
    esp8266:   +92
     cc3200:   +80
2017-09-21 15:24:57 +10:00
Paul Sokolovsky fc9a6dd09e py/objstr: strip: Don't strip "\0" by default.
An issue was due to incorrectly taking size of default strip characters
set.
2017-09-19 21:21:12 +03:00
Damien George 44f0a4d1e7 py/mpconfig.h: Add note that using computed gotos in VM is not C99. 2017-09-18 23:53:33 +10:00
Damien George fdb2aa81b7 py/{objfloat,objcomplex}: Optimise MP_UNARY_OP_ABS by reusing variables. 2017-09-18 14:31:03 +10:00
Paul Sokolovsky 9dce823cfd py/modbuiltins: Implement abs() by dispatching to MP_UNARY_OP_ABS.
This allows user classes to implement __abs__ special method, and saves
code size (104 bytes for x86_64), even though during refactor, an issue
was fixed and few optimizations were made:

* abs() of minimum (negative) small int value is calculated properly.
* objint_longlong and objint_mpz avoid allocating new object is the
  argument is already non-negative.
2017-09-18 00:06:43 +03:00
Damien George 280fb4d928 py/emitbc: Remove stray semicolon in outer scope. 2017-09-13 20:36:06 +10:00
Damien George 89f657f073 py/runtime.h: Change empty mp_warning macro so var-args are non empty.
Variable arguments in a macro should take at least 1 argument.
2017-09-13 20:33:55 +10:00
Damien George da8c4c2653 py/builtinhelp: Change signature of help text var from pointer to array.
As a pointer (const char *) it takes up an extra word of storage which is
in RAM.
2017-09-12 16:03:52 +10:00
ASM 52620c6b0e py/nlrx86: Fix building for Android/x86.
Tested using Clang on self-hosted Termux environment https://termux.com/.
2017-09-12 08:55:14 +03:00
Paul Sokolovsky eb84a830df py/runtime: Implement dispatch for "reverse op" special methods.
If, for class X, X.__add__(Y) doesn't exist (or returns NotImplemented),
try Y.__radd__(X) instead.

This patch could be simpler, but requires undoing operand swap and
operation switch to get non-confusing error message in case __radd__
doesn't exist.
2017-09-10 17:05:57 +03:00
Paul Sokolovsky 9355cca610 esp8266: Set DEFPSIZE=1024, MINCACHE=3 for "btree" module.
Defaults of 4096 and 5 respectively are too high to esp8266, causing
out of memory with a database beyond couple of pages.
2017-09-10 13:54:00 +03:00
Damien George e6fbee0981 py/builtinhelp: Simplify code slightly by extracting object type.
Reduces code size by about 10 bytes.
2017-09-10 15:15:41 +10:00
Paul Sokolovsky b8ee7ab5b9 py/runtime0.h: Put inplace arith ops in front of normal operations.
This is to allow to place reverse ops immediately after normal ops, so
they can be tested as one range (which is optimization for reverse ops
introduction in the next patch).
2017-09-08 00:10:10 +03:00
Paul Sokolovsky c460f6f15a py/runtime0.h: Regroup operations a bit.
Originally, there were grouped in blocks of 5, to make it easier e.g.
to assess and numeric code of each. But now it makes more sense to
group it by semantics/properties, and then split in chunks still,
which usually leads to chunks of ~6 ops.
2017-09-07 13:37:33 +03:00
Paul Sokolovsky 6d4cac088e py/objtype: Make sure mp_binary_op_method_name has full size again.
After recent refactorings to mp_binary_op_t, and make it future refactoring
proof for now, at the cost of extra element in the array.
2017-09-07 12:54:58 +03:00
Paul Sokolovsky 50b9329eba py/runtime0.h: Move MP_BINARY_OP_DIVMOD to the end of mp_binary_op_t.
It starts a dichotomy of mp_binary_op_t values which can't appear in the
bytecode. Another reason to move it is to VALUES of OP_* and OP_INPLACE_*
nicely adjacent. This also will be needed for OP_REVERSE_*, to be soon
introduced.
2017-09-07 11:26:42 +03:00
Paul Sokolovsky d4d1c45a55 py/runtime0.h: Move relational ops to the beginning of mp_binary_op_t.
This is to allow to encode arithmetic operations more efficiently, in
preparation to introduction of __rOP__ method support.
2017-09-07 10:55:43 +03:00
Paul Sokolovsky 5c603bd0fd py/objlist: Properly implement comparison with incompatible types.
Should raise TypeError, unless it's (in)equality comparison.
2017-09-07 00:10:10 +03:00
tll 68c28174d0 py/objstr: Add check for valid UTF-8 when making a str from bytes.
This patch adds a function utf8_check() to check for a valid UTF-8 encoded
string, and calls it when constructing a str from raw bytes.  The feature
is selectable at compile time via MICROPY_PY_BUILTINS_STR_UNICODE_CHECK and
is enabled if unicode is enabled.  It costs about 110 bytes on Thumb-2, 150
bytes on Xtensa and 170 bytes on x86-64.
2017-09-06 16:43:09 +10:00
Damien George 4a93801c12 all: Update Makefiles and others to build with new ports/ dir layout.
Also renames "stmhal" to "stm32" in documentation and everywhere else.
2017-09-06 14:09:13 +10:00
Paul Sokolovsky 1aaba5cabe py/objtuple: Properly implement comparison with incompatible types.
Should raise TypeError, unless it's (in)equality comparison.
2017-09-06 00:23:41 +03:00
Paul Sokolovsky 60749e57f2 py/objtype: Implement fallback for instance inplace special methods.
If __iop__ is not defined, call __op__ instead. This is desired behavior
for immutable types, __iop__ needs to be defined only for mutable types.
2017-09-04 16:44:21 +03:00
Damien George 77a48e8cd4 py/obj: Remove declaration for mp_obj_new_none(), it's never defined. 2017-09-04 23:35:46 +10:00
Damien George d4b75f6b68 py/obj: Fix comparison of float/complex NaN with itself.
IEEE floating point is specified such that a comparison of NaN with itself
returns false, and Python respects these semantics.  This patch makes uPy
also have these semantics.  The fix has a minor impact on the speed of the
object-equality fast-path, but that seems to be unavoidable and it's much
more important to have correct behaviour (especially in this case where
the wrong answer for nan==nan is silently returned).
2017-09-04 14:16:27 +10:00
Paul Sokolovsky 9950865c39 py/objfloat: Fix binary ops with incompatible objects.
These are now returned as "operation not supported" instead of raising
TypeError. In particular, this fixes equality for float vs incompatible
types, which now properly results in False instead of exception. This
also paves the road to support reverse operation (e.g. __radd__) with
float objects.

This is achieved by introducing mp_obj_get_float_maybe(), similar to
existing mp_obj_get_int_maybe().
2017-09-02 23:05:24 +03:00
Damien George dd376a239d py/nlrthumb: Get working again on standard Thumb arch (ie not Thumb2).
"b" on Thumb might not be long enough for the jump to nlr_push_tail so it
must be done indirectly.
2017-09-01 15:25:29 +10:00
Damien George 860eeeea9b py/qstrdefs: Remove unused qstrs.
They are not used by any component and take up valuable flash space.
2017-09-01 15:22:25 +10:00
Damien George 2daacc5cee py/modstruct: Check and prevent buffer-write overflow in struct packing.
Prior to this patch, the size of the buffer given to pack_into() was checked
for being too small by using the count of the arguments, not their actual
size.  For example, a format spec of '4I' would only check that there was 4
bytes available, not 16; and 'I' would check for 1 byte, not 4.

The pack() function is ok because its buffer is created to be exactly the
correct size.

The fix in this patch calculates the total size of the format spec at the
start of pack_into() and verifies that the buffer is large enough.  This
adds some computational overhead, to iterate through the whole format spec.
The alternative is to check during the packing, but that requires extra
code to handle alignment, and the check is anyway not needed for pack().
So to maintain minimal code size the check is done using struct_calcsize.
2017-09-01 11:11:09 +10:00
Damien George 79d5acbd01 py/modstruct: Check and prevent buffer-read overflow in struct unpacking
Prior to this patch, the size of the buffer given to unpack/unpack_from was
checked for being too small by using the count of the arguments, not their
actual size.  For example, a format spec of '4I' would only check that
there was 4 bytes available, not 16; and 'I' would check for 1 byte, not 4.

This bug is fixed in this patch by calculating the total size of the format
spec at the start of the unpacking function.  This function anyway needs to
calculate the number of items at the start, so calculating the total size
can be done at the same time.
2017-09-01 10:53:29 +10:00
Damien George 793d826d9d py/modstruct: In struct.pack, stop converting if there are no args left.
This patch makes a repeat counter behave the same as repeating the
typecode, when there are not enough args.  For example:
struct.pack('2I', 1) now behave the same as struct.pack('II', 1).
2017-09-01 10:10:51 +10:00
Damien George ca21aed0a1 py: Make m_malloc_fail() have void return type, since it doesn't return. 2017-08-31 17:00:14 +10:00
Damien George 6c9fca2aa9 py/map: Remove unused new/free functions.
Maps are always allocated "statically" and (de)initialised via mp_map_init
and mp_map_deinit.
2017-08-31 16:46:13 +10:00
Damien George 0e420d48ee py/map: Replace always-false condition with assertion. 2017-08-31 16:45:02 +10:00
Paul Sokolovsky 4556bd2acd py/objtype: mp_obj_class_lookup: Improve debug logging.
Now traces more explicitly thru the lookup process.
2017-08-31 00:44:51 +03:00
Paul Sokolovsky df6605eaba py/objtype: mp_obj_instance_make_new: Fix typos in comment. 2017-08-30 20:55:34 +03:00
Damien George 0102ee092b py: Change obsolete "///" comment formatting to normal comments.
This comment style is no longer used because the docs are written by hand,
not generated.
2017-08-30 21:02:00 +10:00
Paul Sokolovsky 784909ce16 py/objtype: Handle NotImplemented return from binary special methods.
NotImplemented means "try other fallbacks (like calling __rop__
instead of __op__) and if nothing works, raise TypeError". As
MicroPython doesn't implement any fallbacks, signal to raise
TypeError right away.
2017-08-30 01:39:24 +03:00
Damien George 58321dd985 all: Convert mp_uint_t to mp_unary_op_t/mp_binary_op_t where appropriate
The unary-op/binary-op enums are already defined, and there are no
arithmetic tricks used with these types, so it makes sense to use the
correct enum type for arguments that take these values.  It also reduces
code size quite a bit for nan-boxing builds.
2017-08-29 13:16:30 +10:00
Damien George be8e5744e6 py/nlrx86,x64: Replace #define of defined() with portable macro usage.
Using gcc -Wpedantic will warn that #define of defined() is non-portable
and this patch fixes this.
2017-08-29 12:52:18 +10:00
Paul Sokolovsky 37379a2974 py/objstr: startswith, endswith: Check arg to be a string.
Otherwise, it will silently get incorrect result on other values types,
including CPython tuple form like "foo.png".endswith(("png", "jpg"))
(which MicroPython doesn't support for unbloatedness).
2017-08-29 00:06:21 +03:00
Damien George f9ecaa132f py/asmthumb: Use existing macro to properly clear the D-cache.
This macro is provided by stmhal/mphalport.h and makes sure the addr and
size arguments are correctly aligned.
2017-08-23 11:32:27 +10:00
Damien George f5309fc4ff py/formatfloat: Don't post-increment variable that won't be used again. 2017-08-21 22:04:23 +10:00
Damien George 103ae43f95 py/objcomplex: Remove unnecessary assignment of variable. 2017-08-21 22:03:27 +10:00
Damien George ad6aae13a4 py/compile: Remove unused pn_colon code when compiling func params. 2017-08-21 22:00:34 +10:00
Damien George b16a755a0b py/mkrules.mk: Use "find -path" when searching for frozen obj files.
This allows the command to succeed without error even if there is no
$(BUILD)/build directory, which is the case for mpy-cross.
2017-08-21 20:32:30 +10:00
Paul Sokolovsky e3383e9352 py/stream: seek: Consistently handle negative offset for SEEK_SET.
Per POSIX, this is EINVAL, so raises OSError(EINVAL).
2017-08-20 22:02:41 +03:00
Paul Sokolovsky 0cd9ab7755 py/objstringio: Fix regression with handling SEEK_SET.
For SEEK_SET, offset should be treated as unsigned, to allow full-width
stream sizes (e.g. 32-bit instead of 31-bit). This is now fully documented
in stream.h. Also, seek symbolic constants are added.
2017-08-20 22:02:41 +03:00
Tom Collins 168350cd98 py/objstringio: Prevent offset wraparound for io.BytesIO objects.
Too big positive, or too big negative offset values could lead to overflow
and address space wraparound and thus access to unrelated areas of memory
(a security issue).
2017-08-20 22:00:05 +03:00
Damien George 025e5f2b33 py/binary: Change internal bytearray typecode from 0 to 1.
The value of 0 can't be used because otherwise mp_binary_get_size will let
a null byte through as the type code (intepreted as byterray).  This can
lead to invalid type-specifier strings being let through without an error
in the struct module, and even buffer overruns.
2017-08-17 16:19:35 +10:00
Stefan Naumann ace9fb5405 py: Add verbose debug compile-time flag MICROPY_DEBUG_VERBOSE.
It enables all the DEBUG_printf outputs in the py/ source code.
2017-08-15 11:53:36 +10:00
Bas van Sisseren a14ce77b28 py/binary.c: Fix bug when packing big-endian 'Q' values.
Without bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x10\x00\x00\x00\x00'

With bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x00\x00\x00\x00\x10'
2017-08-15 11:33:43 +10:00
Javier Candeira 35a1fea90b all: Raise exceptions via mp_raise_XXX
- Changed: ValueError, TypeError, NotImplementedError
  - OSError invocations unchanged, because the corresponding utility
    function takes ints, not strings like the long form invocation.
  - OverflowError, IndexError and RuntimeError etc. not changed for now
    until we decide whether to add new utility functions.
2017-08-13 22:52:33 +10:00
Paul Sokolovsky bfc2092dc5 py/modsys: Initial implementation of sys.getsizeof().
Implemented as a new MP_UNARY_OP. This patch adds support lists, dicts and
instances.
2017-08-11 09:43:07 +03:00
Damien George 7d4a2f773c all: Make use of $(TOP) variable in Makefiles, instead of "..".
$(TOP) is defined in py/mkenv.mk and should be used to refer to the top
level of this repository.
2017-08-11 12:22:19 +10:00
Damien George 3d25d9c7d9 py/objstr: Raise an exception for wrong type on RHS of str binary op.
The main case to catch is invalid types for the containment operator, of
the form str.__contains__(non-str).
2017-08-09 21:25:48 +10:00
Damien George eb2784e8a2 py/objtuple: Allow to use inplace-multiplication operator on tuples. 2017-08-09 21:20:42 +10:00
Paul Sokolovsky 4dc7c5649b py/mkrules.mk: Show frozen modules sizes together with executable size.
This works for Unix and similar ports so far.
2017-08-06 15:43:21 +03:00
Damien George 0f12082f5b py,extmod,stmhal: Use "static inline" for funcs that should be inline.
"STATIC inline" can expand to "inline" if STATIC is defined to nothing, and
this case can lead to link errors.
2017-08-02 13:42:34 +10:00
Alexander Steffen 55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Damien George b62bb53d0e py/modsys: Use MP_ROM_INT for int values in an mp_rom_map_elem_t. 2017-07-31 12:59:39 +10:00
Paul Sokolovsky 456450437f py/modio: BufferedWriter: Convert to mp_rom_map_elem_t. 2017-07-28 21:54:59 +03:00
Damien George 04552ff71b py: Implement raising a big-int to a negative power.
Before this patch raising a big-int to a negative power would just return
0.  Now it returns a floating-point number with the correct value.
2017-07-25 11:49:22 +10:00
Damien George 4d1fb6107f py/mpz: Make mpz_is_zero() an inline function.
It's more efficient as an inline function, and saves code size.
2017-07-25 11:32:04 +10:00
Damien George aa7be82a4d all: Don't include system errno.h when it's not needed. 2017-07-24 18:43:14 +10:00
Damien George a559098fec py/mperrno: Allow mperrno.h to be correctly included before other hdrs.
Before this patch the mperrno.h file could be included and would silently
succeed with incorrect config settings, because mpconfig.h was not yet
included.
2017-07-24 18:41:24 +10:00
Damien George 1ed3356540 py/py.mk: Make berkeley-db C-defs apply only to relevant source files.
Otherwise they can interfere (eg redefinition of "abort") with other source
files in a given uPy port.
2017-07-24 15:50:47 +10:00
Tom Collins bb3bddabb5 py/builtinevex: Add typechecking of globals/locals args to eval/exec. 2017-07-21 15:16:51 +10:00
Damien George 761e4c7ff6 all: Remove trailing spaces, per coding conventions. 2017-07-19 13:12:10 +10:00
Damien George cadbd7f3e6 py/modmicropython: Cast stack_limit value so it prints correctly.
Without this cast the print will give a wrong result on nan-boxing builds.
2017-07-18 22:30:22 +10:00
Damien George 3235b95f08 py/asmx64: Support moving a 64-bit immediate to one of top 8 registers.
If constants (eg mp_const_none_obj) are placed in very high memory
locations that require 64-bits for the pointer then the assembler must be
able to emit instructions to move such pointers to one of the top 8
registers (ie r8-r15).
2017-07-18 17:30:23 +10:00
Damien George 016325dd0a py/vm: Make n_state variable local to just set-up part of VM.
It's not used anywhere else in the VM loop, and clashes with (is shadowed
by) the n_state variable that's redeclared towards the end of the
mp_execute_bytecode function.  Code size is unchanged.
2017-07-18 16:17:23 +10:00
Alexander Steffen 299bc62586 all: Unify header guard usage.
The code conventions suggest using header guards, but do not define how
those should look like and instead point to existing files. However, not
all existing files follow the same scheme, sometimes omitting header guards
altogether, sometimes using non-standard names, making it easy to
accidentally pick a "wrong" example.

This commit ensures that all header files of the MicroPython project (that
were not simply copied from somewhere else) follow the same pattern, that
was already present in the majority of files, especially in the py folder.

The rules are as follows.

Naming convention:
* start with the words MICROPY_INCLUDED
* contain the full path to the file
* replace special characters with _

In addition, there are no empty lines before #ifndef, between #ifndef and
one empty line before #endif. #endif is followed by a comment containing
the name of the guard macro.

py/grammar.h cannot use header guards by design, since it has to be
included multiple times in a single C file. Several other files also do not
need header guards as they are only used internally and guaranteed to be
included only once:
* MICROPY_MPHALPORT_H
* mpconfigboard.h
* mpconfigport.h
* mpthreadport.h
* pin_defs_*.h
* qstrdefs*.h
2017-07-18 11:57:39 +10:00
Damien George 12d4fa9b37 py/gc: Refactor assertions in gc_free function.
gc_free() expects either NULL or a valid pointer into the heap, so the
checks for a valid pointer can be turned into assertions.
2017-07-12 12:17:38 +10:00
Damien George 0291a624cb py/compile: Fix enum variable declaration. 2017-07-09 13:18:14 +10:00
Damien George f69ab79ec8 py/objgenerator: Allow to hash generators and generator instances.
Adds nothing to the code size, since it uses existing empty slots in the
type structures.
2017-07-07 11:47:38 +10:00
Tom Collins 145796f037 py,extmod: Some casts and minor refactors to quiet compiler warnings. 2017-07-07 11:32:22 +10:00
Krzysztof Blazewicz 91a385db98 py/compile: Use switch-case to match token and operator.
Reduces code size.
2017-07-05 15:50:36 +10:00
Krzysztof Blazewicz a040fb89e7 py/compile: Combine arith and bit-shift ops into 1 compile routine.
This refactoring saves code space.
2017-07-05 15:49:00 +10:00
Damien George e66fd56852 py/repl: Change mp_uint_t to size_t in repl helpers. 2017-07-04 23:44:54 +10:00
Damien George 7bd10c1ffe py: Change mp_uint_t to size_t in builtins code. 2017-07-04 23:44:22 +10:00
Damien George 8b84b8ab8a py/objdict: Factorise dict accessor helper to reduce code size.
Code size change in bytes for this patch is:

   bare-arm:   -72
minimal x86:   -48
   unix x64:   -32
unix nanbox:  -120
     stmhal:   -68
     cc3200:   -64
    esp8266:   -56
2017-07-04 23:24:59 +10:00
Damien George b51919f5b7 py/makeversionhdr.py: Update to parse new release line in docs/conf.py.
The line in docs/conf.py with the release/version number was recently
changed and this patch makes the makeversionhdr.py script work again.
2017-07-04 22:37:41 +10:00
Damien George 6b8b56f859 py/modmath: Check for zero division in log with 2 args. 2017-07-04 02:15:11 +10:00
Damien George 9ed5e80eea py/vm: Make "if" control flow more obvious in YIELD_FROM opcode. 2017-07-04 02:14:25 +10:00
Damien George 9d2c72ad4f py/objstr: Remove unnecessary "sign" variable in formatting code. 2017-07-04 02:13:27 +10:00
Damien George 2138258fea py/runtime: Mark m_malloc_fail() as NORETURN. 2017-07-04 02:12:36 +10:00
Damien George 9aeba3e41b py/binary: Add missing "break" statements. 2017-07-04 02:11:46 +10:00
Damien George 65417c5ad9 py/objstr: Move uPy function wrappers to just after the C function.
This matches the coding/layout style of all the other objects.
2017-07-02 23:35:42 +10:00
Damien George adf22c19ae py/mpprint: Remove unreachable check for neg return of mp_format_float. 2017-06-30 12:10:50 +10:00
Damien George 265500c5c8 py/objnamedtuple: Simplify and remove use of alloca building namedtuple.
Prior to this patch there were 2 paths for creating the namedtuple, one for
when no keyword args were passed, and one when there were keyword args.
And alloca was used in the keyword-arg path to temporarily create the array
of elements for the namedtuple, which would then be copied to a
heap-allocated object (the namedtuple itself).

This patch simplifies the code by combining the no-keyword and keyword
paths, and removing the need for the alloca by constructing the namedtuple
on the heap before populating it.

Heap usage in unchanged, stack usage is reduced, use of alloca is removed,
and code size is not increased and is actually reduced by between 20-30
bytes for most ports.
2017-06-29 17:40:25 +10:00
Damien George 2f7fad66a2 py/builtinimport: Remove unreachable code for relative imports.
The while-loop that calls chop_component will guarantee that level==-1 at
the end of the loop.  Hence the code following it is unnecessary.

The check for p==this_name will catch imports that are beyond the
top-level, and also covers the case of new_mod_q==MP_QSTR_ (equivalent to
new_mod_l==0) so that check is removed.

There is also a new check at the start for level>=0 to guard against
__import__ being called with bad level values.
2017-06-28 12:21:29 +10:00
Alexander Steffen cd0987f5b7 py/frozenmod.h: Add missing header guards 2017-06-28 00:51:45 +03:00
Damien George c408ed9fb1 py/mpconfig.h: Remove spaces in "Micro Python" and remove blank line. 2017-06-26 12:29:20 +10:00
Damien George d94bc675e8 py/compile: Optimise emitter label indices to save a word of heap.
Previous to this patch, a label with value "0" was used to indicate an
invalid label, but that meant a wasted word (at slot 0) in the array of
label offsets.  This patch adjusts the label indices so the first one
starts at 0, and the maximum value indicates an invalid label.
2017-06-22 15:05:58 +10:00
Damien George 4c5f108321 py/compile: Fix bug with break/continue in else of optimised for-range.
This patch fixes a bug whereby the Python stack was not correctly reset if
there was a break/continue statement in the else black of an optimised
for-range loop.

For example, in the following code the "j" variable from the inner for loop
was not being popped off the Python stack:

    for i in range(4):
        for j in range(4):
            pass
        else:
            continue

This is now fixed with this patch.
2017-06-22 13:50:33 +10:00
Damien George e269cabe3e py/objint: In to_bytes(), allow length arg to be any int and check sign. 2017-06-15 14:21:02 +10:00
Damien George 8c5632a869 py/objint: Support "big" byte-order in int.to_bytes(). 2017-06-15 13:56:21 +10:00
Damien George 48d867b4a6 all: Make more use of mp_raise_{msg,TypeError,ValueError} helpers. 2017-06-15 11:54:41 +10:00
Damien George 1e70fda69f py/compile: Raise SyntaxError if positional args are given after */**.
In CPython 3.4 this raises a SyntaxError.  In CPython 3.5+ having a
positional after * is allowed but uPy has the wrong semantics and passes
the arguments in the incorrect order.  To prevent incorrect use of a
function going unnoticed it is important to raise the SyntaxError in uPy,
until the behaviour is fixed to follow CPython 3.5+.
2017-06-14 18:18:01 +10:00
Damien George e374cfff80 py/modthread: Raise RuntimeError in release() if lock is not acquired. 2017-06-14 14:43:50 +10:00
Damien George 6ed4581f54 py/formatfloat: Fix number of digits and exponent sign when rounding.
This patch fixes 2 things when printing a floating-point number that
requires rounding up of the mantissa:
- retain the correct precision; eg 0.99 becomes 1.0, not 1.00
- if the exponent goes from -1 to 0 then render it as +0, not -0
2017-06-13 13:36:56 +10:00
Paul Sokolovsky 07241cd37a py/objstringio: If created from immutable object, follow copy on write policy.
Don't create copy of immutable object's contents until .write() is called
on BytesIO.
2017-06-09 17:33:01 +03:00
Damien George b24ccfc639 py/makeqstrdefs.py: Make script run correctly with Python 2.6. 2017-06-09 13:42:13 +10:00
Damien George a8a5d1e8c8 py: Provide mp_decode_uint_skip() to help reduce stack usage.
Taking the address of a local variable leads to increased stack usage, so
the mp_decode_uint_skip() function is added to reduce the need for taking
addresses.  The changes in this patch reduce stack usage of a Python call
by 8 bytes on ARM Thumb, by 16 bytes on non-windowing Xtensa archs, and by
16 bytes on x86-64.  Code size is also slightly reduced on most archs by
around 32 bytes.
2017-06-09 13:36:33 +10:00
Damien George e1cda00387 py/modsys: Allow to compile with obj-repr D and PY_ATTRTUPLE disabled. 2017-06-08 00:41:27 +10:00
Damien George 326e8860ab py/objstr: Allow to compile with obj-repr D, and unicode disabled. 2017-06-08 00:40:38 +10:00
Damien George 72732fea1a py/persistentcode: Allow to compile with complex numbers disabled. 2017-06-08 00:28:28 +10:00
Paul Sokolovsky 5da8de2b66 extmod/modlwip: Fix error codes for duplicate calls to connect().
If socket is already connected, POSIX requires returning EISCONN. If
connection was requested, but not yet complete (for non-blocking
socket), error code is EALREADY.

http://pubs.opengroup.org/onlinepubs/7908799/xns/connect.html
2017-06-04 12:30:41 +03:00
Damien George 9f85c4fe48 py/objstr: Catch case of negative "maxsplit" arg to str.rsplit().
Negative values mean no limit on the number of splits so should delegate to
the .split() method.
2017-06-02 13:07:22 +10:00
Damien George bc76302eab py/modbuiltins: Add core-provided version of input() function.
The implementation is taken from stmhal/input.c, with code added to handle
ctrl-C.  This built-in is controlled by MICROPY_PY_BUILTINS_INPUT and is
disabled by default.  It uses readline() to capture input but this can be
overridden by defining the mp_hal_readline macro.
2017-06-01 16:02:49 +10:00
Ville Skyttä ca16c38210 various: Spelling fixes 2017-05-29 11:36:05 +03:00
Tom Collins e26fb3ad73 py/objstringio: Catch mp_uint_t overflow of stream position in write(). 2017-05-26 13:40:08 +10:00
Damien George 85f7b0b468 py/mkrules.mk: Fix auto-qstr generation when "make -B" is used.
For make v3.81, using "make -B" can set $? to empty and in this case the
auto-qstr generation needs to pass all args (ie $^) to cpp.  The previous
fix for this (which was removed in 23a693ec2d)
used if statements in the shell command, which gave very long lines that
didn't work on certain systems (eg cygwin).

The fix in this patch is to use an $if(...) expression, which will evaluate
to $? (only newer prerequisites) if it's non empty, otherwise it will use
$^ (all prerequisites).
2017-05-26 13:12:42 +10:00
Damien George 8f064e469d py/emitbc: Fix bug with BC emitter computing Python stack size.
Previous to this patch the mp_emit_bc_adjust_stack_size function would
adjust the current stack size but would not increase the maximum stack size
if the current size went above it.  This meant that certain Python code
(eg a try-finally block with no statements inside it) would not have enough
Python stack allocated to it.

This patch fixes the problem by always checking if the current stack size
goes above the maximum, and adjusting the latter if it does.
2017-05-25 20:42:30 +10:00
Damien George 04d05db27e py/vm: Fix bug with unwind jump popping the iterator from a for loop.
This patch fixes a regression introduced by
088740ecc4
2017-05-25 20:39:08 +10:00
Damien George 68e71eacb8 py/vm: Fix bug with stackless mode and unwinding of exceptions.
This patch fixes a regression introduced by
71a3d6ec3b

Previous to this patch the n_state variable was referring to that computed
at the very start of the mp_execute_bytecode function.  This patch fixes it
so that n_state is recomputed when the code_state changes.
2017-05-25 20:35:57 +10:00
Damien George 338f0849d9 py/mkenv.mk: Use $(TOP) instead of ".." to reference tools, mpy-cross. 2017-05-19 15:53:55 +10:00
Damien George 03659c51ca py/objrange: Fix slicing of range when step of slice is negative. 2017-05-18 17:32:42 +10:00
Damien George eb4c37f7a4 py/sequence: Fix boundary errors when slicing with a negative step. 2017-05-18 17:32:42 +10:00
Tom Collins 53461deb04 py/objstringio: Fix StringIO reads at or beyond EOF.
Existing code failed if seek() went past EOF (which is acceptable when writing).
2017-05-15 23:58:04 +03:00
Tom Collins f06d0839bd py/modsys: update conditionals for code referencing sys.stdout
Working on a build with PY_IO enabled (for PY_UJSON support) but PY_SYS_STDFILES disabled (no filesystem).  There are multiple references to mp_sys_stdout_obj that should only be enabled if both PY_IO and PY_SYS_STDFILES are enabled.
2017-05-14 18:24:50 +03:00
Tom Collins 6f56412ec3 py/lexer: Process CR earlier to allow newlines checks on chr1.
Resolves an issue where lexer failed to accept CR after line continuation
character.  It also simplifies the code.
2017-05-12 15:14:24 +10:00
Damien George a1f254028d py/mkrules.mk: Add dependency of .mpy files upon mpy-cross.
This ensures that mpy-cross is automatically built (and is up-to-date) for
ports that use frozen bytecode.  It also makes sure that .mpy files are
re-built if mpy-cross is changed.
2017-05-11 23:40:16 +10:00
Paul Sokolovsky edc02bd952 unix/main: Implement -m option for packages. 2017-05-09 14:22:21 +03:00
Tom Collins 2998647c4e py/lexer: Simplify lexer startup by using dummy bytes and next_char().
Now consistently uses the EOL processing ("\r" and "\r\n" convert to "\n")
and EOF processing (ensure "\n" before EOF) provided by next_char().

In particular the lexer can now correctly handle input that starts with CR.
2017-05-09 14:43:23 +10:00
Damien George 6cfa61a4cc py/binary: Handle storing big-ints to all arrays types.
Prior to this patch only 'q' and 'Q' type arrays could store big-int
values.  With this patch any big int that is stored to an array is handled
by the big-int implementation, regardless of the typecode of the array.
This allows arrays to work with all type sizes on all architectures.
2017-05-09 10:41:00 +10:00
Paul Sokolovsky 4a4490ffcc py/modio: resource_stream: Implement "package" param handling. 2017-05-06 18:42:35 +03:00
Damien George 58bb73e010 py/objint: In int.from_bytes, only create big-int if really needed.
This patch ensures that int.from_bytes only creates a big-int if necessary,
by checking the value for a small-int overflow as it's being parsed.
2017-05-06 10:29:09 +10:00
Paul Sokolovsky d7da2dba07 py/modio: Implement uio.resource_stream(package, resource_path).
The with semantics of this function is close to
pkg_resources.resource_stream() function from setuptools, which
is the canonical way to access non-source files belonging to a package
(resources), regardless of what medium the package uses (e.g. individual
source files vs zip archive). In the case of MicroPython, this function
allows to access resources which are frozen into the executable, besides
accessing resources in the file system.

This is initial stage of the implementation, which actually doesn't
implement "package" part of the semantics, just accesses frozen resources
from "root", or filesystem resource - from current dir.
2017-05-03 01:47:08 +03:00
stijn 2f0ce2a6f5 py: Cleanup use of global DEBUG preprocessor definition
The standard preprocessor definition to differentiate debug and non-debug
builds is NDEBUG, not DEBUG, so don't rely on the latter:
- just delete the use of it in objint_longlong.c as it has been stale code
  for years anyway (since commit [c4029e5]): SUFFIX isn't used anywhere.
- replace DEBUG with MICROPY_DEBUG_NLR in nlr.h: it is rarely used anymore
  so can be off by default
2017-04-30 14:28:37 +03:00
Damien George f85fd79c6c py/mpz: In mpn_sub, use existing function to remove trailing zeros. 2017-04-25 12:22:04 +10:00
Damien George c7aa86ce6f py/mpz: Strip trailing zeros from mpz value when set from bytes. 2017-04-25 12:06:10 +10:00
Damien George dd11af209d py: Add LOAD_SUPER_METHOD bytecode to allow heap-free super meth calls.
This patch allows the following code to run without allocating on the heap:

    super().foo(...)

Before this patch such a call would allocate a super object on the heap and
then load the foo method and call it right away.  The super object is only
needed to perform the lookup of the method and not needed after that.  This
patch makes an optimisation to allocate the super object on the C stack and
discard it right after use.

Changes in code size due to this patch are:

   bare-arm: +128
    minimal: +232
   unix x64: +416
unix nanbox: +364
     stmhal: +184
    esp8266: +340
     cc3200: +128
2017-04-22 23:39:20 +10:00
Damien George 5335942b59 py/compile: Refactor handling of special super() call.
This patch refactors the handling of the special super() call within the
compiler.  It removes the need for a global (to the compiler) state variable
which keeps track of whether the subject of an expression is super.  The
handling of super() is now done entirely within one function, which makes
the compiler a bit cleaner and allows to easily add more optimisations to
super calls.

Changes to the code size are:

   bare-arm: +12
    minimal:  +0
   unix x64: +48
unix nanbox: -16
     stmhal:  +4
     cc3200:  +0
    esp8266: -56
2017-04-22 21:46:32 +10:00
Damien George 0dd6a59c89 py/compile: Don't do unnecessary check if iter parse node is a struct.
If we get to this point in the code then pn_iter is guaranteed to be a
struct.
2017-04-22 21:43:42 +10:00
Damien George ae54fbf166 py/compile: Add COMP_RETURN_IF_EXPR option to enable return-if-else opt.
With this optimisation enabled the compiler optimises the if-else
expression within a return statement.  The optimisation reduces bytecode
size by 2 bytes for each use of such a return-if-else statement.  Since
such a statement is not often used, and costs bytes for the code, the
feature is disabled by default.

For example the following code:

    def f(x):
        return 1 if x else 2

compiles to this bytecode with the optimisation disabled (left column is
bytecode offset in bytes):

    00 LOAD_FAST 0
    01 POP_JUMP_IF_FALSE 8
    04 LOAD_CONST_SMALL_INT 1
    05 JUMP 9
    08 LOAD_CONST_SMALL_INT 2
    09 RETURN_VALUE

and to this bytecode with the optimisation enabled:

    00 LOAD_FAST 0
    01 POP_JUMP_IF_FALSE 6
    04 LOAD_CONST_SMALL_INT 1
    05 RETURN_VALUE
    06 LOAD_CONST_SMALL_INT 2
    07 RETURN_VALUE

So the JUMP to RETURN_VALUE is optimised and replaced by RETURN_VALUE,
saving 2 bytes and making the code a bit faster.
2017-04-22 14:58:01 +10:00
Damien George 40b40ffc98 py/compile: Extract parse-node kind at start of func for efficiency.
Otherwise the type of parse-node and its kind has to be re-extracted
multiple times.  This optimisation reduces code size by a bit (16 bytes on
bare-arm).
2017-04-22 14:23:47 +10:00
Damien George fa03bbf0fd py/compile: Don't do unnecessary check if parse node is a struct.
PN_atom_expr_normal parse nodes always have structs for their second
sub-node, so simplify the check for the sub-node kind to save code size.
2017-04-22 14:13:37 +10:00
Damien George 4df013c8cc py/objtype: mp_obj_new_super doesn't need to be public, so inline it.
Saves code size (20 bytes on bare-arm) and makes it a tiny bit more
efficient.
2017-04-22 12:14:04 +10:00
Paul Sokolovsky 9e8f316392 extmod/moductypes: Fix bigint handling for 32-bit ports. 2017-04-21 16:43:21 +03:00
Damien George 7a72c0db5a py: Reduce str/repr precision of float numbers when floats are 30-bit.
With 30-bit floats there aren't enough bits to faithfully print 7 decimal
digits, so reduce the precision to 6 digits.
2017-04-21 16:21:56 +10:00
Damien George bbb4b9822f py/modmicropython: Add micropython.kbd_intr() function.
It controls the character that's used to (asynchronously) raise a
KeyboardInterrupt exception.  Passing "-1" allows to disable the
interception of the interrupt character (as long as a port allows such a
behaviour).
2017-04-18 17:24:30 +10:00
Damien George c7e8c6f7de py/gc: Execute finaliser code in a protected environment.
If a finaliser raises an exception then it must not propagate through the
GC sweep function.  This patch protects against such a thing by running
finaliser code via the mp_call_function_1_protected call.

This patch also adds scheduler lock/unlock calls around the finaliser
execution to further protect against any possible reentrancy issues: the
memory manager is already locked when doing a collection, but we also don't
want to allow any scheduled code to run, KeyboardInterrupts to interupt the
code, nor threads to switch.
2017-04-12 13:52:04 +10:00
Damien George 08242eed26 py/nlrsetjmp: Add check for failed NLR jump.
Also optimise the function so it only needs to call the MP_STATE_THREAD
macro once (following how other nlr code is written).
2017-04-12 13:50:31 +10:00
Damien George a73501b1d6 py/objfloat: Add implementation of high-quality float hashing.
Disabled by default.
2017-04-12 13:38:17 +10:00
Damien George 816413e4b2 py: Optimise types for common case where type has a single parent type.
The common cases for inheritance are 0 or 1 parent types, for both built-in
types (eg built-in exceptions) as well as user defined types.  So it makes
sense to optimise the case of 1 parent type by storing just the type and
not a tuple of 1 value (that value being the single parent type).

This patch makes such an optimisation.  Even though there is a bit more
code to handle the two cases (either a single type or a tuple with 2 or
more values) it helps reduce overall code size because it eliminates the
need to create a static tuple to hold single parents (eg for the built-in
exceptions).  It also helps reduce RAM usage for user defined types that
only derive from a single parent.

Changes in code size (in bytes) due to this patch:

    bare-arm:       -16
    minimal (x86): -176
    unix (x86-64): -320
    unix nanbox:   -384
    stmhal:         -64
    cc3200:         -32
    esp8266:       -108
2017-04-12 13:22:21 +10:00
Damien George fc710169b7 py/obj: Clean up and add comments describing mp_obj_type_t struct. 2017-04-12 13:20:26 +10:00
Damien George e31fbd9b41 py/objint: Use unsigned arithmetic when formatting an integer.
Otherwise the edge case of the most negative integer value will not convert
correctly.
2017-04-11 15:18:35 +10:00
Damien George f66df1efc8 py/objint: Extract small int value directly because type is known. 2017-04-11 15:16:09 +10:00
Damien George 79ce664952 py/runtime: When init'ing kbd intr exc, use tuple ptr instead of object. 2017-04-10 17:07:26 +10:00
Damien George ee86de1f1a py: Make sure that static emg-exc-buffer is aligned to size of mp_obj_t.
This buffer is used to allocate objects temporarily, and such objects
require that their underlying memory be correctly aligned for their data
type.  Aligning for mp_obj_t should be sufficient for emergency exceptions,
but in general the memory buffer should aligned to the maximum alignment of
the machine (eg on a 32-bit machine with mp_obj_t being 4 bytes, a double
may not be correctly aligned).

This patch fixes a bug for certain nan-boxing builds, where mp_obj_t is 8
bytes and must be aligned to 8 bytes (even though the machine is 32 bit).
2017-04-10 16:02:56 +10:00
Damien George 81d2ca2b12 py/objtuple: Add support for inplace add (same as normal add). 2017-04-05 12:38:02 +10:00
Damien George de9b53695d py: Raise a ValueError if range() step is zero.
Following CPython.  Otherwise one gets either an infinite loop (if code is
optimised by the uPy compiler) or possibly a divide-by-zero CPU exception.
2017-04-05 10:50:26 +10:00
Damien George fc245d1ca4 py/objint: Consolidate mp_obj_new_int_from_float to one implementation.
This reduces code duplication and allows to make mp_classify_fp_as_int
static, which reduces code size.
2017-04-04 16:45:49 +10:00
Damien George 19f2e47d59 py: Add very simple but correct hashing for float and complex numbers.
Hashing of float and complex numbers that are exact (real) integers should
return the same integer hash value as hashing the corresponding integer
value.  Eg hash(1), hash(1.0) and hash(1+0j) should all be the same (this
is how Python is specified: if x==y then hash(x)==hash(y)).

This patch implements the simplest way of doing float/complex hashing by
just converting the value to int and returning that value.
2017-04-04 11:57:21 +10:00
Paul Sokolovsky 9a973977bb py/objstr: Use MICROPY_FULL_CHECKS for range checking when constructing bytes.
Split this setting from MICROPY_CPYTHON_COMPAT. The idea is to be able to
keep MICROPY_CPYTHON_COMPAT disabled, but still pass more of regression
testsuite. In particular, this fixes last failing test in basics/ for
Zephyr port.
2017-04-02 21:20:07 +03:00
Damien George a5500a8aad py/obj.h: Make sequence grow more efficient and support overlapping.
The first memmove now copies less bytes in some cases (because len_adj <=
slice_len), and the memcpy is replaced with memmove to support the
possibility that dest and slice regions are overlapping.
2017-04-02 17:28:24 +10:00
Damien George 4c307bfba1 all: Move BYTES_PER_WORD definition from ports to py/mpconfig.h
It can still be overwritten by a port in mpconfigport.h but for almost
all cases one can use the provided default.
2017-04-01 11:39:38 +11:00
Damien George b6c7e4b143 all: Use full path name when including mp-readline/timeutils/netutils.
This follows the pattern of how all other headers are now included, and
makes it explicit where the header file comes from.  This patch also
removes -I options from Makefile's that specify the mp-readline/timeutils/
netutils directories, which are no longer needed.
2017-03-31 22:29:39 +11:00
Damien George 4b753515c7 py/objzip: Convert mp_uint_t to size_t. 2017-03-30 22:58:08 +11:00
Paul Sokolovsky 29dd92c82a zephyr/Makefile: Rework to use modern, official build integration.
Build happens in 3 stages:

1. Zephyr config header and make vars are generated from prj.conf.
2. libmicropython is built using them.
3. Zephyr is built and final link happens.
2017-03-30 00:08:27 +03:00
Damien George 6b34107537 py: Change mp_uint_t to size_t for mp_obj_str_get_data len arg. 2017-03-29 12:56:45 +11:00
Damien George 6213ad7f46 py: Convert mp_uint_t to size_t for tuple/list accessors.
This patch changes mp_uint_t to size_t for the len argument of the
following public facing C functions:

mp_obj_tuple_get
mp_obj_list_get
mp_obj_get_array

These functions take a pointer to the len argument (to be filled in by the
function) and callers of these functions should update their code so the
type of len is changed to size_t.  For ports that don't use nan-boxing
there should be no change in generate code because the size of the type
remains the same (word sized), and in a lot of cases there won't even be a
compiler warning if the type remains as mp_uint_t.

The reason for this change is to standardise on the use of size_t for
variables that count memory (or memory related) sizes/lengths.  It helps
builds that use nan-boxing.
2017-03-29 12:56:17 +11:00
Damien George f9b0e644e5 py/compile: Provide terse error message for invalid dict/set literals. 2017-03-29 12:44:27 +11:00
Damien George 18c059febf py: Shorten a couple of error messages. 2017-03-29 12:36:46 +11:00
Damien George f55a059e7a py/compile: Simplify syntax-error messages for illegal assignments.
With this patch all illegal assignments are reported as "can't assign to
expression".  Before the patch there were special cases for a literal on
the LHS, and for augmented assignments (eg +=), but it seems a waste of
bytes (and there are lots of bytes used in error messages) to spend on
distinguishing such errors which a user will rarely encounter.
2017-03-29 12:28:33 +11:00
Damien George 5010d1958f py/lexer: Simplify and reduce code size for operator tokenising.
By removing the 'E' code from the operator token encoding mini-language the
tokenising can be simplified.  The 'E' code was only used for the !=
operator which is now handled as a special case; the optimisations for the
general case more than make up for the addition of this single, special
case.  Furthermore, the . and ... operators can be handled in the same way
as != which reduces the code size a little further.

This simplification also removes a "goto".

Changes in code size for this patch are (measured in bytes):

bare-arm:       -48
minimal x86:    -64
unix x86-64:   -112
unix nanbox:    -64
stmhal:         -48
cc3200:         -48
esp8266:        -76
2017-03-29 10:56:52 +11:00
Damien George 94c41bb06f py: Use mp_raise_TypeError/mp_raise_ValueError helpers where possible.
Saves 168 bytes on bare-arm.
2017-03-28 22:37:26 +11:00
Damien George 54507f78ee py/objmap: Convert mp_uint_t to size_t. 2017-03-27 12:19:24 +11:00
Damien George 40c1272e55 py/compile: When compiling super(), handle closed-over self variable.
The self variable may be closed-over in the function, and in that case the
call to super() should load the contents of the closure cell using
LOAD_DEREF (before this patch it would just load the cell directly).
2017-03-27 11:27:08 +11:00
Damien George a0973b09ce py/vm: Fix VM opcode tracing to print correct stack pointer.
Also const_table is now moved to the code_state->fun_bc structure.
2017-03-27 10:52:04 +11:00
Damien George 3f810daeb3 py/obj: Change mp_uint_t to size_t for mp_obj_get_array_fixed_n len arg. 2017-03-26 19:20:06 +11:00
Paul Sokolovsky 99866a00a0 py/nlrx86: Better check for Zephyr (requires 1.7). 2017-03-26 00:33:23 +03:00
Damien George ab5689bc9d py/objarray: Use mp_obj_str_get_str instead of mp_obj_str_get_data. 2017-03-25 19:53:31 +11:00
Damien George 64a4f11b2d py: Remove MP_STATE_CTX, use MP_STATE_THREAD instead (it's an alias).
MP_STATE_CTX was recently aliased to MP_STATE_THREAD and can now be
removed.
2017-03-24 18:43:28 +11:00
Damien George 707f16b05c py: Use mp_locals/mp_globals accessor funcs instead of MP_STATE_CTX.
To improve maintainability of the code.
2017-03-24 18:41:11 +11:00
Damien George f4ee1ba9b4 py/objnamedtuple: Use size_t where appropriate, instead of mp_uint_t. 2017-03-24 17:25:25 +11:00
Damien George d1b93ced78 py/objtype: Use size_t where appropriate, instead of mp_uint_t or uint. 2017-03-24 16:58:13 +11:00
Damien George c236ebfea7 py/modbuiltins: Allow round() to return a big int if necessary.
Previous to this patch, if the result of the round function overflowed a
small int, or was inf or nan, then a garbage value was returned.  With
this patch the correct big-int is returned if necessary and exceptions are
raised for inf or nan.
2017-03-24 11:00:45 +11:00
Damien George 125eae1ba3 py/modbuiltins: For round() builtin use nearbyint instead of round.
The C nearbyint function has exactly the semantics that Python's round()
requires, whereas C's round() requires extra steps to handle rounding of
numbers half way between integers.  So using nearbyint reduces code size
and potentially eliminates any source of errors in the handling of half-way
numbers.

Also, bare-metal implementations of nearbyint can be more efficient than
round, so further code size is saved (and efficiency improved).

nearbyint is provided in the C99 standard so it should be available on all
supported platforms.
2017-03-24 11:00:45 +11:00
Damien George c073519ec8 py/objint: Handle special case of -0 when classifying fp as int.
Otherwise -0.0 is classified as a bigint, which for builds without bigints
will lead unexpectedly to an overflow.
2017-03-23 23:51:35 +11:00
Damien George febeff4af4 py/modmath: Allow trunc/ceil/floor to return a big int if necessary.
Previous to this patch, if the result of the trunc/ceil/floor functions
overflowed a small int, or was inf or nan, then a garbage value was
returned.  With this patch the correct big-int is returned if necessary,
and exceptions are raised for inf or nan.
2017-03-23 22:57:08 +11:00
Damien George f64a3e296e py/lexer: Remove obsolete comment, since lexer can now raise exceptions. 2017-03-23 16:40:24 +11:00
Damien George 60656eaea4 py: Define and use MP_OBJ_ITER_BUF_NSLOTS to get size of stack iter buf.
It improves readability of code and reduces the chance to make a mistake.

This patch also fixes a bug with nan-boxing builds by rounding up the
calculation of the new NSLOTS variable, giving the correct number of slots
(being 4) even if mp_obj_t is larger than the native machine size.
2017-03-23 16:36:08 +11:00
Damien George 507119f4d8 py/sequence: Convert mp_uint_t to size_t where appropriate. 2017-03-23 16:23:20 +11:00
Damien George c88cfe165b py: Use size_t as len argument and return type of mp_get_index.
These values are used to compute memory addresses and so size_t is the
more appropriate type to use.
2017-03-23 16:17:40 +11:00
Damien George 58f23def55 py/bc: Provide better error message for an unexpected keyword argument.
Now, passing a keyword argument that is not expected will correctly report
that fact.  If normal or detailed error messages are enabled then the name
of the unexpected argument will be reported.

This patch decreases the code size of bare-arm and stmhal by 12 bytes, and
cc3200 by 8 bytes.  Other ports (minimal, unix, esp8266) remain the same in
code size.  For terse error message configuration this is because the new
message is shorter than the old one.  For normal (and detailed) error
message configuration this is because the new error message already exists
in py/objnamedtuple.c so there's no extra space in ROM needed for the
string.
2017-03-22 13:40:27 +11:00
Damien George 1a5c8d1053 py/vm: Don't release the GIL if the scheduler is locked.
The scheduler being locked general means we are running a scheduled
function, and switching to another thread violates that, so don't switch in
such a case (even though we technically could).

And if we are running a scheduled function then we want to finish it ASAP,
so we shouldn't switch to another thread.

Furthermore, ports with threading enabled will lock the scheduler during a
hard IRQ, and this patch to the VM will make sure that threads are not
switched during a hard IRQ (which would crash the VM).
2017-03-20 18:42:27 +11:00
Damien George 6e74d24f30 py: Add micropython.schedule() function and associated runtime code. 2017-03-20 15:20:26 +11:00
stijn bf29fe2e13 py/objstr: Use better msg in bad implicit str/bytes conversion exception
Instead of always reporting some object cannot be implicitly be converted
to a 'str', even when it is a 'bytes' object, adjust the logic so that
when trying to convert str to bytes it is shown like that.
This will still report bad implicit conversion from e.g. 'int to bytes'
as 'int to str' but it will not result in the confusing
'can't convert 'str' object to str implicitly' anymore for calls like
b'somestring'.count('a').
2017-03-20 15:11:45 +11:00
Damien George 5640e6dacd py: Provide mp_decode_uint_value to help optimise stack usage.
This has a noticeable improvement on x86-64 and Thumb2 archs, where stack
usage is reduced by 2 machine words in the VM.
2017-03-17 16:50:19 +11:00
Damien George 71a3d6ec3b py: Reduce size of mp_code_state_t structure.
Instead of caching data that is constant (code_info, const_table and
n_state), store just a pointer to the underlying function object from which
this data can be derived.

This helps reduce stack usage for the case when the mp_code_state_t
structure is stored on the stack, as well as heap usage when it's stored
on the heap.

The downside is that the VM becomes a little more complex because it now
needs to derive the data from the underlying function object.  But this
doesn't impact the performance by much (if at all) because most of the
decoding of data is done outside the main opcode loop.  Measurements using
pystone show that little to no performance is lost.

This patch also fixes a nasty bug whereby the bytecode can be reclaimed by
the GC during execution.  With this patch there is always a pointer to the
function object held by the VM during execution, since it's stored in the
mp_code_state_t structure.
2017-03-17 16:39:13 +11:00
Damien George d279bcff8a py/objstr: Fix eager optimisation of str/bytes addition.
The RHS can only be returned if it is the same type as the LHS.
2017-03-16 14:30:04 +11:00
Damien George 23a693ec2d py/mkrules.mk: Remove special check for "-B" in qstr auto generation.
When make is passed "-B" it seems that everything is considered out-of-date
and so $? expands to all prerequisites.  Thus there is no need for a
special check to see if $? is emtpy.
2017-03-15 21:50:48 +11:00
Damien George d65371538d py/mpprint: Fix int formatting so "+" is printed for 0-valued integer. 2017-03-15 17:25:12 +11:00
Damien George 8cd4911e63 py/emitnative: Remove obsolete commented out code. 2017-03-15 10:03:22 +11:00
Damien George a5a84e1f85 py/emitnative: Use assertions and mp_not_implemented correctly.
Assertions are used to check expressions that should always be true,
and mp_not_implemented is used for code that can be reached.
2017-03-14 23:05:40 +11:00
Damien George d1ae6ae080 py/objint: Allow to print long-long ints without using the heap.
Some stack is allocated to format ints, and when the int implementation uses
long-long there should be additional stack allocated compared with the other
cases.  This patch uses the existing "fmt_int_t" type to determine the
amount of stack to allocate.
2017-03-14 14:54:20 +11:00
Damien George 1831034be1 py: Allow lexer to raise exceptions during construction.
This patch refactors the error handling in the lexer, to simplify it (ie
reduce code size).

A long time ago, when the lexer/parser/compiler were first written, the
lexer and parser were designed so they didn't use exceptions (ie nlr) to
report errors but rather returned an error code.  Over time that has
gradually changed, the parser in particular has more and more ways of
raising exceptions.  Also, the lexer never really handled all errors without
raising, eg there were some memory errors which could raise an exception
(and in these rare cases one would get a fatal nlr-not-handled fault).

This patch accepts the fact that the lexer can raise exceptions in some
cases and allows it to raise exceptions to handle all its errors, which are
for the most part just out-of-memory errors during construction of the
lexer.  This makes the lexer a bit simpler, and also the persistent code
stuff is simplified.

What this means for users of the lexer is that calls to it must be wrapped
in a nlr handler.  But all uses of the lexer already have such an nlr
handler for the parser (and compiler) so that doesn't put any extra burden
on the callers.
2017-03-14 11:52:05 +11:00
Paul Sokolovsky 776883cb80 py/objint_longlong: Implement mp_obj_int_from_bytes_impl().
This makes int.from_bytes() work for MICROPY_LONGINT_IMPL_LONGLONG.
2017-03-10 00:22:53 +01:00
Damien George 52b6764894 py/nlrx64: Fixes to support Mac OS.
Two independent fixes:
- need to prefix symbols referenced from asm with underscore;
- need to undo the C-function prelude.
2017-03-08 22:36:02 +11:00
Paul Sokolovsky fd49ff9917 py/nlrx86: Add workaround for Zephyr.
Actually, this removes -fno-omit-frame-pointer workaround for Zephyr.
2017-03-07 16:48:09 +01:00
Krzysztof Blazewicz 7e480e8a30 py: Use mp_obj_get_array where sequence may be a tuple or a list. 2017-03-07 16:48:16 +11:00
Krzysztof Blazewicz 1215dc47e2 py/runtime.c: Remove optimization of '*a,=b', it caused a bug.
*a, = b should always make a copy of b, instead, before this patch
if b was a list it would copy only a reference to it.
2017-03-07 16:48:16 +11:00
Paul Sokolovsky 96aa3a3102 py/modsys: Use MP_SMALL_INT_MAX for sys.maxsize in case of LONGINT_IMPL_NONE.
INT_MAX used previosly is indeed max value for int, whereas on LP64
platforms, long is used for mp_int_t. Using MP_SMALL_INT_MAX is the
correct way to do it anyway.
2017-03-06 12:15:25 +01:00
Damien George 914648ce0e py/py.mk: Force nlr files to be compiled with -Os. 2017-03-06 17:13:43 +11:00
Damien George f0dddb688d py/nlrx86: Convert from assembler to C file with inline asm. 2017-03-06 17:13:43 +11:00
Damien George 321848470c py/nlrx64: Convert from assembler to C file with inline asm. 2017-03-06 17:13:35 +11:00
Damien George a85755aa22 py/nlrxtensa: Convert from assembler to C file with inline asm.
nlr_jump is a little bit inefficient because it now saves a register to
the stack.
2017-03-06 17:13:16 +11:00
Damien George be3d7f91e5 py/nlr.h: Mark nlr_jump_fail as NORETURN. 2017-03-06 17:13:16 +11:00
Damien George 05fe66f68a py: Move locals/globals dicts to the thread-specific state.
Each threads needs to have its own private references to its current
locals/globals dicts, otherwise functions running within different
contexts (eg imported from different files) can behave very strangely.
2017-03-06 17:01:56 +11:00
Damien George 9275c18270 py/map: Fix bugs with deletion of elements from OrderedDict.
There were 2 bugs, now fixed by this patch:
- after deleting an element the len of the dict did not decrease by 1
- after deleting an element searching through the dict could lead to
  a seg fault due to there being an MP_OBJ_SENTINEL in the ordered array
2017-03-03 11:21:19 +11:00
Damien George f4a12dca58 py/objarray: Disallow slice-assignment to read-only memoryview.
Also comes with a test for this.  Fixes issue #2904.
2017-02-27 16:09:57 +11:00
Paul Sokolovsky 4b3da60324 py/runtime: mp_raise_msg(): Accept NULL argument for message.
In this case, raise an exception without a message.

This would allow to shove few code bytes comparing to currently used
mp_raise_msg(..., "") pattern. (Actual savings depend on function code
alignment used by a particular platform.)
2017-02-24 09:57:25 -05:00
Damien George f615d82d5b py/parse: Simplify handling of errors by raising them directly.
The parser was originally written to work without raising any exceptions
and instead return an error value to the caller.  But it's now required
that a call to the parser be wrapped in an nlr handler, so we may as well
make use of that fact and simplify the parser so that it doesn't need to
keep track of any memory errors that it had.  The parser anyway explicitly
raises an exception at the end if there was an error.

This patch simplifies the parser by letting the underlying memory
allocation functions raise an exception if they fail to allocate any
memory.  And if there is an error parsing the "<id> = const(<val>)" pattern
then that also raises an exception right away instead of trying to recover
gracefully and then raise.
2017-02-24 14:56:37 +11:00
Damien George 5255255fb9 py: Create str/bytes objects in the parser, not the compiler.
Previous to this patch any non-interned str/bytes objects would create a
special parse node that held a copy of the str/bytes data.  Then in the
compiler this data would be turned into a str/bytes object.  This actually
lead to 2 copies of the data, one in the parse node and one in the object.
The parse node's copy of the data would be freed at the end of the compile
stage but nevertheless it meant that the peak memory usage of the
parse/compile stage was higher than it needed to be (by an amount equal to
the number of bytes in all the non-interned str/bytes objects).

This patch changes the behaviour so that str/bytes objects are created
directly in the parser and the object stored in a const-object parse node
(which already exists for bignum, float and complex const objects).  This
reduces peak RAM usage of the parse/compile stage, simplifies the parser
and compiler, and reduces code size by about 170 bytes on Thumb2 archs,
and by about 300 bytes on Xtensa archs.
2017-02-24 13:43:43 +11:00
Damien George 74f4d2c659 py/parse: Allow parser/compiler consts to be bignums.
This patch allows uPy consts to be bignums, eg:

    X = const(1 << 100)

The infrastructure for consts to be a bignum (rather than restricted to
small integers) has been in place for a while, ever since constant folding
was upgraded to allow bignums.  It just required a small change (in this
patch) to enable it.
2017-02-24 13:03:44 +11:00
Damien George b1b090255c py/moduerrno: Make list of errno codes configurable.
It's configurable by defining MICROPY_PY_UERRNO_LIST.  If this is not
defined then a default is provided.
2017-02-22 12:58:11 +11:00
Damien George f563406d2e py/moduerrno: Make uerrno.errorcode dict configurable.
It's configured by MICROPY_PY_UERRNO_ERRORCODE and enabled by default
(since that's the behaviour before this patch).

Without this dict the lookup of errno codes to strings must use the
uerrno module itself.
2017-02-22 12:58:11 +11:00
Damien George 89267886cc py/objlist: For list slice assignment, allow RHS to be a tuple or list.
Before this patch, assigning anything other than a list would lead to a
crash.  Fixes issue #2886.
2017-02-20 15:09:59 +11:00
Damien George bdebfaa4bf py/grammar: Remove unused rule.
Since the recent changes to string/bytes literal concatenation, this rule
is no longer used.
2017-02-17 12:48:45 +11:00
Damien George 5124a94067 py/lexer: Convert mp_uint_t to size_t where appropriate. 2017-02-17 12:44:24 +11:00
Damien George 534b7c368d py: Do adjacent str/bytes literal concatenation in lexer, not compiler.
It's much more efficient in RAM and code size to do implicit literal string
concatenation in the lexer, as opposed to the compiler.

RAM usage is reduced because the concatenation can be done right away in the
tokeniser by just accumulating the string/bytes literals into the lexer's
vstr.  Prior to this patch adjacent strings/bytes would create a parse tree
(one node per string/bytes) and then in the compiler a whole new chunk of
memory was allocated to store the concatenated string, which used more than
double the memory compared to just accumulating in the lexer.

This patch also significantly reduces code size:

bare-arm: -204
minimal:  -204
unix x64: -328
stmhal:   -208
esp8266:  -284
cc3200:   -224
2017-02-17 12:12:40 +11:00
Damien George 773278ec30 py/lexer: Simplify handling of line-continuation error.
Previous to this patch there was an explicit check for errors with line
continuation (where backslash was not immediately followed by a newline).

But this check is not necessary: if there is an error then the remaining
logic of the tokeniser will reject the backslash and correctly produce a
syntax error.
2017-02-17 11:30:14 +11:00
Damien George ae43679792 py/lexer: Use strcmp to make keyword searching more efficient.
Since the table of keywords is sorted, we can use strcmp to do the search
and stop part way through the search if the comparison is less-than.

Because all tokens that are names are subject to this search, this
optimisation will improve the overall speed of the lexer when processing
a script.

The change also decreases code size by a little bit because we now use
strcmp instead of the custom str_strn_equal function.
2017-02-17 11:10:35 +11:00
Damien George a68c754688 py/lexer: Move check for keyword to name-tokenising block.
Keywords only needs to be searched for if the token is a MP_TOKEN_NAME, so
we can move the seach to the part of the code that does the tokenising for
MP_TOKEN_NAME.
2017-02-17 10:59:57 +11:00
Damien George 98b3072da5 py/lexer: Simplify handling of indenting of very first token. 2017-02-17 10:56:06 +11:00
Damien George 6a11048af1 py/persistentcode: Bump .mpy version due to change in bytecode. 2017-02-17 00:19:34 +11:00
Damien George c264414746 py/lexer: Don't generate string representation for period or ellipsis.
It's not needed.
2017-02-16 20:23:41 +11:00
Damien George 71019ae4f5 py/grammar: Group no-compile grammar rules together to shrink tables.
Grammar rules have 2 variants: ones that are attached to a specific
compile function which is called to compile that grammar node, and ones
that don't have a compile function and are instead just inspected to see
what form they take.

In the compiler there is a table of all grammar rules, with each entry
having a pointer to the associated compile function.  Those rules with no
compile function have a null pointer.  There are 120 such rules, so that's
120 words of essentially wasted code space.

By grouping together the compile vs no-compile rules we can put all the
no-compile rules at the end of the list of rules, and then we don't need
to store the null pointers.  We just have a truncated table and it's
guaranteed that when indexing this table we only index the first half,
the half with populated pointers.

This patch implements such a grouping by having a specific macro for the
compile vs no-compile grammar rules (DEF_RULE vs DEF_RULE_NC).  It saves
around 460 bytes of code on 32-bit archs.
2017-02-16 19:45:06 +11:00
Damien George e6003f466e py: De-optimise some uses of mp_getiter, so they don't use the C stack.
In these cases the heap is anyway used to create a new object so no real
need to use the C stack for iterating.  It saves a few bytes of code size.
2017-02-16 19:11:34 +11:00
Damien George 4d2bab1444 py/compile: Optimise list/dict/set comprehensions to use stack iter. 2017-02-16 18:38:07 +11:00
Damien George cb6300697c py/runtime: Optimise case of identity iterator so it doesn't alloc RAM. 2017-02-16 18:38:06 +11:00
Damien George 30b42dd72d py: Remove unused "use_stack" argument from for_iter_end emit function. 2017-02-16 18:38:06 +11:00
Damien George 088740ecc4 py: Optimise storage of iterator so it takes only 4 slots on Py stack. 2017-02-16 18:38:06 +11:00
Damien George 6e769da0da py: Make FOR_ITER opcode pop 1+4 slots from the stack when finished.
The extra 4 slots correspond to the iterator object stored on the stack.
2017-02-16 18:38:06 +11:00
Damien George f4df3aaa72 py: Allow bytecode/native to put iter_buf on stack for simple for loops.
So that the "for x in it: ..." statement can now work without using the
heap (so long as the iterator argument fits in an iter_buf structure).
2017-02-16 18:38:06 +11:00
Damien George ae8d867586 py: Add iter_buf to getiter type method.
Allows to iterate over the following without allocating on the heap:
- tuple
- list
- string, bytes
- bytearray, array
- dict (not dict.keys, dict.values, dict.items)
- set, frozenset

Allows to call the following without heap memory:
- all, any, min, max, sum

TODO: still need to allocate stack memory in bytecode for iter_buf.
2017-02-16 18:38:06 +11:00
Damien George 101886f529 py/vm: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George da36f5232d py/objint: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George fa5a591757 py/objexcept: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George efa629028a py/objclosure: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:17 +11:00
Damien George dbcdb9f8d8 py/objfun: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George ccc5254224 py/objarray: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George c0d9500eee py/objstr: Convert mp_uint_t to size_t (and use int) where appropriate. 2017-02-16 16:51:16 +11:00
Damien George 68cd3a93f0 py/objset: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George 1ea2f7a8ce py/objdict: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George 58d9eeb8d9 py/objlist: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George 229823942c py/objtuple: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George 891dc5c62c py/persistentcode: Replace mp_uint_t with size_t where appropriate. 2017-02-16 16:51:16 +11:00
Damien George 6ed77bedbd py/mpz: Change type of "base" args from mp_uint_t to unsigned int. 2017-02-16 16:51:16 +11:00
Damien George eb90edb5c0 py/mpz: Remove obsolete declaration of mpz_as_str_size. 2017-02-16 16:51:16 +11:00
Damien George dcdcc43dad py/mpz: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:13 +11:00
Damien George 4e3bac2e42 py/runtime: Convert mp_uint_t to size_t where appropriate. 2017-02-16 16:51:13 +11:00
Damien George f6c22a0679 py/vm: Add MICROPY_PY_THREAD_GIL_VM_DIVISOR option.
This improves efficiency of GIL release within the VM, by only doing the
release after a fixed number of jump-opcodes have executed in the current
thread.
2017-02-15 11:28:15 +11:00
Damien George 234f07f16c py/modthread: Use system-provided mutexs for _thread locks.
It's more efficient using the system mutexs instead of synthetic ones with
a busy-wait loop.  The system can do proper scheduling and blocking of the
threads waiting on the mutex.
2017-02-15 11:28:02 +11:00
Damien George adc80b8f84 py/objtype: Replace non-ASCII single-quote char with ASCII version. 2017-02-14 20:55:31 +11:00
Damien George cc2dbdd1fe py/emitbc: Produce correct line number info for large bytecode chunks.
Previous to this patch, for large chunks of bytecode that originated from
a single source-code line, the bytecode-line mapping would generate
something like (for 42 bytecode bytes and 1 line):

  BC_SKIP=31  LINE_SKIP=1
  BC_SKIP=11  LINE_SKIP=0

This would mean that any errors in the last 11 bytecode bytes would be
reported on the following line.  This patch fixes it to generate instead:

  BC_SKIP=31  LINE_SKIP=0
  BC_SKIP=11  LINE_SKIP=1
2017-02-10 11:58:10 +11:00
dmazzella 18e6569166 py/objtype: Implement __delattr__ and __setattr__.
This patch implements support for class methods __delattr__ and __setattr__
for customising attribute access.  It is controlled by the config option
MICROPY_PY_DELATTR_SETATTR and is disabled by default.
2017-02-09 12:40:15 +11:00
Dave Hylands aa34c553ec py/nlr: Fix execstack builds for ARM.
It seems that the gcc toolchain on the RaspberryPi
likes %progbits instead of @progbits. I verified that
%progbits also works under x86, so this should
fix #2848 and fix #2842

I verified that unix and mpy-cross both compile
on my RaspberryPi and on my x64 machine.
2017-02-08 11:12:26 +11:00
Damien George af622eb2a6 py/map: Change mp_uint_t to size_t where appropriate.
The internal map/set functions now use size_t exclusively for computing
addresses.  size_t is enough to reach all of available memory when
computing addresses so is the right type to use.  In particular, for
nanbox builds it saves quite a bit of code size and RAM compared to the
original use of mp_uint_t (which is 64-bits on nanbox builds).
2017-02-08 11:00:15 +11:00
Damien George a25aa2bcc3 py/asmxtensa.h: Explicitly cast args to 32-bits so left-shift is legal.
For archs that have 16-bit pointers, the asmxtensa.h file can give compiler
warnings about left-shift being greater than the width of the type (due to
the inline functions in this header file).  Explicitly casting the
constants to uint32_t stops these warnings.
2017-02-08 10:48:51 +11:00
Damien George bd04ed3e8a py/objcomplex: Fix typo in ternary expression.
This typo actually did the correct thing, but it was very obscure (came
about from think in terms of Python's "x if cond else y" expression).
2017-02-04 00:23:56 +11:00
Damien George 90ab191b65 py/objstr: Convert some instances of mp_uint_t to size_t. 2017-02-03 13:04:56 +11:00
Damien George a19b5a01ce py/mpconfig.h: Move PY_BUILTINS_POW3 config option to diff part of file.
With so many config options it's good to (at least try to) keep them
grouped into logical sections.
2017-02-03 12:35:48 +11:00
Damien George 7317e34383 py/objstr: Give correct behaviour when passing a dict to %-formatting.
This patch fixes two main things:
- dicts can be printed directly using '%s' % dict
- %-formatting should not crash when passed a non-dict to, eg, '%(foo)s'
2017-02-03 12:13:44 +11:00
Nicko van Someren df0117c8ae py: Added optimised support for 3-argument calls to builtin.pow()
Updated modbuiltin.c to add conditional support for 3-arg calls to
pow() using MICROPY_PY_BUILTINS_POW3 config parameter. Added support in
objint_mpz.c for for optimised implementation.
2017-02-02 22:23:10 +03:00
Damien George 2486c4ff46 py/objset: Fix inplace binary ops so frozensets are not modified. 2017-02-03 00:27:56 +11:00
Damien George 3ed0e5e5d4 py/objcomplex: Correctly handle case of 0j to power of something.
0j to the power of negative now raises ZeroDivisionError, and 0j to the
power of positive returns 0.
2017-02-03 00:01:45 +11:00
Damien George 4b8ec5256d py/objfloat: Raise ZeroDivisionError for 0 to negative power. 2017-02-03 00:01:37 +11:00
Damien George 30cf503e97 py/objset: Make inplace binary operators actually modify the set. 2017-02-02 23:33:49 +11:00
Paul Sokolovsky 50d3a9df67 py/objstringio: Allow to specify initial capacity by passing numeric argument.
E.g. uio.BytesIO(100) will allocate buffer with 100 bytes of space.
2017-02-02 00:33:43 +03:00
Dave Hylands aee74a1dae unix: Make stack be non-executable
This PR is to address issue #2812.
2017-02-01 23:03:10 +03:00
Damien George 1808b2e8d5 extmod: Remove MICROPY_FSUSERMOUNT and related files.
Replaced by MICROPY_VFS and the VFS sub-system.
2017-01-30 12:26:07 +11:00
Damien George 8beba7310f extmod/vfs_fat: Remove MICROPY_READER_FATFS component. 2017-01-30 12:26:07 +11:00
Paul Sokolovsky 7a7516d40d extmod/machine_signal: Implement "signal" abstraction for machine module.
A signal is like a pin, but ca also be inverted (active low). As such, it
abstracts properties of various physical devices, like LEDs, buttons,
relays, buzzers, etc. To instantiate a Signal:

pin = machine.Pin(...)
signal = machine.Signal(pin, inverted=True)

signal has the same .value() and __call__() methods as a pin.
2017-01-29 18:57:36 +03:00
Damien George 3f6b4e08e3 extmod/vfs: Expose mp_vfs_mount_t type.
It should only be used for low-level things and with caution, for example
putting mounted VFS data in ROM or the static data section.
2017-01-27 23:22:15 +11:00
Damien George dcb9ea7215 extmod: Add generic VFS sub-system.
This provides mp_vfs_XXX functions (eg mount, open, listdir) which are
agnostic to the underlying filesystem type, and just require an object with
the relevant filesystem-like methods (eg .mount, .open, .listidr) which can
then be mounted.

These mp_vfs_XXX functions would typically be used by a port to implement
the "uos" module, and mp_vfs_open would be the builtin open function.

This feature is controlled by MICROPY_VFS, disabled by default.
2017-01-27 17:19:06 +11:00
Damien George d4464b0050 py/py.mk: Add CFLAGS_MOD flag to set config file for FatFs. 2017-01-27 13:19:10 +11:00
Damien George cc4c1adf6e py/showbc: Make sure to set the const_table before printing bytecode. 2017-01-27 12:34:09 +11:00
Paul Sokolovsky e2e663291d py/objstr: Optimize string concatenation with empty string.
In this, don't allocate copy, just return non-empty string. This helps
with a standard pattern of buffering data in case of short reads:

    buf = b""
    while ...:
        s = f.read(...)
        buf += s
        ...

For a typical case when single read returns all data needed, there won't
be extra allocation. This optimization helps uasyncio.
2017-01-27 00:49:39 +03:00
Damien George e9cb1f8077 py/objmodule: Move module init/deinit code into runtime functions.
They are one-line functions and having them inline in mp_init/mp_deinit
eliminates the overhead of a function call, and matches how other state
is initialised in mp_init.
2017-01-26 23:30:38 +11:00
Damien George eaa77455c3 py/objint: Fix left-shift overflow in checking for large int. 2017-01-25 14:39:13 +11:00
Damien George f5172af1c4 py/builtinhelp: Implement help('modules') to list available modules.
This is how CPython does it, and it's very useful to help users discover
the available modules for a given port, especially built-in and frozen
modules.  The function does not list modules that are in the filesystem
because this would require a fair bit of work to do correctly, and is very
port specific (depending on the filesystem).
2017-01-22 12:12:54 +11:00
Damien George 9de91914fb py: Move weak-link map to objmodule.c, and expose module maps as public. 2017-01-22 11:59:29 +11:00
Damien George 9f04dfb568 py: Add builtin help function to core, with default help msg.
This builtin is configured using MICROPY_PY_BUILTINS_HELP, and is disabled
by default.
2017-01-22 11:56:16 +11:00
Paul Sokolovsky e6ab43e2c0 py/objint_longlong: Add stub for mp_obj_int_from_bytes_impl().
To be implemented later.
2017-01-21 20:15:05 +03:00
Paul Sokolovsky bec7bfb29d py/objint: from_bytes(): Implement "byteorder" param and arbitrary precision.
If result guaranteedly fits in a small int, it is handled in objint.c.
Otherwise, it is delegated to mp_obj_int_from_bytes_impl(), which should
be implemented by individual objint_*.c, similar to
mp_obj_int_to_bytes_impl().
2017-01-21 20:14:18 +03:00
Paul Sokolovsky 1b42f5251f py/mpz: Implement mpz_set_from_bytes() as a foundation for int.from_bytes(). 2017-01-21 20:07:50 +03:00
Damien George e873243aa3 py/objint_mpz: Refactor switch-statement to remove unreachable default. 2017-01-19 23:35:45 +11:00
Damien George 94a587a750 py/formatfloat: Remove unreachable code.
The if-block that this unreachable code is in has a condition "f>=5" so
"fp_isless1(f)" will always fail.
2017-01-19 23:32:16 +11:00
Paul Sokolovsky af90461931 py/binary: mp_binary_get_size: Raise error on unsupported typecodes.
Previouly, we had errors checked in callers, which led to duplicate code
or missing checks in some places.
2017-01-17 22:53:06 +03:00
Damien George d7150b09d7 py/runtime: Refactor default case of switch to remove assert(0). 2017-01-17 17:03:56 +11:00
Damien George 5bea080737 py/objexcept: Replace if-cond and assert(0) with simple assert. 2017-01-17 17:03:16 +11:00
Damien George f51f22dd42 py/emitnative: Remove assert(0)'s or replace with mp_not_implemented. 2017-01-17 17:02:21 +11:00
Damien George 86e942309a py/parse: Refactor code to remove assert(0)'s.
This helps to improve code coverage.  Note that most of the changes in
this patch are just de-denting the cases of the switch statements.
2017-01-17 17:00:55 +11:00
Damien George 239f920299 py/objgenerator: Don't raise RuntimeError if GeneratorExit ignored.
In this case it's allowed to be ignored.
2017-01-17 00:16:56 +11:00
Damien George 681994638b py/objgenerator: When throwing an object, don't make an exc instance.
Arguments to throw() for generators don't need to be exceptions.
2017-01-17 00:14:14 +11:00
Damien George aeb2655073 py/runtime: Fix handling of throw() when resuming generator.
If GeneratorExit is injected as a throw-value then that should lead to
the close() method being called, if it exists.  If close() does not exist
then throw() should not be called, and this patch fixes this.
2017-01-17 00:10:49 +11:00
Damien George 40863fce6f py/runtime: Refactor assert(0) to improve coverage. 2017-01-17 00:09:56 +11:00
Damien George d23834bc96 py/builtinimport: Remove unreachable code and change obj-import comment. 2017-01-16 16:41:43 +11:00
Damien George 63e291de70 py/builtinimport: Raise ValueError for bad relative import, per CPython. 2017-01-16 16:21:04 +11:00
Damien George b528e9a428 py/builtinimport: Fix bug when importing names from frozen packages.
The commit d9047d3c8a introduced a bug
whereby "from a.b import c" stopped working for frozen packages.  This is
because the path was not properly truncated and became "a//b".  Such a
path resolves correctly for a "real" filesystem, but not for a search in
the list of frozen modules.
2017-01-08 20:17:23 +11:00
Damien George 343b4189b0 py/mkrules.mk: Add MPY_CROSS_FLAGS option to pass flags to mpy-cross.
So that ports can pass their own custom options to mpy-cross.
2017-01-05 15:51:36 +11:00
Damien George 3f9c45efd1 py/asmarm: Fix assembler's PASS_EMIT constant name. 2017-01-03 15:40:50 +11:00
Damien George afc5063539 py/unicode: Comment-out unused function unichar_isprint. 2016-12-28 17:50:10 +11:00
Damien George ea6a958393 py/objint: Simplify mp_int_format_size and remove unreachable code.
One never needs to format integers with a base larger than 16 (but code
can be easily extended beyond this value if needed in the future).
2016-12-28 12:46:20 +11:00
Damien George 44bf8e1f2b py/mpprint: Add assertion for, and comment about, valid base values. 2016-12-28 12:45:33 +11:00
Damien George ca7af9a778 py/parsenum: Fix warning for signed/unsigned comparison. 2016-12-28 12:25:00 +11:00
Damien George 2d9440e2d1 py/mpz: Fix assertion in mpz_set_from_str which checks value of base. 2016-12-28 12:04:19 +11:00
Damien George c2dd494bd9 py/parsenum: Simplify and generalise decoding of digit values.
This function should be able to parse integers with any value for the
base, because it is called by int('xxx', base).
2016-12-28 12:02:49 +11:00
Paul Sokolovsky 25f44c19f1 cc3200: Re-add support for UART REPL (MICROPY_STDIO_UART setting).
UART REPL support was lost in os.dupterm() refactorings, etc. As
os.dupterm() is there, implement UART REPL support at the high level -
if MICROPY_STDIO_UART is set, make default boot.py contain os.dupterm()
call for a UART. This means that changing MICROPY_STDIO_UART value will
also require erasing flash on a module to force boot.py re-creation.
2016-12-27 01:05:37 +03:00
Paul Sokolovsky cf96be60dc py/misc.h: Typo fix in comment. 2016-12-27 01:05:30 +03:00
Damien George c305ae3243 py/lexer: Permanently disable the mp_lexer_show_token function.
The lexer is very mature and this debug function is no longer used.  If
it's really needed one can uncomment it and recompile.
2016-12-22 10:49:54 +11:00
Damien George f4aebafe7a py/lexer: Remove unnecessary check for EOF in lexer's next_char func.
This check always fails (ie chr0 is never EOF) because the callers of this
function never call it past the end of the input stream.  And even if they
did it would be harmless because 1) reader.readbyte must continue to
return an EOF char if the stream is exhausted; 2) next_char would just
count the subsequent EOF's as characters worth 1 column.
2016-12-22 10:39:06 +11:00
Damien George b9c4783273 py/lexer: Remove unreachable code in string tokeniser. 2016-12-22 10:37:13 +11:00
Damien George adccafb42a tests/basics/lexer: Add a test for newline-escaping within a string. 2016-12-22 10:32:06 +11:00
Paul Sokolovsky d02f6a9956 extmod/modutimeq: Refactor into optimized class.
import utimeq, utime
    # Max queue size, the queue allocated statically on creation
    q = utimeq.utimeq(10)
    q.push(utime.ticks_ms(), data1, data2)
    res = [0, 0, 0]
    # Items in res are filled up with results
    q.pop(res)
2016-12-22 00:29:32 +03:00
Damien George 46a6592f9a py/emitglue: Refactor to remove assert(0), to improve coverage. 2016-12-21 11:52:05 +11:00
Damien George e4af712125 py/objint: Rename mp_obj_int_as_float to mp_obj_int_as_float_impl.
And also simplify it to remove the check for small int.  This can be done
because this function is only ever called if the argument is not a small
int.
2016-12-21 11:46:27 +11:00
Damien George 7318949c46 py/modbuiltins: Remove unreachable code. 2016-12-20 14:00:59 +11:00
Damien George de9cd00b39 py/compile: Add an extra pass for Xtensa inline assembler.
It needs an extra pass to compute the size of the constant table for the
l32r instructions.
2016-12-19 17:42:25 +11:00
Damien George 7f1da0a03b py: Add MICROPY_KBD_EXCEPTION config option to provide mp_kbd_exception.
Defining and initialising mp_kbd_exception is boiler-plate code and so the
core runtime can provide it, instead of each port needing to do it
themselves.

The exception object is placed in the VM state rather than on the heap.
2016-12-15 13:00:19 +11:00
Paul Sokolovsky 403c93053e py/mpconfig.h: Enable MICROPY_PY_SYS_EXIT by default.
sys.exit() is an important function to terminate a program. In particular,
the testsuite relies on it to skip tests (i.e. any other functionality may
be disabled, but sys.exit() is required to at least report that properly).
2016-12-14 21:22:05 +03:00
Damien George e8f2db7da3 py/runtime: Zero out fs_user_mount array in mp_init.
There's no need to force ports to copy-and-paste this initialisation
code.  If FSUSERMOUNT is enabled then this zeroing out must be done.
2016-12-14 11:40:11 +11:00
Damien George e83f140463 py/mpz: Remove unreachable code in mpn_or_neg functions. 2016-12-14 10:39:41 +11:00
Damien George d9047d3c8a py/builtinimport: Support importing packages from compiled .mpy files.
This patch ensures that __init__.mpy files are imported if their
containing directory is imported as a package.
2016-12-13 15:09:48 +11:00
Damien George a3c61004c2 py/binary: Do zero extension when storing a value larger than word size. 2016-12-12 15:00:06 +11:00
Paul Sokolovsky 9d787de2a1 py/objint: from_bytes, to_bytes: Require byteorder arg, require "little".
CPython requires byteorder arg, make uPy compatible. As we support only
"little", error out on anything else.
2016-12-09 21:15:16 +03:00
Damien George 93ee6603b1 py/asm: Fix x86 and ARM assemblers due to recent code refactoring. 2016-12-09 22:54:45 +11:00
Damien George 155fdc74d5 py/asm: Remove need for dummy_data when doing initial assembler passes.
For all but the last pass the assembler only needs to count how much space
is needed for the machine code, it doesn't actually need to emit anything.
The dummy_data just uses unnecessary RAM and without it the code is not
any more complex (and code size does not increase for Thumb and Xtensa
archs).
2016-12-09 22:50:58 +11:00
Damien George e920bab976 py/emitinline: Move common code for end of final pass to compiler.
This patch moves some common code from the individual inline assemblers to
the compiler, the code that calls the emit-glue to assign the machine code
to the functions scope.
2016-12-09 21:23:17 +11:00
Damien George dd53b12193 py/emitinline: Move inline-asm align and data methods to compiler.
These are generic methods that don't depend on the architecture and so
can be handled directly by the compiler.
2016-12-09 20:54:54 +11:00
Damien George a7fd786a1f py/emitinline: Embed entire asm struct instead of a pointer to it.
This reduces fragmentation, and memory use by 1 word.  But more
importantly it means the emit_inline_asm_t struct now "derives" from
mp_asm_base.
2016-12-09 20:35:21 +11:00
Damien George f76b1bfa9f py: Add inline Xtensa assembler.
This patch adds the MICROPY_EMIT_INLINE_XTENSA option, which, when
enabled, allows the @micropython.asm_xtensa decorator to be used.

The following opcodes are currently supported (ax is a register, a0-a15):

    ret_n()
    callx0(ax)
    j(label)
    jx(ax)

    beqz(ax, label)
    bnez(ax, label)
    mov(ax, ay)
    movi(ax, imm) # imm can be full 32-bit, uses l32r if needed

    and_(ax, ay, az)
    or_(ax, ay, az)
    xor(ax, ay, az)
    add(ax, ay, az)
    sub(ax, ay, az)
    mull(ax, ay, az)

    l8ui(ax, ay, imm)
    l16ui(ax, ay, imm)
    l32i(ax, ay, imm)
    s8i(ax, ay, imm)
    s16i(ax, ay, imm)
    s32i(ax, ay, imm)
    l16si(ax, ay, imm)
    addi(ax, ay, imm)

    ball(ax, ay, label)
    bany(ax, ay, label)
    bbc(ax, ay, label)
    bbs(ax, ay, label)
    beq(ax, ay, label)
    bge(ax, ay, label)
    bgeu(ax, ay, label)
    blt(ax, ay, label)
    bnall(ax, ay, label)
    bne(ax, ay, label)
    bnone(ax, ay, label)

Upon entry to the assembly function the registers a0, a12, a13, a14 are
pushed to the stack and the stack pointer (a1) decreased by 16.  Upon
exit, these registers and the stack pointer are restored, and ret.n is
executed to return to the caller (caller address is in a0).

Note that the ABI for the Xtensa emitters is non-windowing.
2016-12-09 17:07:38 +11:00
Damien George ad297a1950 py: Allow inline-assembler emitter to be generic.
This patch refactors some code so that it is easier to integrate new
inline assemblers for different architectures other than ARM Thumb.
2016-12-09 17:06:21 +11:00
Damien George 8e5aced1fd py: Integrate Xtensa assembler into native emitter.
The config option MICROPY_EMIT_XTENSA can now be enabled to target the
Xtensa architecture with @micropython.native and @micropython.viper
decorators.
2016-12-09 16:51:49 +11:00
Damien George fcac4b07f1 py/asmxtensa: Add low-level Xtensa assembler. 2016-12-09 16:51:49 +11:00
Damien George 81316fa411 py/asmbase: Add MP_PLAT_COMMIT_EXEC option for handling exec code.
If a port defines MP_PLAT_COMMIT_EXEC then this function is used to turn
RAM data into executable code.  For example a port may want to write the
data to flash for execution.  The function must return a pointer to the
executable data.
2016-12-09 16:51:49 +11:00
Damien George c8746e1e72 py: Move arch-specific assembler macros from emitnative to asmXXX.h. 2016-12-09 16:51:49 +11:00
Damien George c2713592bc py/emit.h: Remove long-obsolete declarations for cpython emitter. 2016-12-09 13:28:25 +11:00
Damien George 080a78b15e py/compile: Simplify configuration of native emitter. 2016-12-07 11:17:17 +11:00
Damien George 304cfda8c4 py/stream: Move ad-hoc ioctl constants to stream.h and rename them.
The constants MP_IOCTL_POLL_xxx, which were stmhal-specific, are moved
from stmhal/pybioctl.h (now deleted) to py/stream.h.  And they are renamed
to MP_STREAM_POLL_xxx to be consistent with other such constants.

All uses of these constants have been updated.
2016-12-02 16:37:29 +11:00
Damien George 63e82dcdfe py/asmthumb: Fix build for F7 MCUs after recent code refactoring. 2016-11-30 00:11:25 +11:00
Damien George 612599587b py: Factor out common code from assemblers into asmbase.[ch].
All assemblers should "derive" from mp_asm_base_t.
2016-11-28 09:24:50 +11:00
Damien George e6cf5fb2cc py/compile: Remove comment about TODO for short circuiting for if-stmt.
Short circuiting is handled correctly by c_if_cond, and constants within
short-circuit expressions are optimised by the parser.
2016-11-26 16:15:55 +11:00
Paul Sokolovsky 037e6912c6 py/objtype: Implement __call__ handling for an instance w/o heap alloc.
By refactoring and reusing code from objboundmeth.
2016-11-22 01:33:55 +03:00
Paul Sokolovsky 8f5bc3ffc0 stmhal/moduselect: Move to extmod/ for reuse by other ports. 2016-11-21 00:05:56 +03:00
Damien George 5bdf1650de py/lexer: Make lexer use an mp_reader as its source. 2016-11-16 18:35:01 +11:00
Damien George 66d955c218 py/lexer: Rewrite mp_lexer_new_from_fd in terms of mp_reader. 2016-11-16 18:13:51 +11:00
Damien George e5ef15a9d7 py/lexer: Provide generic mp_lexer_new_from_file based on mp_reader.
If a port defines MICROPY_READER_POSIX or MICROPY_READER_FATFS then
lexer.c now provides an implementation of mp_lexer_new_from_file using
the mp_reader_new_file function.
2016-11-16 18:13:51 +11:00
Damien George 511c083811 py/lexer: Rewrite mp_lexer_new_from_str_len in terms of mp_reader_mem. 2016-11-16 18:13:50 +11:00
Damien George 6b239c271c py: Factor out persistent-code reader into separate files.
Implementations of persistent-code reader are provided for POSIX systems
and systems using FatFS.  Macros to use these are MICROPY_READER_POSIX and
MICROPY_READER_FATFS respectively.  If an alternative implementation is
needed then a port can define the function mp_reader_new_file.
2016-11-16 18:13:50 +11:00
Damien George 6810f2c134 py: Factor persistent code load/save funcs into persistentcode.[ch]. 2016-11-16 16:14:14 +11:00
Damien George 9b525134d1 py/parse: Add code to fold logical constants in or/and/not operations.
Adds about 200 bytes to the code size when constant folding is enabled.
2016-11-15 16:48:49 +11:00
Damien George ed9c93f0f1 py/parse: Make mp_parse_node_new_leaf an inline function.
It is split into 2 functions, one to make small ints and the other to make
a non-small-int leaf node.  This reduces code size by 32 bytes on
bare-arm, 64 bytes on unix (x64-64) and 144 bytes on stmhal.
2016-11-15 16:48:48 +11:00
Damien George b0cbfb0492 py/parse: Move function to check for const parse node to parse.[ch]. 2016-11-15 16:48:48 +11:00
Damien George 659b06b250 py/*.mk: Replace uses of 'sed' with $(SED). 2016-11-15 16:09:43 +11:00
Dave Hylands 0400fa45ba py/mkrules.mk: Rework find command so it works on OSX.
The Mac version of find doesn't support -printf, so this changes
things to use sed to strip off the leading path element instead.
2016-11-15 16:07:48 +11:00
Paul Sokolovsky a0b2c6ad32 py/runtime: mp_resume: Fix exception handling for nanbox port. 2016-11-15 01:41:22 +03:00
Paul Sokolovsky 79d996a57b py/runtime: mp_resume: Handle exceptions in Python __next__().
This includes StopIteration and thus are important to make Python-coded
iterables work with yield from/await.

Exceptions in Python send() are still not handled and left for future
consideration and optimization.
2016-11-15 01:10:34 +03:00
Paul Sokolovsky c3d96d387c py/objexcept: Allow clearing traceback with 'exc.__traceback__ = None'.
We allow 'exc.__traceback__ = None' assignment as a low-level optimization
of pre-allocating exception instance and raising it repeatedly - this
avoids memory allocation during raise. However, uPy will keep adding
traceback entries to such exception instance, so before throwing it,
traceback should be cleared like above.

'exc.__traceback__ = None' syntax is CPython compatible. However, unlike
it, reading that attribute or setting it to any other value is not
supported (and not intended to be supported, again, the only reason for
adding this feature is to allow zero-memalloc exception raising).
2016-11-14 02:29:09 +03:00
Paul Sokolovsky 59a1201da9 all: Remove readall() method, which is equivalent to read() w/o args.
Its addition was due to an early exploration on how to add CPython-like
stream interface. It's clear that it's not needed and just takes up
bytes in all ports.
2016-11-14 00:24:22 +03:00
Damien George 0f3388de1e py/emitnative: Fix native import emitter when in viper mode. 2016-11-10 22:53:04 +11:00
Damien George fcda6a2a78 py: Strip leading dirs from frozen mpy files, so any path can be used.
With this patch one can now do "make FROZEN_MPY_DIR=../../frozen" to
specify a directory containing scripts to be frozen (as well as absolute
paths).

The compiled .mpy files are now stored in $(BUILD)/frozen_mpy/.
2016-11-08 14:36:06 +11:00
Damien George bdf33bc136 py: Move frozen bytecode Makefile rules from ports to common mk files.
Now, to use frozen bytecode all a port needs to do is define
FROZEN_MPY_DIR to the directory containing the .py files to freeze, and
define MICROPY_MODULE_FROZEN_MPY and MICROPY_QSTR_EXTRA_POOL.
2016-11-08 14:28:30 +11:00
Damien George 561844f3ba py: Add MICROPY_FLOAT_CONST macro for defining float constants.
All float constants in the core should use this macro to prevent
unnecessary creation of double-precision floats, which makes code less
efficient.
2016-11-03 12:33:01 +11:00
Colin Hogben 828df54bfe py: Change config default so m_malloc0 uses memset if GC not enabled.
With MICROPY_ENABLE_GC set to false the alternate memory manager may not
clear all memory that is allocated, so it must be cleared in m_malloc0.
2016-11-03 10:16:31 +11:00
Colin Hogben f9b6b37cf6 py: Fix wrong assumption that m_renew will not move if shrinking
In both parse.c and qstr.c, an internal chunking allocator tidies up
by calling m_renew to shrink an allocated chunk to the size used, and
assumes that the chunk will not move.  However, when MICROPY_ENABLE_GC
is false, m_renew calls the system realloc, which does not guarantee
this behaviour.  Environments where realloc may return a different
pointer include:

(1) mbed-os with MBED_HEAP_STATS_ENABLED (which adds a wrapper around
malloc & friends; this is where I was hit by the bug);

(2) valgrind on linux (how I diagnosed it).

The fix is to call m_renew_maybe with allow_move=false.
2016-11-02 23:15:41 +11:00
Jan Pochyla ffb04a5845 unix: fix symbol references for x86 Mac 2016-11-02 00:42:04 +03:00
Pavol Rusnak 7ffc959c00 py: remove asserts that are always true in emitbc.c 2016-10-31 23:21:22 +03:00
Pavol Rusnak 3679ee9b52 py: fix null pointer dereference in mpz.c, fix missing va_end in warning.c 2016-10-31 23:21:15 +03:00
Fabio Utzig 8908e505ce py/sequence: Fix reverse slicing of lists. 2016-10-30 15:54:19 -02:00
Paul Sokolovsky 76146b3d9a extmod/utime_mphal: Allow ticks functions period be configurable by a port.
Using MICROPY_PY_UTIME_TICKS_PERIOD config var.
2016-10-30 03:02:07 +03:00
Paul Sokolovsky 8a49905a2f py/stream: Typo fix in comment. 2016-10-27 22:13:45 +03:00
Paul Sokolovsky f00ecdb54d extmod/moduos_dupterm: Renamed to uos_dupterm.
As part of file naming clean up (moduos_dupterm doesn't implement a
full module, so should skip "mod" prefix, similar to other files in
extmod/).
2016-10-26 02:08:37 +03:00
Damien George 5076e5c339 py: Add "delattr" builtin, conditional on MICROPY_CPYTHON_COMPAT. 2016-10-24 13:50:03 +11:00
Damien George bdb0d2d0bc py/modbuiltins: Add builtin "slice", pointing to existing slice type. 2016-10-24 13:35:39 +11:00
Paul Sokolovsky 3730090d8f py/{modbuiltins,obj}: Use MP_PYTHON_PRINTER where possible. 2016-10-22 01:07:07 +03:00
Erik Moqvist f64e806f50 lib/utils/pyhelp.c: Use mp_printf() instead of printf()
This patch introduces MP_PYTHON_PRINTER for general use.
2016-10-21 18:30:58 +11:00
Damien George 571e6f26db py: Specialise builtin funcs to use separate type for fixed arg count.
Builtin functions with a fixed number of arguments (0, 1, 2 or 3) are
quite common.  Before this patch the wrapper for such a function cost
3 machine words.  After this patch it only takes 2, which can reduce the
code size by quite a bit (and pays off even more, the more functions are
added).  It also makes function dispatch slightly more efficient in CPU
usage, and furthermore reduces stack usage for these cases.  On x86 and
Thumb archs the dispatch functions are now tail-call optimised by the
compiler.

The bare-arm port has its code size increase by 76 bytes, but stmhal drops
by 904 bytes.  Stack usage by these builtin functions is decreased by 48
bytes on Thumb2 archs.
2016-10-21 16:26:01 +11:00
Damien George 4ebdb1f2b2 py: Be more specific with MP_DECLARE_CONST_FUN_OBJ macros.
In order to have more fine-grained control over how builtin functions are
constructed, the MP_DECLARE_CONST_FUN_OBJ macros are made more specific,
with suffix of _0, _1, _2, _3, _VAR, _VAR_BETEEN or _KW.  These names now
match the MP_DEFINE_CONST_FUN_OBJ macros.
2016-10-21 16:26:01 +11:00
Paul Sokolovsky b440307b4a py/py.mk: Automatically add frozen.c to source list if FROZEN_DIR is defined.
Now frozen modules generation handled fully by py.mk and available for reuse
by any port.
2016-10-21 01:08:43 +03:00
Damien George 7d0d7215d2 py: Use mp_raise_msg helper function where appropriate.
Saves the following number of bytes of code space: 176 for bare-arm, 352
for minimal, 272 for unix x86-64, 140 for stmhal, 120 for esp8266.
2016-10-17 12:17:37 +11:00
Damien George 2750a7b38e py/objdict: Actually provide the key that failed in KeyError exception.
The failed key is available as exc.args[0], as per CPython.
2016-10-17 12:00:19 +11:00
Damien George a3edeb9ea5 py/objdict: Fix optimisation for allocating result in fromkeys.
Iterables don't respond to __len__, so call __len__ on the original
argument.
2016-10-17 11:58:57 +11:00
Paul Sokolovsky a97284423e extmod/utime_mphal: Factor out implementations in terms of mp_hal_* for reuse.
As long as a port implement mp_hal_sleep_ms(), mp_hal_ticks_ms(), etc.
functions, it can just use standard implementations of utime.sleel_ms(),
utime.ticks_ms(), etc. Python-level functions.
2016-10-14 20:14:01 +03:00
Damien George 824f5c5a32 py/vstr: Combine vstr_new_size with vstr_new since they are rarely used.
Now there is just one function to allocate a new vstr, namely vstr_new
(in addition to vstr_init etc).  The caller of this function should know
what initial size to allocate for the buffer, or at least have some policy
or config option, instead of leaving it to a default (as it was before).
2016-10-14 16:46:34 +11:00
Damien George e93c1ca5da extmod/modujson: Implement ujson.load() to load JSON from a stream.
This refactors ujson.loads(s) to behave as ujson.load(StringIO(s)).

Increase in code size is: 366 bytes for unix x86-64, 180 bytes for
stmhal, 84 bytes for esp8266.
2016-10-13 11:46:14 +11:00
Damien George af8d791bd0 esp8266: Enable importing of precompiled .mpy files. 2016-10-12 11:03:58 +11:00
Damien George 31101d91ce py/lexer: Remove unnecessary code, and unreachable code.
Setting emit_dent=0 is unnecessary because arriving in that part of the
if-logic will guarantee that emit_dent is already zero.

The block to check indent_top(lex)>0 is unreachable because a newline is
always inserted an the end of the input stream, and hence dedents are
always processed before EOF.
2016-10-12 11:00:17 +11:00
Damien George deaa57acf3 py/compile: Remove debugging code for compiler dispatch.
It was a relic from the days of developing the compiler and is no longer
needed, and it's impossible to trigger via a test.
2016-10-12 10:20:48 +11:00
Damien George 6dff3df501 py/objint: Use size_t for arguments that measure bytes/sizes. 2016-10-11 13:20:11 +11:00
Damien George 8bb7d958f1 py: Factor duplicated function to calculate size of formatted int. 2016-10-11 13:11:32 +11:00
Damien George df3e5d2b2f py/mpz: Use assert to verify mpz does not have a fixed digit buffer. 2016-10-11 13:00:56 +11:00
Damien George 48874942f0 py/mpz: In divmod, replace check for rhs!=0 with assert.
The check for division by zero is made by the caller of this function.
2016-10-11 13:00:01 +11:00
Damien George e49153fb98 py/compile: Remove unreachable code. 2016-10-11 12:29:54 +11:00
Damien George 7dc2345715 py/modmicropython: Add micropython.opt_level([value]) function.
This allows to get/set at runtime the optimisation level of the compiler.
2016-10-11 10:56:22 +11:00
Paul Sokolovsky d22a04d9c4 py/objstringio: Add readinto() method.
Also, drop deprecated (as for MicroPython) readall() method.
2016-10-09 11:56:11 +03:00
Damien George 3a0a771730 py: Add mp_raise_OSError(errno) helper function.
This is an often used code pattern, and its use reduces code size of the
core by about 100 bytes.
2016-10-07 13:31:59 +11:00
Damien George dffa383b06 py/modstruct: Remove unreachable code, and add comment about CPy diff.
The deleted code is unreachable because calcsize_items guarantees that
num_items corresponds to how many items there are in fmt to unpack.
2016-10-07 12:54:14 +11:00
Damien George 8de270b4fc py/objbool: Make a slight simplification of bool constructor.
Reduces code size for some archs.
2016-09-30 17:02:07 +10:00
Damien George eca1408f16 py/objbool: Defer bool's unary op implementation to small int.
Similar to how binary op already works.  Common unary operations already
have fast paths for bool so there's no need to have explicit handling of
ops in bool_unary_op, especially since they have the same behaviour as
integers.
2016-09-30 17:02:06 +10:00
Damien George 3be4f886ce py/argcheck: Simplify if-chain so that the last one is the default. 2016-09-30 16:45:43 +10:00
Damien George 216a711cd4 py/compile: Fix typo when checking for parse-node kind. 2016-09-30 14:48:06 +10:00
Damien George 0d10517a45 py/scope: Factor common code to find locals and close over them.
Saves 50-100 bytes of code.
2016-09-30 13:53:00 +10:00
Damien George d5495966ce py/scope: Shrink scope_t struct by 1 machine word.
On 32-bit archs this makes the scope_t struct 48 bytes in size, which fits
in 3 GC blocks (previously it used 4 GC blocks).  This will lead to some
savings when compiling scripts because there are usually quite a few scopes,
one for each function and class.

Note that qstrs will fit in 16 bits, this assumption is made in a few other
places.
2016-09-30 12:45:00 +10:00
Damien George 3dea8c9e92 py/scope: Use lookup-table to determine a scope's simple name.
Generates slightly smaller and more efficient code.
2016-09-30 12:34:05 +10:00
Damien George b32c01b748 py/compile: Fix async-for/async-with to work with simpler exc on stack.
There is now just the exception instance on the stack when an exception is
raised, not the full (type, exc, traceback).
2016-09-28 11:52:13 +10:00
Damien George 2c7716fed0 py/objset: Ensure that use of frozenset.update raises an exception. 2016-09-28 11:06:18 +10:00
Damien George dd4135aeaf py/objset: Use mp_check_self() to check args of set/frozenset methods.
Following how other objects work, set/frozenset methods should use the
mp_check_self() macro to check the type of the self argument, because in
most cases this check can be a null operation.

Saves about 100-180 bytes of code for builds with set and frozenset
enabled.
2016-09-28 10:55:23 +10:00
Damien George 0c595fa094 py/objfun: Use if instead of switch to check return value of VM execute.
It's simpler and improves code coverage.
2016-09-27 23:08:10 +10:00
Damien George c71edaed73 py/objfun: Remove unnecessary check for viper fun with 5 or more args.
The native emitter/compiler restricts viper functions to 4 args, so there
is no need for an extra check in the dynamic dispatch.
2016-09-27 23:05:51 +10:00
Damien George 7385b018ed py/emitbc: Remove/refactor unreachable code, to improve coverage. 2016-09-27 15:46:50 +10:00
Damien George 897129a7ff py/objstr: Remove unreachable function used only for terse error msgs. 2016-09-27 15:45:42 +10:00
Damien George 791b65f4b2 py/modmicropython: Add micropython.const, alias for identity function.
Having a micropython.const identity function, and writing "from micropython
import const" at the start of scripts that use the const feature, allows to
write scripts which are compatible with CPython, and with uPy builds that
don't include const optimisation.

This patch adds such a function and updates the tests to do the import.
2016-09-27 13:34:21 +10:00
Damien George 71fec076dc py/vm: Use MP_OBJ_FROM_PTR to cast a type to an object. 2016-09-27 13:21:23 +10:00
Damien George f040685b0c py: Only store the exception instance on Py stack in bytecode try block.
When an exception is raised and is to be handled by the VM, it is stored
on the Python value stack so the bytecode can access it.  CPython stores
3 objects on the stack for each exception: exc type, exc instance and
traceback.  uPy followed this approach, but it turns out not to be
necessary.  Instead, it is enough to store just the exception instance on
the Python value stack.  The only place where the 3 values are needed
explicitly is for the __exit__ handler of a with-statement context, but
for these cases the 3 values can be extracted from the single exception
instance.

This patch removes the need to store 3 values on the stack, and instead
just stores the exception instance.

Code size is reduced by about 50-100 bytes, the compiler and VM are
slightly simpler, generate bytecode is smaller (by 2 bytes for each try
block), and the Python value stack is reduced in size for functions that
handle exceptions.
2016-09-27 12:37:21 +10:00
Paul Sokolovsky 9cc8ec843e py/py.mk: Add support for building modussl_mbedtls. 2016-09-23 14:30:46 +03:00
Damien George 6d310a5552 py/parse: Only replace constants that are standalone identifiers.
This fixes constant substitution so that only standalone identifiers are
replaced with their constant value (if they have one).  I.e. don't
replace NAME in expressions like obj.NAME or NAME = expr.
2016-09-23 17:23:16 +10:00
Damien George 7df9291b6c py: Update opcode format table because 3 opcodes were removed, 1 added.
LIST_APPEND, MAP_ADD and SET_ADD have been removed, and STORE_COMP has
been added in adaf0d865c.
2016-09-23 12:48:57 +10:00
Damien George e97df97600 py: Shrink mp_arg_t struct by using reduced-size integer members.
qstrs ids are restricted to fit within 2 bytes already (eg in persistent
bytecode) so it's safe to use a uint16_t to store them in mp_arg_t.  And
the flags member only needs a maximum of 2 bytes so can also use uint16_t.

Savings in code size can be significant when many mp_arg_t structs are
used for argument parsing.  Eg, this patch reduces stmhal by 480 bytes.
2016-09-23 12:13:51 +10:00
Damien George 6c79980b0e py/py.mk: Suppress some compiler warnings when building berkeley-db. 2016-09-22 11:09:21 +10:00
Damien George 79ec869f95 py/stream: Remove unnecessary check for NULL return from vstr_extend.
vstr_extend will now only return NULL if the vstr is a fixed buffer, which
in this case it is not.
2016-09-22 10:50:47 +10:00
Damien George 93c4a6a3f7 all: Remove 'name' member from mp_obj_module_t struct.
One can instead lookup __name__ in the modules dict to get the value.
2016-09-22 00:23:16 +10:00
Paul Sokolovsky 7ea3fa2641 py/builtinimport: Fix nanbox build after change to better handle -m modules. 2016-09-20 17:55:42 +03:00
Delio Brignoli 21c719bd0a builtinimport: add the module specified by -m to sys.modules as '__main__' 2016-09-20 14:01:31 +03:00
Damien George fbddea929d py/showbc: Make printf's go to the platform print stream.
The system printf is no longer used by the core uPy code.  Instead, the
platform print stream or DEBUG_printf is used.  Using DEBUG_printf in the
showbc functions would mean that the code can't be tested by the test
suite, so use the normal output instead.

This patch also fixes parsing of bytecode-line-number mappings.
2016-09-20 11:30:54 +10:00
Damien George e60835bac5 py/qstr: Remove a comment.
qstrs are always null terminated so qstr_str will stay as part of the API.
2016-09-19 13:18:54 +10:00
Damien George 8dd5960ac0 py/objnone: Use mp_generic_unary_op instead of custom one. 2016-09-19 12:28:55 +10:00
Damien George 5da0d29d3c py/vstr: Remove vstr.had_error flag and inline basic vstr functions.
The vstr.had_error flag was a relic from the very early days which assumed
that the malloc functions (eg m_new, m_renew) returned NULL if they failed
to allocate.  But that's no longer the case: these functions will raise an
exception if they fail.

Since it was impossible for had_error to be set, this patch introduces no
change in behaviour.

An alternative option would be to change the malloc calls to the _maybe
variants, which return NULL instead of raising, but then a lot of code
will need to explicitly check if the vstr had an error and raise if it
did.

The code-size savings for this patch are, in bytes: bare-arm:188,
minimal:456, unix(NDEBUG,x86-64):368, stmhal:228, esp8266:360.
2016-09-19 12:28:55 +10:00
Damien George adaf0d865c py: Combine 3 comprehension opcodes (list/dict/set) into 1.
With the previous patch combining 3 emit functions into 1, it now makes
sense to also combine the corresponding VM opcodes, which is what this
patch does.  This eliminates 2 opcodes which simplifies the VM and reduces
code size, in bytes: bare-arm:44, minimal:64, unix(NDEBUG,x86-64):272,
stmhal:92, esp8266:200.  Profiling (with a simple script that creates many
list/dict/set comprehensions) shows no measurable change in performance.
2016-09-19 12:28:03 +10:00
Damien George a5624bf381 py: Combine 3 comprehension emit functions (list/dict/set) into 1.
The 3 kinds of comprehensions are similar enough that merging their emit
functions reduces code size.  Decreases in code size in bytes are:
bare-arm:24, minimal:96, unix(NDEBUG,x86-64):328, stmhal:80, esp8266:76.
2016-09-19 12:23:31 +10:00
Paul Sokolovsky f28efa1971 py: Move frozen modules rules from esp8266 port for reuse across ports.
A port now just needs to define FROZEN_DIR var and add $(BUILD)/frozen.c
to SRC_C to support frozen modules.
2016-09-17 21:00:40 +03:00
Damien George f84b341618 py/objnone: Remove unnecessary handling of MP_UNARY_OP_BOOL.
bool(None) has a fast path in mp_obj_is_true so doesn't need to be
handled in none_unary_op.  The only caveat is that subclassing may
bypass the mp_obj_is_true function, but actually you aren't allowed to
subclass classes that have singleton instances like NoneType (see
https://mail.python.org/pipermail/python-dev/2002-March/020822.html for
reference on this point).
2016-09-16 12:30:09 +10:00
Chris Packham a50b26e4b0 py/makeqstrdefs.py: Use python 2.6 syntax for set creation.
py/makeqstrdefs.py declares that it works with python 2.6 however the
syntax used to initialise of a set with values was only added in python
2.7. This leads to build failures when the host system doesn't have
python 2.7 or newer.

Instead of using the new syntax pass a list of initial values through
set() to achieve the same result. This should work for python versions
from at least 2.6 onwards.

Helped-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Chris Packham <judge.packham@gmail.com>
2016-09-09 23:01:23 +10:00
Antonin ENFRUN f3b19ef634 py/asmthumb: Flush D-cache, and invalidate I-cache on STM32F7.
Tested on a STM32F7DISCO at 216MHz.  All tests generating code (inlineasm,
native, viper) now pass, except pybnative/while.py, but that's because
there is no LED(2).
2016-09-09 14:48:15 +10:00
Delio Brignoli e2ac8bb3f1 py: Add MICROPY_USE_INTERNAL_PRINTF option, defaults to enabled.
This new config option allows to control whether MicroPython uses its own
internal printf or not (if not, an external one should be linked in).
Accompanying this new option is the inclusion of lib/utils/printf.c in the
core list of source files, so that ports no longer need to include it
themselves.
2016-09-05 12:18:53 +10:00
Chris Packham 8c6856d2e7 py/emitglue.c: provide mp_raw_code_load_file for any unix architecture
Signed-off-by: Chris Packham <chris.packham@alliedtelesis.co.nz>
2016-09-04 16:17:46 +10:00
Damien George 5f3bda422a py: If str/bytes hash is 0 then explicitly compute it. 2016-09-02 14:49:50 +10:00
Damien George f127bef3e4 py/makeqstrdata.py: Compute the qstr hash from bytes, not characters. 2016-09-02 14:32:47 +10:00
Delio Brignoli f98bb2ddcb py/mpprint: Fail an assertion with unsupported format specifiers.
Arguments of an unknown type cannot be skipped and continuing to parse a
format string after encountering an unknown format specifier leads to
undefined behaviour.  This patch helps to find use of unsupported formats.
2016-09-01 18:09:44 +10:00
Damien George 0823c1baf8 extmod: Add machine_spi with generic SPI C-protocol and helper methods.
The idea is that all ports can use these helper methods and only need to
provide initialisation of the SPI bus, as well as a single transfer
function.  The coding pattern follows the stream protocol and helper
methods.
2016-09-01 15:07:20 +10:00
Paul Sokolovsky 1f61fe07a2 py/mkrules.mk: Allow to override name of libmicropython.a
Or alternatively, refer to an exact library file, not just phony target
"lib".
2016-08-30 00:59:17 +03:00
Damien George f4ee9f8853 py/bc.h: Rename _mp_code_state to _mp_code_state_t.
This rename was missed in the previous patch.
2016-08-27 23:23:51 +10:00
Damien George 581a59a456 py: Rename struct mp_code_state to mp_code_state_t.
Also at _t to mp_exc_stack pre-declaration in struct typedef.
2016-08-27 23:21:00 +10:00
Damien George 26295e04ff py/emitglue: Use more compact mp_raise_ValueError function. 2016-08-26 23:45:07 +10:00
Damien George fe3cc5bb53 py/modstruct: Use more compact mp_raise_ValueError function.
Saves a few bytes of code size.
2016-08-26 23:42:40 +10:00
Damien George 24df30c133 py/compile: Don't compile assert statements when optimisations enabled.
As per CPython.
2016-08-26 22:28:22 +10:00
Damien George 5ffe1d8dc0 py/gc: Add MICROPY_GC_CONSERVATIVE_CLEAR option to always zero memory.
There can be stray pointers in memory blocks that are not properly zero'd
after allocation.  This patch adds a new config option to always zero all
allocated memory (via gc_alloc and gc_realloc) and hence help to eliminate
stray pointers.

See issue #2195.
2016-08-26 15:35:26 +10:00
Paul Sokolovsky c428367543 extmod/modubinascii: Make crc32() support configurable.
Disable by default, enable in unix port.
2016-08-24 18:28:43 +03:00
Krzysztof Blazewicz 6562076454 py/stream.c: use mp_obj_get_type in mp_get_stream_raise
In current state `mp_get_stream_raise` assumes that `self_in` is an object
and always performs a pointer derefence which may cause a segfault.

This function shall throw an exception whenever `self_in` does not implement
a stream protocol, that includes qstr's and numbers.

fixes #2331
2016-08-24 01:33:31 +03:00
Damien George 095e43a9a5 py/sequence: Allow to use bignums as indices in slice objects.
See issue #2264.
2016-08-15 23:26:34 +10:00
Damien George 41fceae559 py/obj.h: For obj reprs A,B,C use void* explicitly for mp_obj_t typedef.
The machine_ptr_t type is long obsolete as the type of mp_obj_t is now
defined by the object representation, ie by MICROPY_OBJ_REPR.  So just use
void* explicitly for the typedef of mp_obj_t.

If a port wants to use something different then they should define a new
object representation.
2016-08-15 10:56:55 +10:00
Damien George 2196799051 py/objstr: Use mp_raise_{Type,Value}Error instead of mp_raise_msg.
This patch does further refactoring using the new mp_raise_TypeError
and mp_raise_ValueError functions.
2016-08-14 16:51:54 +10:00
Damien George 1a0d3fd632 py/runtime.h: Move comment about mp_not_implemented to correct place. 2016-08-14 16:35:10 +10:00
Damien George 4aaa5adf9f py/objtuple: In tuple_cmp_helper, use mp_check_self instead of raising.
Only tuple, namedtuple and attrtuple use the tuple_cmp_helper function,
and they all have getiter=mp_obj_tuple_getiter, so the check here is only
to ensure that the self object is consistent.  Hence use mp_check_self.
2016-08-14 16:28:05 +10:00
Paul Sokolovsky c4a8004933 py: Get rid of assert() in method argument checking functions.
Checks for number of args removes where guaranteed by function descriptor,
self checking is replaced with mp_check_self(). In few cases, exception
is raised instead of assert.
2016-08-12 22:39:03 +03:00
Paul Sokolovsky 83e0ebabb4 py/objdict: Get rid of asserts (remove/replace with mp_check_self()). 2016-08-12 22:01:11 +03:00
Paul Sokolovsky 8c50f93a41 py/runtime.h: Define mp_check_self(pred) helper macro.
Indended to replace raw asserts in bunch of files. Expands to empty
if MICROPY_BUILTIN_METHOD_CHECK_SELF_ARG is defined, otehrwise by
default still to assert, though a particular port may define it to
something else.
2016-08-12 21:58:56 +03:00
Paul Sokolovsky 9e1b61dedd py/runtime: Factor out exception raising helpers.
Introduce mp_raise_msg(), mp_raise_ValueError(), mp_raise_TypeError()
instead of previous pattern nlr_raise(mp_obj_new_exception_msg(...)).
Save few bytes on each call, which are many.
2016-08-12 21:28:45 +03:00
Paul Sokolovsky 1bc2911174 py/mpconfig.h: Define MP_ALWAYSINLINE for reuse.
Similar to existing MP_NOINLINE.
2016-08-07 22:36:05 +03:00
Paul Sokolovsky 1563388001 py/objstr,objstrunicode: Fix inconistent #if indentation. 2016-08-07 15:24:57 +03:00
Paul Sokolovsky 56eb25f049 py/objstr: Make .partition()/.rpartition() methods configurable.
Default is disabled, enabled for unix port. Saves 600 bytes on x86.
2016-08-07 06:46:55 +03:00
Paul Sokolovsky 7fb31479bf py/mkrules.mk: Allow to add more items for "clean" target using CLEAN_EXTRA. 2016-08-04 23:46:08 +03:00
Paul Sokolovsky 0dfe849413 py/py.mk: Extra switches to build "embedded" BerkeleyDB BTree lib. 2016-07-31 00:39:09 +03:00
Paul Sokolovsky 61e77a4e88 py/mpconfig.h: Add MICROPY_STREAMS_POSIX_API setting.
To filter out even prototypes of mp_stream_posix_*() functions, which
require POSIX types like ssize_t & off_t, which may be not available in
some ports.
2016-07-30 20:05:56 +03:00
Paul Sokolovsky c8b80e4740 lib/embed/abort_: Implementation of abort_() function raising uPy exception.
Helpful when porting existing C libraries to MicroPython. abort()ing in
embedded environment isn't a good idea, so when compiling such library,
-Dabort=abort_ option can be given to redirect standard abort() to this
"safe" version.
2016-07-30 00:35:50 +03:00
Paul Sokolovsky 4f1b0292db py/stream: Add adapter methods with POSIX-compatible signatures.
Previoussly such read() and write() methods were used by modussl_axtls,
move to py/stream for reuse.
2016-07-30 00:25:06 +03:00
Paul Sokolovsky 3990b1715d py/objstringio: Implement MP_STREAM_SEEK ioctl and add seek() method. 2016-07-28 01:53:44 +03:00
Paul Sokolovsky f039ac5bd7 py/objstringio: Add MP_STREAM_FLUSH ioctl and flush() method.
No-op for this object.
2016-07-28 01:14:32 +03:00
Paul Sokolovsky a60b0263ba py/stream: Implement generic flush() method, in terms of C-level ioctl. 2016-07-27 00:39:10 +03:00
Paul Sokolovsky f2f761c0c3 py/stream: Stream module works with errno's, so should include mperrno.h. 2016-07-26 15:05:15 +03:00
Paul Sokolovsky ed1c194ebf py/objstrunicode: str_index_to_ptr: Implement positive indexing properly.
Order out-of-bounds check, completion check, and increment in the right way.
2016-07-25 19:28:04 +03:00
Paul Sokolovsky 6af90b2972 py/objstrunicode: str_index_to_ptr: Should handle bytes too.
There's single str_index_to_ptr() function, called for both bytes and
unicode objects, so should handle each properly.
2016-07-25 14:45:08 +03:00
Paul Sokolovsky 16f324641f py/stream.h: Remove dated comment of POSIX-specificity of EAGAIN.
We have adopted POSIX-compatible error numbers as MicroPython's native.
2016-07-25 00:47:00 +03:00
Paul Sokolovsky 1a7e28d8b7 py/stream.h: Move mp_stream_write_adaptor() inside ifdef block. 2016-07-25 00:16:51 +03:00
Paul Sokolovsky 918851e836 unix: Enable MICROPY_PY_STR_BYTES_CMP_WARN.
Also, fix a warning text (remove "duplicate" BytesWarning).
2016-07-22 00:52:07 +03:00
Paul Sokolovsky a1b442bc07 py/mpconfig.h: Fix description for MICROPY_PY_STR_BYTES_CMP_WARN. 2016-07-22 00:46:24 +03:00
Paul Sokolovsky 707cae7494 py/obj: Issue a warning when str and bytes objects are compared.
Something like:

if foo == "bar":

will be always false if foo is b"bar". In CPython, warning is issued if
interpreter is started as "python3 -b". In MicroPython,
MICROPY_PY_STR_BYTES_CMP_WARN setting controls it.
2016-07-22 00:34:34 +03:00