3128 Commits

Author SHA1 Message Date
tll
68c28174d0 py/objstr: Add check for valid UTF-8 when making a str from bytes.
This patch adds a function utf8_check() to check for a valid UTF-8 encoded
string, and calls it when constructing a str from raw bytes.  The feature
is selectable at compile time via MICROPY_PY_BUILTINS_STR_UNICODE_CHECK and
is enabled if unicode is enabled.  It costs about 110 bytes on Thumb-2, 150
bytes on Xtensa and 170 bytes on x86-64.
2017-09-06 16:43:09 +10:00
Damien George
4a93801c12 all: Update Makefiles and others to build with new ports/ dir layout.
Also renames "stmhal" to "stm32" in documentation and everywhere else.
2017-09-06 14:09:13 +10:00
Scott Shawcroft
6baacf46b7 py: Only load frozen modules when the filename has the prefix. (#235)
* py: Only load frozen modules when the filename has the prefix.

This allows one to override a built-in module by loading a newer
version onto the file system.

* Unbreak mpys
2017-09-05 22:01:17 -04:00
Paul Sokolovsky
1aaba5cabe py/objtuple: Properly implement comparison with incompatible types.
Should raise TypeError, unless it's (in)equality comparison.
2017-09-06 00:23:41 +03:00
Paul Sokolovsky
60749e57f2 py/objtype: Implement fallback for instance inplace special methods.
If __iop__ is not defined, call __op__ instead. This is desired behavior
for immutable types, __iop__ needs to be defined only for mutable types.
2017-09-04 16:44:21 +03:00
Damien George
77a48e8cd4 py/obj: Remove declaration for mp_obj_new_none(), it's never defined. 2017-09-04 23:35:46 +10:00
Damien George
d4b75f6b68 py/obj: Fix comparison of float/complex NaN with itself.
IEEE floating point is specified such that a comparison of NaN with itself
returns false, and Python respects these semantics.  This patch makes uPy
also have these semantics.  The fix has a minor impact on the speed of the
object-equality fast-path, but that seems to be unavoidable and it's much
more important to have correct behaviour (especially in this case where
the wrong answer for nan==nan is silently returned).
2017-09-04 14:16:27 +10:00
Paul Sokolovsky
9950865c39 py/objfloat: Fix binary ops with incompatible objects.
These are now returned as "operation not supported" instead of raising
TypeError. In particular, this fixes equality for float vs incompatible
types, which now properly results in False instead of exception. This
also paves the road to support reverse operation (e.g. __radd__) with
float objects.

This is achieved by introducing mp_obj_get_float_maybe(), similar to
existing mp_obj_get_int_maybe().
2017-09-02 23:05:24 +03:00
Dan Halbert
a0d0b27faf Fix issue #207, esp8266 file operations problems (#222)
The frozen module `_boot.py` was not being loaded on restart
because `pyexec_frozen_module()` did not know about the new `.frozen`
pseudo-directory. Updated lower-level routine to look in the right place.
Also made ".frozen" and related values be `#define`s.
2017-09-01 16:12:26 -04:00
Damien George
dd376a239d py/nlrthumb: Get working again on standard Thumb arch (ie not Thumb2).
"b" on Thumb might not be long enough for the jump to nlr_push_tail so it
must be done indirectly.
2017-09-01 15:25:29 +10:00
Damien George
860eeeea9b py/qstrdefs: Remove unused qstrs.
They are not used by any component and take up valuable flash space.
2017-09-01 15:22:25 +10:00
Damien George
2daacc5cee py/modstruct: Check and prevent buffer-write overflow in struct packing.
Prior to this patch, the size of the buffer given to pack_into() was checked
for being too small by using the count of the arguments, not their actual
size.  For example, a format spec of '4I' would only check that there was 4
bytes available, not 16; and 'I' would check for 1 byte, not 4.

The pack() function is ok because its buffer is created to be exactly the
correct size.

The fix in this patch calculates the total size of the format spec at the
start of pack_into() and verifies that the buffer is large enough.  This
adds some computational overhead, to iterate through the whole format spec.
The alternative is to check during the packing, but that requires extra
code to handle alignment, and the check is anyway not needed for pack().
So to maintain minimal code size the check is done using struct_calcsize.
2017-09-01 11:11:09 +10:00
Damien George
79d5acbd01 py/modstruct: Check and prevent buffer-read overflow in struct unpacking
Prior to this patch, the size of the buffer given to unpack/unpack_from was
checked for being too small by using the count of the arguments, not their
actual size.  For example, a format spec of '4I' would only check that
there was 4 bytes available, not 16; and 'I' would check for 1 byte, not 4.

This bug is fixed in this patch by calculating the total size of the format
spec at the start of the unpacking function.  This function anyway needs to
calculate the number of items at the start, so calculating the total size
can be done at the same time.
2017-09-01 10:53:29 +10:00
Damien George
793d826d9d py/modstruct: In struct.pack, stop converting if there are no args left.
This patch makes a repeat counter behave the same as repeating the
typecode, when there are not enough args.  For example:
struct.pack('2I', 1) now behave the same as struct.pack('II', 1).
2017-09-01 10:10:51 +10:00
Scott Shawcroft
4d0dc4b298 py: Add temporary expanded mpy mismatch message for switch to 2.0.0. (#218)
Fixes #214
2017-08-31 15:27:50 -04:00
Damien George
ca21aed0a1 py: Make m_malloc_fail() have void return type, since it doesn't return. 2017-08-31 17:00:14 +10:00
Damien George
6c9fca2aa9 py/map: Remove unused new/free functions.
Maps are always allocated "statically" and (de)initialised via mp_map_init
and mp_map_deinit.
2017-08-31 16:46:13 +10:00
Damien George
0e420d48ee py/map: Replace always-false condition with assertion. 2017-08-31 16:45:02 +10:00
Paul Sokolovsky
4556bd2acd py/objtype: mp_obj_class_lookup: Improve debug logging.
Now traces more explicitly thru the lookup process.
2017-08-31 00:44:51 +03:00
Paul Sokolovsky
df6605eaba py/objtype: mp_obj_instance_make_new: Fix typos in comment. 2017-08-30 20:55:34 +03:00
Damien George
0102ee092b py: Change obsolete "///" comment formatting to normal comments.
This comment style is no longer used because the docs are written by hand,
not generated.
2017-08-30 21:02:00 +10:00
Paul Sokolovsky
784909ce16 py/objtype: Handle NotImplemented return from binary special methods.
NotImplemented means "try other fallbacks (like calling __rop__
instead of __op__) and if nothing works, raise TypeError". As
MicroPython doesn't implement any fallbacks, signal to raise
TypeError right away.
2017-08-30 01:39:24 +03:00
Damien George
58321dd985 all: Convert mp_uint_t to mp_unary_op_t/mp_binary_op_t where appropriate
The unary-op/binary-op enums are already defined, and there are no
arithmetic tricks used with these types, so it makes sense to use the
correct enum type for arguments that take these values.  It also reduces
code size quite a bit for nan-boxing builds.
2017-08-29 13:16:30 +10:00
Damien George
be8e5744e6 py/nlrx86,x64: Replace #define of defined() with portable macro usage.
Using gcc -Wpedantic will warn that #define of defined() is non-portable
and this patch fixes this.
2017-08-29 12:52:18 +10:00
Paul Sokolovsky
37379a2974 py/objstr: startswith, endswith: Check arg to be a string.
Otherwise, it will silently get incorrect result on other values types,
including CPython tuple form like "foo.png".endswith(("png", "jpg"))
(which MicroPython doesn't support for unbloatedness).
2017-08-29 00:06:21 +03:00
Dan Halbert
c679c80c71 Modernize module and class static dicts; update freetouch 2017-08-27 15:02:50 -04:00
Dan Halbert
ef61b5ecb5 Initial merge of micropython v1.9.2 into circuitpython 2.0.0 (in development) master.
cpx build compiles and loads and works in repl; test suite not run yet
esp8266 not tested yet
2017-08-25 22:17:07 -04:00
Dan Halbert
74cfdeb316 Added ability to freeze multiple directories; freeze neopixel library in cpx build (#199)
Reworked frozen module support: clean up makefiles and handle multiple directories.
Modules to freeze are included as git submodules.

Add neopixel to circuitplayground express build.

Fixes #56
2017-08-23 11:05:59 -07:00
Damien George
f9ecaa132f py/asmthumb: Use existing macro to properly clear the D-cache.
This macro is provided by stmhal/mphalport.h and makes sure the addr and
size arguments are correctly aligned.
2017-08-23 11:32:27 +10:00
Damien George
f5309fc4ff py/formatfloat: Don't post-increment variable that won't be used again. 2017-08-21 22:04:23 +10:00
Damien George
103ae43f95 py/objcomplex: Remove unnecessary assignment of variable. 2017-08-21 22:03:27 +10:00
Damien George
ad6aae13a4 py/compile: Remove unused pn_colon code when compiling func params. 2017-08-21 22:00:34 +10:00
Damien George
b16a755a0b py/mkrules.mk: Use "find -path" when searching for frozen obj files.
This allows the command to succeed without error even if there is no
$(BUILD)/build directory, which is the case for mpy-cross.
2017-08-21 20:32:30 +10:00
Paul Sokolovsky
e3383e9352 py/stream: seek: Consistently handle negative offset for SEEK_SET.
Per POSIX, this is EINVAL, so raises OSError(EINVAL).
2017-08-20 22:02:41 +03:00
Paul Sokolovsky
0cd9ab7755 py/objstringio: Fix regression with handling SEEK_SET.
For SEEK_SET, offset should be treated as unsigned, to allow full-width
stream sizes (e.g. 32-bit instead of 31-bit). This is now fully documented
in stream.h. Also, seek symbolic constants are added.
2017-08-20 22:02:41 +03:00
Tom Collins
168350cd98 py/objstringio: Prevent offset wraparound for io.BytesIO objects.
Too big positive, or too big negative offset values could lead to overflow
and address space wraparound and thus access to unrelated areas of memory
(a security issue).
2017-08-20 22:00:05 +03:00
Damien George
025e5f2b33 py/binary: Change internal bytearray typecode from 0 to 1.
The value of 0 can't be used because otherwise mp_binary_get_size will let
a null byte through as the type code (intepreted as byterray).  This can
lead to invalid type-specifier strings being let through without an error
in the struct module, and even buffer overruns.
2017-08-17 16:19:35 +10:00
Stefan Naumann
ace9fb5405 py: Add verbose debug compile-time flag MICROPY_DEBUG_VERBOSE.
It enables all the DEBUG_printf outputs in the py/ source code.
2017-08-15 11:53:36 +10:00
Bas van Sisseren
a14ce77b28 py/binary.c: Fix bug when packing big-endian 'Q' values.
Without bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x10\x00\x00\x00\x00'

With bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x00\x00\x00\x00\x10'
2017-08-15 11:33:43 +10:00
Javier Candeira
35a1fea90b all: Raise exceptions via mp_raise_XXX
- Changed: ValueError, TypeError, NotImplementedError
  - OSError invocations unchanged, because the corresponding utility
    function takes ints, not strings like the long form invocation.
  - OverflowError, IndexError and RuntimeError etc. not changed for now
    until we decide whether to add new utility functions.
2017-08-13 22:52:33 +10:00
Scott Shawcroft
fab634e3ee Turn on Rosie CI testing to test new builds on real hardware.
This introduces a skip_if module that can be used by tests to
determine when they should be skipped due to the environment.

Some tests have been split in order to have finer grained skip
control.
2017-08-11 17:16:13 -07:00
Paul Sokolovsky
bfc2092dc5 py/modsys: Initial implementation of sys.getsizeof().
Implemented as a new MP_UNARY_OP. This patch adds support lists, dicts and
instances.
2017-08-11 09:43:07 +03:00
Damien George
7d4a2f773c all: Make use of $(TOP) variable in Makefiles, instead of "..".
$(TOP) is defined in py/mkenv.mk and should be used to refer to the top
level of this repository.
2017-08-11 12:22:19 +10:00
Scott Shawcroft
f6a702538a py: Pretend frozen files are stored under .frozen rather than the empty path.
This makes it clear when frozen modules are loaded as opposed to the empty path
which represents the current working directory. Furthermore, by splitting the two
apart this allows one to control in what order frozen modules are loaded.

This is a prerequisite for #56.
2017-08-10 15:46:17 -07:00
Scott Shawcroft
5c32a5aa6a py: Remove mpy-cross depend when making frozen mpy's because we don't know the state under which its being built. Instead, build it before running the subsequent make. 2017-08-10 15:44:47 -07:00
Damien George
3d25d9c7d9 py/objstr: Raise an exception for wrong type on RHS of str binary op.
The main case to catch is invalid types for the containment operator, of
the form str.__contains__(non-str).
2017-08-09 21:25:48 +10:00
Damien George
eb2784e8a2 py/objtuple: Allow to use inplace-multiplication operator on tuples. 2017-08-09 21:20:42 +10:00
Scott Shawcroft
6ace744667 esp8266: Introduce multiterminal module for managing a secondary
serial connection such as WebREPL.

Fixes #181.
2017-08-07 14:30:11 -07:00
Paul Sokolovsky
4dc7c5649b py/mkrules.mk: Show frozen modules sizes together with executable size.
This works for Unix and similar ports so far.
2017-08-06 15:43:21 +03:00
Dan Halbert
5d509ecace Allow max stack checking to be used with -flto build by determining top
of stack in a different way.
2017-08-05 17:48:16 -07:00