Commit Graph

911 Commits

Author SHA1 Message Date
Damien George 82c494a97e py/vm: Fix handling of unwind jump out of active finally.
Prior to this commit, when unwinding through an active finally the stack
was not being correctly popped/folded, which resulting in the VM crashing
for complicated unwinding of nested finallys.

This should be fixed with this commit, and more tests for return/break/
continue within a finally have been added to exercise this.
2019-10-04 23:01:29 +10:00
Damien George 4102320e90 tests/basics: Add test for getting name of func with closed over locals.
Tests correct decoding of the prelude to get the function name.
2019-10-01 12:26:22 +10:00
Damien George fe4e1fe4b9 tests/basics: Add test for matmul operator.
This is a Python 3.5 feature so the .exp file is needed.
2019-09-26 15:15:34 +10:00
Damien George b29fae0c56 py/bc: Fix size calculation of UNWIND_JUMP opcode in mp_opcode_format.
Prior to this patch mp_opcode_format would calculate the incorrect size of
the MP_BC_UNWIND_JUMP opcode, missing the additional byte.  But, because
opcodes below 0x10 are unused and treated as bytes in the .mpy load/save
and freezing code, this bug did not show any symptoms, since nested unwind
jumps would rarely (if ever) reach a depth of 16 (so the extra byte of this
opcode would be between 0x01 and 0x0f and be correctly loaded/saved/frozen
simply as an undefined opcode).

This patch fixes this bug by correctly accounting for the additional byte.
        .
2019-09-02 13:30:16 +10:00
Damien George 24c3e9b283 py/modstruct: Fix struct.pack_into with unaligned offset of native type.
Following the same fix for unpack.
2019-09-02 13:14:16 +10:00
Tom McDermott 1022f9cc35 py/modstruct: Fix struct.unpack with unaligned offset of native type.
With this patch alignment is done relative to the start of the buffer that
is being unpacked, not the raw pointer value, as per CPython.

Fixes issue #3314.
2019-09-02 13:10:55 +10:00
Damien George acfbb9febd py/objarray: Fix amount of free space in array when doing slice assign.
Prior to this patch the amount of free space in an array (including
bytearray) was not being maintained correctly for the case of slice
assignment which changed the size of the array.  Under certain cases (as
encoded in the new test) it was possible that the array could grow beyond
its allocated memory block and corrupt the heap.

Fixes issue #4127.
2019-08-15 23:02:04 +10:00
Damien George 48f43b77aa tests: Add tests for overriding builtins.__import__. 2019-07-31 22:37:44 +10:00
stijn fb54736bdb py/objarray: Add decode method to bytearray.
Reuse the implementation for bytes since it works the same way regardless
of the underlying type.  This method gets added for CPython compatibility
of bytearray, but to keep the code simple and small array.array now also
has a working decode method, which is non-standard but doesn't hurt.
2019-05-21 14:24:04 +10:00
Damien George a474ddf959 tests/basics: Add coverage tests for memoryview attributes. 2019-05-14 17:22:49 +10:00
stijn 90fae9172a py/objarray: Add support for memoryview.itemsize attribute.
This allows figuring out the number of bytes in the memoryview object as
len(memview) * memview.itemsize.

The feature is enabled via MICROPY_PY_BUILTINS_MEMORYVIEW_ITEMSIZE and is
disabled by default.
2019-05-14 17:15:17 +10:00
Dan Halbert 8664a6574b use approx of original @godlygeek code for smallints; add tests 2019-05-12 11:17:29 -04:00
Matt Wozniski e041df73bb Add tests for overflows converting ints to bytes 2019-05-09 03:22:24 -04:00
Damien George dac9d47671 py/objgenerator: Fix handling of None passed as 2nd arg to throw().
Fixes issue #4527.
2019-05-09 13:40:28 +10:00
Damien George c2bb451908 tests/basics/sys1.py: Add test for calling sys.exit() without any args. 2019-05-03 23:21:08 +10:00
Damien George ca39ea7cef tests: Skip tests needing machine module if (u)machine doesn't exist. 2019-04-28 22:12:17 +10:00
Damien George e1fb03f3e2 py: Fix VM crash with unwinding jump out of a finally block.
This patch fixes a bug in the VM when breaking within a try-finally.  The
bug has to do with executing a break within the finally block of a
try-finally statement.  For example:

    def f():
        for x in (1,):
            print('a', x)
            try:
                raise Exception
            finally:
                print(1)
                break
            print('b', x)
    f()

Currently in uPy the above code will print:

    a 1
    1
    1
    segmentation fault (core dumped)  micropython

Not only is there a seg fault, but the "1" in the finally block is printed
twice.  This is because when the VM executes a finally block it doesn't
really know if that block was executed due to a fall-through of the try (no
exception raised), or because an exception is active.  In particular, for
nested finallys the VM has no idea which of the nested ones have active
exceptions and which are just fall-throughs.  So when a break (or continue)
is executed it tries to unwind all of the finallys, when in fact only some
may be active.

It's questionable whether break (or return or continue) should be allowed
within a finally block, because they implicitly swallow any active
exception, but nevertheless it's allowed by CPython (although almost never
used in the standard library).  And uPy should at least not crash in such a
case.

The solution here relies on the fact that exception and finally handlers
always appear in the bytecode after the try body.

Note: there was a similar bug with a return in a finally block, but that
was previously fixed in b735208403
2019-03-05 16:05:05 +11:00
Damien George 12ce9f2689 py/compile: Fix handling of unwinding BaseException in async with.
All exceptions that unwind through the async-with must be caught and
BaseException is the top-level class, which includes Exception and others.

Fixes issue #4552.
2019-02-26 23:52:10 +11:00
Damien George be41d6d6f9 tests/basics: Add tests for try-except-else and try-except-else-finally. 2019-02-21 16:22:41 +11:00
stijn 42863830be py: Add optional support for 2-argument version of built-in next().
Configurable via MICROPY_PY_BUILTINS_NEXT2, disabled by default.
2019-01-27 13:01:28 +11:00
Paul Sokolovsky d4d4bc5827 tests/basics/special_methods2: Typo fix in comment. 2018-12-13 01:29:01 +11:00
Paul Sokolovsky d690c2e148 tests/basics/special_methods: Add testcases for __int__. 2018-12-07 17:28:04 +11:00
Damien George 113f00a9ab py/objboundmeth: Support loading generic attrs from the method.
Instead of assuming that the method is a bytecode object, and only
supporting load of __name__, make the operation generic by delegating the
load to the method object itself.  Saves a bit of code size and fixes the
case of attempting to load __name__ on a native method, see issue #4028.
2018-12-06 18:02:41 +11:00
Damien George 9201f46cc8 py/compile: Fix case of eager implicit conversion of local to nonlocal.
This ensures that implicit variables are only converted to implicit
closed-over variables (nonlocals) at the very end of the function scope.
If variables are closed-over when first used (read from, as was done prior
to this commit) then this can be incorrect because the variable may be
assigned to later on in the function which means they are just a plain
local, not closed over.

Fixes issue #4272.
2018-10-28 00:33:08 +11:00
Paul Sokolovsky a527313382 tests: Make bytes/str.count() tests skippable. 2018-10-22 22:50:28 +11:00
Damien George a07e56cbd8 tests/basics/class_getattr: Remove invalid test for __getattribute__.
Part of this test was trying to test some functionality of __getattribute__
but this method name was misspelt so it wasn't doing anything useful.
Fixing the typo in this name makes the test fail because MicroPython
doesn't support user defined __getattribute__ methods.  So this part of the
test is removed.  The remaining tests are modified slightly to make it
clearer what they are testing.
2018-10-18 12:28:09 +11:00
Damien George 7eb29c2000 py/objtype: Remove comment about catching exc from user __getattr__.
Any exception raised in a user __getattr__ should be propagated out.  A
test is added to verify these semantics.
2018-10-18 12:15:16 +11:00
Damien George dd288904db py/objtype: Support full object model for get/set/delitem special meths.
This makes these special methods have the same calling behaviour as other
methods in a class instance (mp_convert_member_lookup() is already called
by mp_obj_class_lookup()).
2018-09-28 23:22:34 +10:00
Damien George 0c9d452370 py/vm: Fix case of throwing GeneratorExit type into yield-from.
mp_make_raise_obj must be used to convert a possible exception type to an
instance object, otherwise the VM may raise a non-exception object.

An existing test is adjusted to test this case, with the original test
already moved to generator_throw.py.
2018-09-28 11:39:35 +10:00
Damien George e6078dfed2 tests/basics: Split out gen throw tests from yield-from-throw tests. 2018-09-28 11:35:31 +10:00
Damien George fc1bb51af5 py/objgenerator: Remove TODO about returning gen being called again.
The code implements correct behaviour, as tested by the new test case added
in this commit.
2018-09-27 15:18:24 +10:00
Damien George 3f6ffe059f py/objgenerator: Implement PEP479, StopIteration convs to RuntimeError.
This commit implements PEP479 which disallows raising StopIteration inside
a generator to signal that it should be finished.  Instead, the generator
should simply return when it is complete.

See https://www.python.org/dev/peps/pep-0479/ for details.
2018-09-20 15:36:59 +10:00
Damien George f2de9d60f7 py/emitnative: Fix try-finally in outer scope, so finally is cancelled. 2018-09-11 15:33:25 +10:00
Paul Sokolovsky 674e069ba9 py/objarray: bytearray: Allow 2nd/3rd arg to constructor.
If bytearray is constructed from str, a second argument of encoding is
required (in CPython), and third arg of Unicode error handling is allowed,
e.g.:

bytearray("str", "utf-8", "strict")

This is similar to bytes:

bytes("str", "utf-8", "strict")

This patch just allows to pass 2nd/3rd arguments to bytearray, but
doesn't try to validate them to not impact code size. (This is also
similar to how bytes constructor is handled, though it does a bit
more validation, e.g. check that in case of str arg, encoding argument
is passed.)
2018-09-11 15:10:10 +10:00
Damien George 4970e9bc8c tests/basics: Add test cases for context manager raising in enter/exit. 2018-09-04 14:37:30 +10:00
Damien George b14c705c18 tests/basics: Add more tests for return within try-finally. 2018-09-04 14:37:07 +10:00
Damien George 3cd2c281d7 py/emitnative: Cancel caught exception once handled to prevent reraise.
The native emitter keeps the current exception in a slot in its C stack
(instead of on its Python value stack), so when it catches an exception it
must explicitly clear that slot so the same exception is not reraised later
on.
2018-09-03 17:41:02 +10:00
Damien George b735208403 py/vm: Fix handling of finally-return with complex nested finallys.
Back in 8047340d75 basic support was added in
the VM to handle return statements within a finally block.  But it didn't
cover all cases, in particular when some finally's were active and others
inactive when the "return" was executed.

This patch adds further support for return-within-finally by correctly
managing the currently_in_except_block flag, and should fix all cases.  The
main point is that finally handlers remain on the exception stack even if
they are active (currently being executed), and the unwind return code
should only execute those finally's which are inactive.

New tests are added for the cases which now pass.
2018-09-03 13:08:16 +10:00
Damien George 828f771e32 tests/basics: Provide .exp files for generator tests that fail PEP479.
PEP479 (see https://www.python.org/dev/peps/pep-0479/) prohibited raising
StopIteration from within a generator (it is turned into a RuntimeError).
This behaviour was introduced in Python 3.5 and in 3.7 was made compulsory.
Until uPy implements PEP479, this patch adds .py.exp files for the relevant
tests so they can be run under Python 3.7.
2018-08-17 15:50:21 +10:00
Damien George 8979ce1671 tests: Modify tests that print repr of an exception with 1 arg.
In Python 3.7 the behaviour of repr() of an exception with one argument
changed: it no longer prints a trailing comma in the argument list.  See
https://bugs.python.org/issue30399

This patch modifies tests that rely on this behaviour to not rely on it.
And the python34.py test is updated to include a test for this behaviour
with a .exp file.
2018-08-17 15:46:04 +10:00
Damien George 0988b14cd6 tests/basics/int_big_error.py: Use bytearray to test for int overflow.
In Python 3.7 "1 >> (big int)" is now allowed, it no longer raises an
OverflowError.  So use bytearray to test big-int conversion overflow.
2018-08-17 15:43:47 +10:00
Damien George 96e1fd480d tests/basics/set_pop.py: Sort set before printing for consistent output. 2018-08-17 15:42:51 +10:00
Ayke van Laethem 6572029dc0 tests: Make tests work on targets without float support. 2018-08-04 15:14:23 +10:00
Dan Halbert 7c219600a2 WIP: after merge; before testing 2018-07-11 16:45:30 -04:00
Damien George e2e22e3d7e py/objgenerator: Implement __name__ with normal fun attr accessor code.
With the recent change b488a4a848, a
generating function now has the same layout in memory as a normal bytecode
function, and so can reuse the latter's attribute accessor code to
implement __name__.
2018-07-10 16:33:57 +10:00
Damien George d8dc918deb py/compile: Handle return/break/continue correctly in async with.
Before this patch the context manager's __aexit__() method would not be
executed if a return/break/continue statement was used to exit an async
with block.  async with now has the same semantics as normal with.

The fix here applies purely to the compiler, and does not modify the
runtime at all. It might (eventually) be better to define new bytecode(s)
to handle async with (and maybe other async constructs) in a cleaner, more
efficient way.

One minor drawback with addressing this issue purely in the compiler is
that it wasn't possible to get 100% CPython semantics.  The thing that is
different here to CPython is that the __aexit__ method is not looked up in
the context manager until it is needed, which is after the body of the
async with statement has executed.  So if a context manager doesn't have
__aexit__ then CPython raises an exception before the async with is
executed, whereas uPy will raise it after it is executed.  Note that
__aenter__ is looked up at the beginning in uPy because it needs to be
called straightaway, so if the context manager isn't a context manager then
it'll still raise an exception at the same location as CPython.  The only
difference is if the context manager has the __aenter__ method but not the
__aexit__ method, then in that case uPy has different behaviour.  But this
is a very minor, and acceptable, difference.
2018-06-27 16:57:42 +10:00
Damien George 726804ea40 tests: Move non-filesystem io tests to basics dir with io_ prefix. 2018-06-27 16:55:05 +10:00
Paul Sokolovsky bdceea1d12 tests/basics/namedtuple*: Import ucollections first.
Otherwise, test may have artefacts in the presence of the micropython-lib
module.
2018-06-27 14:58:14 +10:00
Damien George 6a445b60fa py/lexer: Add support for underscores in numeric literals.
This is a very convenient feature introduced in Python 3.6 by PEP 515.
2018-06-12 12:17:43 +10:00
Damien George 36c1052183 py/objtype: Optimise instance get/set/del by skipping special accessors.
This patch is a code optimisation, trading text bytes for speed.  On
pyboard it's an increase of 0.06% in code size for a gain (in pystone
performance) of roughly 6.5%.

The patch optimises load/store/delete of attributes in user defined classes
by not looking up special accessors (@property, __get__, __delete__,
__set__, __setattr__ and __getattr_) if they are guaranteed not to exist in
the class.

Currently, if you do my_obj.foo() then the runtime has to do a few checks
to see if foo is a property or has __get__, and if so delegate the call.
And for stores things like my_obj.foo = 1 has to first check if foo is a
property or has __set__ defined on it.

Doing all those checks each and every time the attribute is accessed has a
performance penalty.  This patch eliminates all those checks for cases when
it's guaranteed that the checks will always fail, ie no attributes are
properties nor have any special accessor methods defined on them.

To make this guarantee it checks all attributes of a user-defined class
when it is first created.  If any of the attributes of the user class are
properties or have special accessors, or any of the base classes of the
user class have them, then it sets a flag in the class to indicate that
special accessors must be checked for.  Then in the load/store/delete code
it checks this flag to see if it can take the shortcut and optimise the
lookup.

It's an optimisation that's pretty widely applicable because it improves
lookup performance for all methods of user defined classes, and stores of
attributes, at least for those that don't have special accessors.  And, it
allows to enable descriptors with minimal additional runtime overhead if
they are not used for a particular user class.

There is one restriction on dynamic class creation that has been introduced
by this patch: a user-defined class cannot go from zero special accessors
to one special accessor (or more) after that class has been subclassed.  If
the script attempts this an AttributeError is raised (see addition to
tests/misc/non_compliant.py for an example of this case).

The cost in code space bytes for the optimisation in this patch is:

   unix x64:  +528
unix nanbox:  +508
      stm32:  +192
     cc3200:  +200
    esp8266:  +332
      esp32:  +244

Performance tests that were done:

- on unix x86-64, pystone improved by about 5%
- on pyboard, pystone improved by about 6.5%, from 1683 up to 1794
- on pyboard, bm_chaos (from CPython benchmark suite) improved by about 5%
- on esp32, pystone improved by about 30% (but there are caching effects)
- on esp32, bm_chaos improved by about 11%
2018-06-08 12:12:08 +10:00
Jeff Epler c60589c02b py/objtype: Fix assertion failures in super_attr by checking type.
Fixes assertion failures and segmentation faults when making calls like:

    super(1, 1).x
2018-05-30 11:14:07 +10:00
Jeff Epler 05b13fd292 py/objtype: Fix assertion failures in mp_obj_new_type by checking types.
Fixes assertion failures when the arguments to type() were not of valid
types, e.g., when making calls like:

    type("", (), 3)
    type("", 3, {})
2018-05-30 11:11:24 +10:00
Damien George dfeaea1441 py/objtype: Remove TODO comment about needing to check for property.
Instance members are always treated as values, even if they are properties.
A test is added to show this is the case.
2018-05-25 10:59:40 +10:00
Damien George 400273a799 py/objgenerator: Protect against reentering a generator.
Generators that are already executing cannot be reexecuted.  This patch
puts in a check for such a case.

Thanks to @jepler for finding the bug.
2018-05-22 16:54:03 +10:00
Jan Klusacek b318ebf101 py/modbuiltins: Add support for rounding integers.
As per CPython semantics.  This feature is controlled by
MICROPY_PY_BUILTINS_ROUND_INT which is disabled by default.
2018-05-22 14:18:16 +10:00
Damien George 1ad0013dec tests: Add some tests for bigint hash, float hash and float parsing.
Following outcome of recent fuzz testing and sanitizing by @jepler.
2018-05-21 13:05:40 +10:00
Dan Halbert 54293397c5
Merge pull request #837 from godlygeek/human_readable_oserror
Human readable OSError messages
2018-05-15 10:12:09 -04:00
Matt Wozniski 086bffb594 Update errno tests to match new expected behavior
Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
2018-05-14 23:16:18 -04:00
bildzeitung bf26ffbf56 Updated tests; removed try/catch for ucollections 2018-05-14 14:43:34 -04:00
Damien George 7541be5637 tests/basics/special_methods2: Enable some additional tests that work.
These special methods are all available if MICROPY_PY_ALL_SPECIAL_METHODS
is enabled.
2018-05-11 17:37:16 +10:00
Damien George 3678a6bdc6 py/modbuiltins: Make built-in dir support the __dir__ special method.
If MICROPY_PY_ALL_SPECIAL_METHODS is enabled then dir() will now delegate
to the special method __dir__ if the object it is listing has this method.
2018-05-10 23:14:23 +10:00
Damien George 529860643b py/modbuiltins: Make built-in hasattr work properly for user types.
It now allows __getattr__ in a user type to raise AttributeError when the
attribute does not exist.
2018-05-10 23:03:30 +10:00
Jeff Epler d6cf5c6749 py/objstr: In find/rfind, don't crash when end < start. 2018-04-05 16:14:17 +10:00
Damien George 1bfc774a08 tests/basics/string_compare.py: Add test with string that hashes to 0.
The string "Q+?" is special in that it hashes to zero with the djb2
algorithm (among other strings), and a zero hash should be incremented to a
hash of 1.
2018-04-05 01:04:38 +10:00
Damien George 22161acf47 tests/basics/class_super.py: Add tests for store/delete of super attr. 2018-04-05 01:03:57 +10:00
Damien George 7b7bbd0ee7 tests/basics: Add tests for edge cases of nan-box's 47-bit small int. 2018-04-05 00:59:49 +10:00
Damien George dd48ccb1e3 tests/basics: Add test for subclassing an iterable native type. 2018-04-04 15:26:18 +10:00
Damien George df02f5620a tests/basics/int_big1.py: Add test for big int in mp_obj_get_int_maybe. 2018-04-04 15:23:32 +10:00
Damien George 7d5c753b17 tests/basics: Modify int-big tests to prevent constant folding.
So that these tests test the runtime behaviour, not the compiler (which may
be executed offline).
2018-04-04 13:57:22 +10:00
Damien George f684e9e1ab tests/basics/int_big1.py: Add test converting str with non-print chars. 2018-04-04 13:56:00 +10:00
Damien George 430efb0444 tests/basics: Add test for use of return within try-except.
The case of a return statement in the try suite of a try-except statement
was previously only tested by builtin_compile.py, and only then in the part
of this test which checked for the existence of the compile builtin.  So
this patch adds an explicit unit test for this case.
2018-04-04 01:43:16 +10:00
Scott Shawcroft e4ae1e3d59
Merge pull request #734 from jepler/str-find-backwards-circuitpython
py/objstr: Don't crash when end < start
2018-04-01 22:41:59 -07:00
Jeff Epler a909007fef py/objgenerator: Check stack before resuming a generator
This turns a hard crash in a recursive generator into
a 'maximum recursion depth exceeded' exception.
2018-04-01 16:40:15 -05:00
Jeff Epler 0041df0c6b py/objstr: Don't crash when end < start
.. and add testcases for the same.

(crash found by afl-fuzz)
2018-03-31 22:17:11 -05:00
Scott Shawcroft 3215b85568
Merge pull request #728 from jepler/double-splat-crash-circuitpython
py/bc: Turn assertion error into exception
2018-03-31 09:57:25 -07:00
Jeff Epler 853f7ac4d0 py/bc: Turn assertion error into exception 2018-03-31 08:44:29 -05:00
Damien George bcfff4fc98 tests/basics/iter1.py: Add more tests for walking a user-defined iter.
Some code in mp_iternext() was only tested by the native emitter, so the
tests added here test this function using just the bytecode emitter.
2018-03-30 14:23:13 +11:00
Damien George f50b64cab5 py/runtime: Be sure that non-intercepted thrown object is an exception.
The VM expects that, if mp_resume() returns MP_VM_RETURN_EXCEPTION, then
the returned value is an exception instance (eg to add a traceback to it).
It's possible that a value passed to a generator's throw() is not an
exception so must be explicitly checked for if the thrown value is not
intercepted by the generator.

Thanks to @jepler for finding the bug.
2018-03-30 12:43:38 +11:00
Damien George 3280788195 py/runtime: Check that keys in dicts passed as ** args are strings.
Prior to this patch the code would crash if a key in a ** dict was anything
other than a str or qstr.  This is because mp_setup_code_state() assumes
that keys in kwargs are qstrs (for efficiency).

Thanks to @jepler for finding the bug.
2018-03-30 11:13:32 +11:00
Jeff Epler ff06a45599 Fix assertion failures in super_attr
micropython: ../../py/objtype.c:1100: super_attr: Assertion `MP_OBJ_IS_TYPE(self->type, &mp_type_type)' failed.

e.g., when making calls like
    super(1, 1).x
2018-03-29 06:42:10 -05:00
Jeff Epler 6da8d7c465 Fix assertion failures in mp_obj_new_type
Fixes the following assertion failures when the arguments to type()
were not of valid types:
micropython: ../../py/objtype.c:984: mp_obj_new_type: Assertion `MP_OBJ_IS_TYPE(bases_tuple, &mp_type_tuple)' failed.
micropython: ../../py/objtype.c:994: mp_obj_new_type: Assertion `MP_OBJ_IS_TYPE(items[i], &mp_type_type)' failed.

e.g., when making calls like
    type("", (), 3)
    type("", 3, {})
2018-03-29 06:42:10 -05:00
Jeff Epler a55988a547 Fix assertion failure in mpz_divmod_inpl
.. turning this from an assertion failure into an exception:
    pow(1,1,0)
2018-03-29 06:42:10 -05:00
Scott Shawcroft 4517ab8ba4
Merge pull request #709 from jepler/core-class-superproperty
Make test core_class_superproperty.py succeed
2018-03-26 21:59:05 -07:00
Jeff Epler 047a4f59c5 This test now passes, make it run regularly 2018-03-26 18:47:06 -05:00
Jeff Epler 355bf8b553 Conditionally compile out nonstandard array/struct typecodes
.. defaulting to off for circuitpython-supported boards, on for others.

.. fixing up the tests that fail when it is turned off, so that they skip
instead of failing
2018-03-26 18:13:49 -05:00
Jeff Epler cdb83b18ec Implement * and *= for array.array 2018-03-23 07:37:07 -05:00
Damien George 72adc381fb tests/basics/builtin_enumerate: Add test for many pos args to enumerate. 2018-03-08 12:51:06 +11:00
Damien George 439acddc60 tests/basics/gc1: Add test which triggers GC threshold. 2018-02-27 22:39:17 +11:00
Damien George 22ade2f5c4 py/vm: Fix case of handling raised StopIteration within yield from.
This patch concerns the handling of an NLR-raised StopIteration, raised
during a call to mp_resume() which is handling the yield from opcode.

Previously, commit 6738c1dded introduced code
to handle this case, along with a test.  It seems that it was lucky that
the test worked because the code did not correctly handle the stack pointer
(sp).

Furthermore, commit 79d996a57b improved the
way mp_resume() propagated certain exceptions: it changed raising an NLR
value to returning MP_VM_RETURN_EXCEPTION.  This change meant that the
test introduced in gen_yield_from_ducktype.py was no longer hitting the
code introduced in 6738c1dded.

The patch here does two things:

1. Fixes the handling of sp in the VM for the case that yield from is
   interrupted by a StopIteration raised via NLR.

2. Introduces a new test to check this handling of sp and re-covers the
   code in the VM.
2018-02-27 15:39:31 +11:00
Damien George 90da791a08 tests/basics: Add test for calling a subclass of a native class.
Adding this test gets py/objtype.c to 100% coverage.
2018-02-24 23:13:42 +11:00
Damien George 160d670868 py/objdeque: Protect against negative maxlen in deque constructor.
Otherwise passing -1 as maxlen will lead to a zero allocation and
subsequent unbound buffer overflow in deque.append() because i_put is
allowed to grow without bound.
2018-02-21 23:34:17 +11:00
Damien George 8f9b113be2 tests/basics: Add tests to improve coverage of py/objdeque.c. 2018-02-21 23:19:06 +11:00
Paul Sokolovsky 4668ec801e tests/basics/deque*: Tests for ucollections.deque. 2018-02-21 22:58:14 +11:00
Damien George 4e469085c1 py/objstr: Protect against creating bytes(n) with n negative.
Prior to this patch uPy (on a 32-bit arch) would have severe issues when
calling bytes(-1): such a call would call vstr_init_len(vstr, -1) which
would then +1 on the len and call vstr_init(vstr, 0), which would then
round this up and allocate a small amount of memory for the vstr.  The
bytes constructor would then attempt to zero out all this memory, thinking
it had allocated 2^32-1 bytes.
2018-02-19 16:25:30 +11:00
Damien George 98647e83c7 py/modbuiltins: Simplify and generalise dir() by probing qstrs.
This patch improves the builtin dir() function by probing the target object
with all possible qstrs via mp_load_method_maybe.  This is very simple (in
terms of implementation), doesn't require recursion, and allows to list all
methods of user-defined classes (without duplicates) even if they have
multiple inheritance with a common parent.  The downside is that it can be
slow because it has to iterate through all the qstrs in the system, but
the "dir()" function is anyway mostly used for testing frameworks and user
introspection of types, so speed is not considered a priority.

In addition to providing a more complete implementation of dir(), this
patch is simpler than the previous implementation and saves some code
space:

   bare-arm:   -80
minimal x86:   -80
   unix x64:   -56
unix nanbox:   -48
      stm32:   -80
     cc3200:   -80
    esp8266:  -104
      esp32:   -64
2018-02-19 16:12:44 +11:00
Mike Wadsten a3e01d3642 py/objdict: Disallow possible modifications to fixed dicts. 2018-02-18 21:51:04 -06:00
Damien George d77da83d55 py/objrange: Implement (in)equality comparison between range objects.
This feature is not often used so is guarded by the config option
MICROPY_PY_BUILTINS_RANGE_BINOP which is disabled by default.  With this
option disabled MicroPython will always return false when comparing two
range objects for equality (unless they are exactly the same object
instance).  This does not match CPython so if (in)equality between range
objects is needed then this option should be enabled.

Enabling this option costs between 100 and 200 bytes of code space
depending on the machine architecture.
2018-02-14 23:17:06 +11:00
Damien George 04c55f5828 tests: Rewrite some tests so they can run without needing eval/exec.
For builds without the compiler enabled (and hence without eval/exec) it is
useful to still be able to run as many tests as possible.
2018-02-14 16:50:20 +11:00
Damien George 6031957473 tests: Automatically skip tests that require eval, exec or frozenset. 2018-02-14 16:46:44 +11:00
Damien George 1f53ff61ff tests/basics: Rename remaining tests that are for built-in functions.
For consistency with all of the other tests that are named builtin_XXX.py.
2018-02-07 15:55:52 +11:00
Damien George b45c8c17f0 py/objtype: Check and prevent delete/store on a fixed locals map.
Note that the check for elem!=NULL is removed for the
MP_MAP_LOOKUP_ADD_IF_NOT_FOUND case because mp_map_lookup will always
return non-NULL for such a case.
2018-02-07 15:44:29 +11:00
Damien George a1d85d6199 tests/basics/memoryerror: Add test for out-of-memory using realloc. 2017-12-20 16:58:27 +11:00
Damien George 35a759dc1d tests: Add some more tests to improve coverage of py/parse.c. 2017-12-19 16:13:00 +11:00
Damien George 2bfa531798 tests/basics/builtin_pow3: Add tests for edge cases of pow3. 2017-12-19 15:44:10 +11:00
Damien George 8e6113a188 tests/basics/generator_pend_throw: Add test for just-started generator. 2017-12-19 15:02:34 +11:00
Damien George 7208cad97a tests/basics: Add more set tests to improve coverage of py/objset.c. 2017-12-19 13:59:54 +11:00
Paul Sokolovsky 6364401666 py/objgenerator: Allow to pend an exception for next execution.
This implements .pend_throw(exc) method, which sets up an exception to be
triggered on the next call to generator's .__next__() or .send() method.
This is unlike .throw(), which immediately starts to execute the generator
to process the exception. This effectively adds Future-like capabilities
to generator protocol (exception will be raised in the future).

The need for such a method arised to implement uasyncio wait_for() function
efficiently (its behavior is clearly "Future" like, and normally would
require to introduce an expensive Future wrapper around all native
couroutines, like upstream asyncio does).

py/objgenerator: pend_throw: Return previous pended value.

This effectively allows to store an additional value (not necessary an
exception) in a coroutine while it's not being executed. uasyncio has
exactly this usecase: to mark a coro waiting in I/O queue (and thus
not executed in the normal scheduling queue), for the purpose of
implementing wait_for() function (cancellation of such waiting coro
by a timeout).
2017-12-15 20:20:36 +02:00
Damien George 36f79523ab tests: Add tests to improve coverage of py/objtype.c. 2017-12-14 12:25:30 +11:00
Paul Sokolovsky da34b6ef45 tests: Fix few test for proper "skipped" detection with qemu-arm's tinytest.
"Builtin" tinytest-based testsuite as employed by qemu-arm (and now
generalized by me to be reusable for other targets) performs simplified
detection of skipped tests, it treats as such tests which raised SystemExit
(instead of checking got "SKIP" output). Consequently, each "SKIP" must
be accompanied by SystemExit (and conversely, SystemExit should not be
used if test is not skipped, which so far seems to be true).
2017-12-12 23:45:48 +02:00
Damien George fd0b0db873 tests/basics: Add test for overriding a native base-class's init method. 2017-12-12 16:47:38 +11:00
Scott Shawcroft 8a50fe8982 Always test enumerate. This will ensure that all supported ports
include it and therefore support our drivers that use it.

Related to https://github.com/adafruit/Adafruit_CircuitPython_NeoPixel/pull/18
2017-12-11 17:07:28 -08:00
Damien George c3bc8d7b2b tests/basics/builtin_locals: Add test for using locals() in class body. 2017-11-27 14:14:57 +11:00
Damien George c7a0e1472d tests/basics/builtin_range: Add test for corner case of range slicing. 2017-11-24 15:30:12 +11:00
Damien George 505671b698 tests/basics: Add test for containment of a subclass of a native type. 2017-11-24 14:48:41 +11:00
Scott Shawcroft 99153569f4 tests: Fixup tests for 3.x on Rosie 2017-11-16 10:56:20 -08:00
stijn 79ed58f87b py/objnamedtuple: Add _asdict function if OrderedDict is supported 2017-11-12 14:16:54 +02:00
Scott Shawcroft 73c15dcf8b Merge commit 'f869d6b2e339c04469c6c9ea3fb2fabd7bbb2d8c' into nrf2_merge
This is prep for merging in the NRF5 pull request.
2017-10-24 22:31:16 -07:00
Damien George dfa563c71f py/objstr: Make empty bytes object have a null-terminating byte.
Because a lot of string processing functions assume there is a null
terminating byte, so they can work in an efficient way.

Fixes issue #3334.
2017-10-04 17:59:22 +11:00
Damien George 1394258f37 py/objset: Include the failed key in a KeyError raised from set.remove. 2017-10-03 18:03:06 +11:00
Damien George 2ac1364688 py/objset: Check that RHS of a binary op is a set/frozenset.
CPython docs explicitly state that the RHS of a set/frozenset binary op
must be a set to prevent user errors.  It also preserves commutativity of
the ops, eg: "abc" & set() is a TypeError, and so should be set() & "abc".

This change actually decreases unix (x64) code by 160 bytes; it increases
stm32 by 4 bytes and esp8266 by 28 bytes (but previous patch already
introduced a much large saving).
2017-10-03 17:56:27 +11:00
Paul Sokolovsky fc9a6dd09e py/objstr: strip: Don't strip "\0" by default.
An issue was due to incorrectly taking size of default strip characters
set.
2017-09-19 21:21:12 +03:00
Paul Sokolovsky d6f9d64d97 tests/class_reverse_op: Test for reverse arith ops special methods.
This test should be run only if support for reverse ops is enabled, so
the corresponding feature_check is added to run-tests.
2017-09-10 17:05:57 +03:00
Paul Sokolovsky 5c603bd0fd py/objlist: Properly implement comparison with incompatible types.
Should raise TypeError, unless it's (in)equality comparison.
2017-09-07 00:10:10 +03:00
Paul Sokolovsky 1aaba5cabe py/objtuple: Properly implement comparison with incompatible types.
Should raise TypeError, unless it's (in)equality comparison.
2017-09-06 00:23:41 +03:00
Paul Sokolovsky 376618cd8a tests/class_inplace_op: Test for inplace op fallback to normal one. 2017-09-04 16:44:38 +03:00
Damien George 2daacc5cee py/modstruct: Check and prevent buffer-write overflow in struct packing.
Prior to this patch, the size of the buffer given to pack_into() was checked
for being too small by using the count of the arguments, not their actual
size.  For example, a format spec of '4I' would only check that there was 4
bytes available, not 16; and 'I' would check for 1 byte, not 4.

The pack() function is ok because its buffer is created to be exactly the
correct size.

The fix in this patch calculates the total size of the format spec at the
start of pack_into() and verifies that the buffer is large enough.  This
adds some computational overhead, to iterate through the whole format spec.
The alternative is to check during the packing, but that requires extra
code to handle alignment, and the check is anyway not needed for pack().
So to maintain minimal code size the check is done using struct_calcsize.
2017-09-01 11:11:09 +10:00
Damien George 79d5acbd01 py/modstruct: Check and prevent buffer-read overflow in struct unpacking
Prior to this patch, the size of the buffer given to unpack/unpack_from was
checked for being too small by using the count of the arguments, not their
actual size.  For example, a format spec of '4I' would only check that
there was 4 bytes available, not 16; and 'I' would check for 1 byte, not 4.

This bug is fixed in this patch by calculating the total size of the format
spec at the start of the unpacking function.  This function anyway needs to
calculate the number of items at the start, so calculating the total size
can be done at the same time.
2017-09-01 10:53:29 +10:00
Damien George 793d826d9d py/modstruct: In struct.pack, stop converting if there are no args left.
This patch makes a repeat counter behave the same as repeating the
typecode, when there are not enough args.  For example:
struct.pack('2I', 1) now behave the same as struct.pack('II', 1).
2017-09-01 10:10:51 +10:00
Paul Sokolovsky b349479a49 tests/class_new: Add another testcase for __new__/__init__ interaction.
Similar to the existing testcase, but test that returning both value of
native type and instance of another user class from __new__ lead to
__init__ not being called, for better coverage.
2017-09-01 00:43:52 +03:00
Paul Sokolovsky 35be9e805f tests/class_new: Add checks for __init__ being called and other improvements. 2017-08-30 21:33:42 +03:00
Paul Sokolovsky b565c36963 tests/object_new: Better messages, check user __new__() method.
Make messages more verbose and easier to follow and check that user class'
__new__() is not called by object.__new__(user_class).
2017-08-30 21:29:23 +03:00
Paul Sokolovsky 784909ce16 py/objtype: Handle NotImplemented return from binary special methods.
NotImplemented means "try other fallbacks (like calling __rop__
instead of __op__) and if nothing works, raise TypeError". As
MicroPython doesn't implement any fallbacks, signal to raise
TypeError right away.
2017-08-30 01:39:24 +03:00
Paul Sokolovsky 37379a2974 py/objstr: startswith, endswith: Check arg to be a string.
Otherwise, it will silently get incorrect result on other values types,
including CPython tuple form like "foo.png".endswith(("png", "jpg"))
(which MicroPython doesn't support for unbloatedness).
2017-08-29 00:06:21 +03:00
Dan Halbert ef61b5ecb5 Initial merge of micropython v1.9.2 into circuitpython 2.0.0 (in development) master.
cpx build compiles and loads and works in repl; test suite not run yet
esp8266 not tested yet
2017-08-25 22:17:07 -04:00
Damien George 025e5f2b33 py/binary: Change internal bytearray typecode from 0 to 1.
The value of 0 can't be used because otherwise mp_binary_get_size will let
a null byte through as the type code (intepreted as byterray).  This can
lead to invalid type-specifier strings being let through without an error
in the struct module, and even buffer overruns.
2017-08-17 16:19:35 +10:00
Bas van Sisseren a14ce77b28 py/binary.c: Fix bug when packing big-endian 'Q' values.
Without bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x10\x00\x00\x00\x00'

With bugfix:

    struct.pack('>Q', 16)
    b'\x00\x00\x00\x00\x00\x00\x00\x10'
2017-08-15 11:33:43 +10:00
Scott Shawcroft fab634e3ee Turn on Rosie CI testing to test new builds on real hardware.
This introduces a skip_if module that can be used by tests to
determine when they should be skipped due to the environment.

Some tests have been split in order to have finer grained skip
control.
2017-08-11 17:16:13 -07:00
Damien George 3d25d9c7d9 py/objstr: Raise an exception for wrong type on RHS of str binary op.
The main case to catch is invalid types for the containment operator, of
the form str.__contains__(non-str).
2017-08-09 21:25:48 +10:00
Damien George eb2784e8a2 py/objtuple: Allow to use inplace-multiplication operator on tuples. 2017-08-09 21:20:42 +10:00
Alexander Steffen 55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Tom Collins 6cfe737597 tests/basics/builtin_exec: Test various globals/locals args to exec(). 2017-07-21 15:17:33 +10:00
Damien George 6c1b7e008d tests: Rename exec1.py to builtin_exec.py. 2017-07-21 15:11:24 +10:00
Damien George f69ab79ec8 py/objgenerator: Allow to hash generators and generator instances.
Adds nothing to the code size, since it uses existing empty slots in the
type structures.
2017-07-07 11:47:38 +10:00
Krzysztof Blazewicz 7feb7301b2 tests/basics: Add tests for arithmetic operators precedence. 2017-07-05 15:51:03 +10:00
Damien George 8f6ef8de48 tests/basics/namedtuple1: Add test for creating with pos and kw args. 2017-06-29 17:50:09 +10:00
Damien George 44922934f5 tests/basics: Add tests for for-else statement. 2017-06-22 14:02:14 +10:00
Scott Shawcroft 30ee7019ca Merge tag 'v1.9.1'
Fixes for stmhal USB mass storage, lwIP bindings and VFS regressions

This release provides an important fix for the USB mass storage device in
the stmhal port by implementing the SCSI SYNCHRONIZE_CACHE command, which
is now require by some Operating Systems.  There are also fixes for the
lwIP bindings to improve non-blocking sockets and error codes.  The VFS has
some regressions fixed including the ability to statvfs the root.

All changes are listed below.

py core:
- modbuiltins: add core-provided version of input() function
- objstr: catch case of negative "maxsplit" arg to str.rsplit()
- persistentcode: allow to compile with complex numbers disabled
- objstr: allow to compile with obj-repr D, and unicode disabled
- modsys: allow to compile with obj-repr D and PY_ATTRTUPLE disabled
- provide mp_decode_uint_skip() to help reduce stack usage
- makeqstrdefs.py: make script run correctly with Python 2.6
- objstringio: if created from immutable object, follow copy on write policy

extmod:
- modlwip: connect: for non-blocking mode, return EINPROGRESS
- modlwip: fix error codes for duplicate calls to connect()
- modlwip: accept: fix error code for non-blocking mode
- vfs: allow to statvfs the root directory
- vfs: allow "buffering" and "encoding" args to VFS's open()
- modframebuf: fix signed/unsigned comparison pendantic warning

lib:
- libm: use isfinite instead of finitef, for C99 compatibility
- utils/interrupt_char: remove support for KBD_EXCEPTION disabled

tests:
- basics/string_rsplit: add tests for negative "maxsplit" argument
- float: convert "sys.exit()" to "raise SystemExit"
- float/builtin_float_minmax: PEP8 fixes
- basics: convert "sys.exit()" to "raise SystemExit"
- convert remaining "sys.exit()" to "raise SystemExit"

unix port:
- convert to use core-provided version of built-in import()
- Makefile: replace references to make with $(MAKE)

windows port:
- convert to use core-provided version of built-in import()

qemu-arm port:
- Makefile: adjust object-file lists to get correct dependencies
- enable micropython.mem_*() functions to allow more tests

stmhal port:
- boards: enable DAC for NUCLEO_F767ZI board
- add support for NUCLEO_F446RE board
- pass USB handler as parameter to allow more than one USB handler
- usb: use local USB handler variable in Start-of-Frame handler
- usb: make state for USB device private to top-level USB driver
- usbdev: for MSC implement SCSI SYNCHRONIZE_CACHE command
- convert from using stmhal's input() to core provided version

cc3200 port:
- convert from using stmhal's input() to core provided version

teensy port:
- convert from using stmhal's input() to core provided version

esp8266 port:
- Makefile: replace references to make with $(MAKE)
- Makefile: add clean-modules target
- convert from using stmhal's input() to core provided version

zephyr port:
- modusocket: getaddrinfo: Fix mp_obj_len() usage
- define MICROPY_PY_SYS_PLATFORM (to "zephyr")
- machine_pin: use native Zephyr types for Zephyr API calls

docs:
- machine.Pin: remove out_value() method
- machine.Pin: add on() and off() methods
- esp8266: consistently replace Pin.high/low methods with .on/off
- esp8266/quickref: polish Pin.on()/off() examples
- network: move confusingly-named cc3200 Server class to its reference
- uos: deconditionalize, remove minor port-specific details
- uos: move cc3200 port legacy VFS mounting functions to its ref doc
- machine: sort machine classes in logical order, not alphabetically
- network: first step to describe standard network class interface

examples:
- embedding: use core-provided KeyboardInterrupt object
2017-06-20 10:56:05 -07:00
Damien George e269cabe3e py/objint: In to_bytes(), allow length arg to be any int and check sign. 2017-06-15 14:21:02 +10:00
Damien George 8c5632a869 py/objint: Support "big" byte-order in int.to_bytes(). 2017-06-15 13:56:21 +10:00
Damien George 1e70fda69f py/compile: Raise SyntaxError if positional args are given after */**.
In CPython 3.4 this raises a SyntaxError.  In CPython 3.5+ having a
positional after * is allowed but uPy has the wrong semantics and passes
the arguments in the incorrect order.  To prevent incorrect use of a
function going unnoticed it is important to raise the SyntaxError in uPy,
until the behaviour is fixed to follow CPython 3.5+.
2017-06-14 18:18:01 +10:00