Commit Graph

771 Commits

Author SHA1 Message Date
Damien George e1fb03f3e2 py: Fix VM crash with unwinding jump out of a finally block.
This patch fixes a bug in the VM when breaking within a try-finally.  The
bug has to do with executing a break within the finally block of a
try-finally statement.  For example:

    def f():
        for x in (1,):
            print('a', x)
            try:
                raise Exception
            finally:
                print(1)
                break
            print('b', x)
    f()

Currently in uPy the above code will print:

    a 1
    1
    1
    segmentation fault (core dumped)  micropython

Not only is there a seg fault, but the "1" in the finally block is printed
twice.  This is because when the VM executes a finally block it doesn't
really know if that block was executed due to a fall-through of the try (no
exception raised), or because an exception is active.  In particular, for
nested finallys the VM has no idea which of the nested ones have active
exceptions and which are just fall-throughs.  So when a break (or continue)
is executed it tries to unwind all of the finallys, when in fact only some
may be active.

It's questionable whether break (or return or continue) should be allowed
within a finally block, because they implicitly swallow any active
exception, but nevertheless it's allowed by CPython (although almost never
used in the standard library).  And uPy should at least not crash in such a
case.

The solution here relies on the fact that exception and finally handlers
always appear in the bytecode after the try body.

Note: there was a similar bug with a return in a finally block, but that
was previously fixed in b735208403
2019-03-05 16:05:05 +11:00
Damien George 12ce9f2689 py/compile: Fix handling of unwinding BaseException in async with.
All exceptions that unwind through the async-with must be caught and
BaseException is the top-level class, which includes Exception and others.

Fixes issue #4552.
2019-02-26 23:52:10 +11:00
Damien George be41d6d6f9 tests/basics: Add tests for try-except-else and try-except-else-finally. 2019-02-21 16:22:41 +11:00
stijn 42863830be py: Add optional support for 2-argument version of built-in next().
Configurable via MICROPY_PY_BUILTINS_NEXT2, disabled by default.
2019-01-27 13:01:28 +11:00
Paul Sokolovsky d4d4bc5827 tests/basics/special_methods2: Typo fix in comment. 2018-12-13 01:29:01 +11:00
Paul Sokolovsky d690c2e148 tests/basics/special_methods: Add testcases for __int__. 2018-12-07 17:28:04 +11:00
Damien George 113f00a9ab py/objboundmeth: Support loading generic attrs from the method.
Instead of assuming that the method is a bytecode object, and only
supporting load of __name__, make the operation generic by delegating the
load to the method object itself.  Saves a bit of code size and fixes the
case of attempting to load __name__ on a native method, see issue #4028.
2018-12-06 18:02:41 +11:00
Damien George 9201f46cc8 py/compile: Fix case of eager implicit conversion of local to nonlocal.
This ensures that implicit variables are only converted to implicit
closed-over variables (nonlocals) at the very end of the function scope.
If variables are closed-over when first used (read from, as was done prior
to this commit) then this can be incorrect because the variable may be
assigned to later on in the function which means they are just a plain
local, not closed over.

Fixes issue #4272.
2018-10-28 00:33:08 +11:00
Paul Sokolovsky a527313382 tests: Make bytes/str.count() tests skippable. 2018-10-22 22:50:28 +11:00
Damien George a07e56cbd8 tests/basics/class_getattr: Remove invalid test for __getattribute__.
Part of this test was trying to test some functionality of __getattribute__
but this method name was misspelt so it wasn't doing anything useful.
Fixing the typo in this name makes the test fail because MicroPython
doesn't support user defined __getattribute__ methods.  So this part of the
test is removed.  The remaining tests are modified slightly to make it
clearer what they are testing.
2018-10-18 12:28:09 +11:00
Damien George 7eb29c2000 py/objtype: Remove comment about catching exc from user __getattr__.
Any exception raised in a user __getattr__ should be propagated out.  A
test is added to verify these semantics.
2018-10-18 12:15:16 +11:00
Damien George dd288904db py/objtype: Support full object model for get/set/delitem special meths.
This makes these special methods have the same calling behaviour as other
methods in a class instance (mp_convert_member_lookup() is already called
by mp_obj_class_lookup()).
2018-09-28 23:22:34 +10:00
Damien George 0c9d452370 py/vm: Fix case of throwing GeneratorExit type into yield-from.
mp_make_raise_obj must be used to convert a possible exception type to an
instance object, otherwise the VM may raise a non-exception object.

An existing test is adjusted to test this case, with the original test
already moved to generator_throw.py.
2018-09-28 11:39:35 +10:00
Damien George e6078dfed2 tests/basics: Split out gen throw tests from yield-from-throw tests. 2018-09-28 11:35:31 +10:00
Damien George fc1bb51af5 py/objgenerator: Remove TODO about returning gen being called again.
The code implements correct behaviour, as tested by the new test case added
in this commit.
2018-09-27 15:18:24 +10:00
Damien George 3f6ffe059f py/objgenerator: Implement PEP479, StopIteration convs to RuntimeError.
This commit implements PEP479 which disallows raising StopIteration inside
a generator to signal that it should be finished.  Instead, the generator
should simply return when it is complete.

See https://www.python.org/dev/peps/pep-0479/ for details.
2018-09-20 15:36:59 +10:00
Damien George f2de9d60f7 py/emitnative: Fix try-finally in outer scope, so finally is cancelled. 2018-09-11 15:33:25 +10:00
Paul Sokolovsky 674e069ba9 py/objarray: bytearray: Allow 2nd/3rd arg to constructor.
If bytearray is constructed from str, a second argument of encoding is
required (in CPython), and third arg of Unicode error handling is allowed,
e.g.:

bytearray("str", "utf-8", "strict")

This is similar to bytes:

bytes("str", "utf-8", "strict")

This patch just allows to pass 2nd/3rd arguments to bytearray, but
doesn't try to validate them to not impact code size. (This is also
similar to how bytes constructor is handled, though it does a bit
more validation, e.g. check that in case of str arg, encoding argument
is passed.)
2018-09-11 15:10:10 +10:00
Damien George 4970e9bc8c tests/basics: Add test cases for context manager raising in enter/exit. 2018-09-04 14:37:30 +10:00
Damien George b14c705c18 tests/basics: Add more tests for return within try-finally. 2018-09-04 14:37:07 +10:00
Damien George 3cd2c281d7 py/emitnative: Cancel caught exception once handled to prevent reraise.
The native emitter keeps the current exception in a slot in its C stack
(instead of on its Python value stack), so when it catches an exception it
must explicitly clear that slot so the same exception is not reraised later
on.
2018-09-03 17:41:02 +10:00
Damien George b735208403 py/vm: Fix handling of finally-return with complex nested finallys.
Back in 8047340d75 basic support was added in
the VM to handle return statements within a finally block.  But it didn't
cover all cases, in particular when some finally's were active and others
inactive when the "return" was executed.

This patch adds further support for return-within-finally by correctly
managing the currently_in_except_block flag, and should fix all cases.  The
main point is that finally handlers remain on the exception stack even if
they are active (currently being executed), and the unwind return code
should only execute those finally's which are inactive.

New tests are added for the cases which now pass.
2018-09-03 13:08:16 +10:00
Damien George 828f771e32 tests/basics: Provide .exp files for generator tests that fail PEP479.
PEP479 (see https://www.python.org/dev/peps/pep-0479/) prohibited raising
StopIteration from within a generator (it is turned into a RuntimeError).
This behaviour was introduced in Python 3.5 and in 3.7 was made compulsory.
Until uPy implements PEP479, this patch adds .py.exp files for the relevant
tests so they can be run under Python 3.7.
2018-08-17 15:50:21 +10:00
Damien George 8979ce1671 tests: Modify tests that print repr of an exception with 1 arg.
In Python 3.7 the behaviour of repr() of an exception with one argument
changed: it no longer prints a trailing comma in the argument list.  See
https://bugs.python.org/issue30399

This patch modifies tests that rely on this behaviour to not rely on it.
And the python34.py test is updated to include a test for this behaviour
with a .exp file.
2018-08-17 15:46:04 +10:00
Damien George 0988b14cd6 tests/basics/int_big_error.py: Use bytearray to test for int overflow.
In Python 3.7 "1 >> (big int)" is now allowed, it no longer raises an
OverflowError.  So use bytearray to test big-int conversion overflow.
2018-08-17 15:43:47 +10:00
Damien George 96e1fd480d tests/basics/set_pop.py: Sort set before printing for consistent output. 2018-08-17 15:42:51 +10:00
Ayke van Laethem 6572029dc0 tests: Make tests work on targets without float support. 2018-08-04 15:14:23 +10:00
Damien George e2e22e3d7e py/objgenerator: Implement __name__ with normal fun attr accessor code.
With the recent change b488a4a848, a
generating function now has the same layout in memory as a normal bytecode
function, and so can reuse the latter's attribute accessor code to
implement __name__.
2018-07-10 16:33:57 +10:00
Damien George d8dc918deb py/compile: Handle return/break/continue correctly in async with.
Before this patch the context manager's __aexit__() method would not be
executed if a return/break/continue statement was used to exit an async
with block.  async with now has the same semantics as normal with.

The fix here applies purely to the compiler, and does not modify the
runtime at all. It might (eventually) be better to define new bytecode(s)
to handle async with (and maybe other async constructs) in a cleaner, more
efficient way.

One minor drawback with addressing this issue purely in the compiler is
that it wasn't possible to get 100% CPython semantics.  The thing that is
different here to CPython is that the __aexit__ method is not looked up in
the context manager until it is needed, which is after the body of the
async with statement has executed.  So if a context manager doesn't have
__aexit__ then CPython raises an exception before the async with is
executed, whereas uPy will raise it after it is executed.  Note that
__aenter__ is looked up at the beginning in uPy because it needs to be
called straightaway, so if the context manager isn't a context manager then
it'll still raise an exception at the same location as CPython.  The only
difference is if the context manager has the __aenter__ method but not the
__aexit__ method, then in that case uPy has different behaviour.  But this
is a very minor, and acceptable, difference.
2018-06-27 16:57:42 +10:00
Damien George 726804ea40 tests: Move non-filesystem io tests to basics dir with io_ prefix. 2018-06-27 16:55:05 +10:00
Paul Sokolovsky bdceea1d12 tests/basics/namedtuple*: Import ucollections first.
Otherwise, test may have artefacts in the presence of the micropython-lib
module.
2018-06-27 14:58:14 +10:00
Damien George 6a445b60fa py/lexer: Add support for underscores in numeric literals.
This is a very convenient feature introduced in Python 3.6 by PEP 515.
2018-06-12 12:17:43 +10:00
Damien George 36c1052183 py/objtype: Optimise instance get/set/del by skipping special accessors.
This patch is a code optimisation, trading text bytes for speed.  On
pyboard it's an increase of 0.06% in code size for a gain (in pystone
performance) of roughly 6.5%.

The patch optimises load/store/delete of attributes in user defined classes
by not looking up special accessors (@property, __get__, __delete__,
__set__, __setattr__ and __getattr_) if they are guaranteed not to exist in
the class.

Currently, if you do my_obj.foo() then the runtime has to do a few checks
to see if foo is a property or has __get__, and if so delegate the call.
And for stores things like my_obj.foo = 1 has to first check if foo is a
property or has __set__ defined on it.

Doing all those checks each and every time the attribute is accessed has a
performance penalty.  This patch eliminates all those checks for cases when
it's guaranteed that the checks will always fail, ie no attributes are
properties nor have any special accessor methods defined on them.

To make this guarantee it checks all attributes of a user-defined class
when it is first created.  If any of the attributes of the user class are
properties or have special accessors, or any of the base classes of the
user class have them, then it sets a flag in the class to indicate that
special accessors must be checked for.  Then in the load/store/delete code
it checks this flag to see if it can take the shortcut and optimise the
lookup.

It's an optimisation that's pretty widely applicable because it improves
lookup performance for all methods of user defined classes, and stores of
attributes, at least for those that don't have special accessors.  And, it
allows to enable descriptors with minimal additional runtime overhead if
they are not used for a particular user class.

There is one restriction on dynamic class creation that has been introduced
by this patch: a user-defined class cannot go from zero special accessors
to one special accessor (or more) after that class has been subclassed.  If
the script attempts this an AttributeError is raised (see addition to
tests/misc/non_compliant.py for an example of this case).

The cost in code space bytes for the optimisation in this patch is:

   unix x64:  +528
unix nanbox:  +508
      stm32:  +192
     cc3200:  +200
    esp8266:  +332
      esp32:  +244

Performance tests that were done:

- on unix x86-64, pystone improved by about 5%
- on pyboard, pystone improved by about 6.5%, from 1683 up to 1794
- on pyboard, bm_chaos (from CPython benchmark suite) improved by about 5%
- on esp32, pystone improved by about 30% (but there are caching effects)
- on esp32, bm_chaos improved by about 11%
2018-06-08 12:12:08 +10:00
Jeff Epler c60589c02b py/objtype: Fix assertion failures in super_attr by checking type.
Fixes assertion failures and segmentation faults when making calls like:

    super(1, 1).x
2018-05-30 11:14:07 +10:00
Jeff Epler 05b13fd292 py/objtype: Fix assertion failures in mp_obj_new_type by checking types.
Fixes assertion failures when the arguments to type() were not of valid
types, e.g., when making calls like:

    type("", (), 3)
    type("", 3, {})
2018-05-30 11:11:24 +10:00
Damien George dfeaea1441 py/objtype: Remove TODO comment about needing to check for property.
Instance members are always treated as values, even if they are properties.
A test is added to show this is the case.
2018-05-25 10:59:40 +10:00
Damien George 400273a799 py/objgenerator: Protect against reentering a generator.
Generators that are already executing cannot be reexecuted.  This patch
puts in a check for such a case.

Thanks to @jepler for finding the bug.
2018-05-22 16:54:03 +10:00
Jan Klusacek b318ebf101 py/modbuiltins: Add support for rounding integers.
As per CPython semantics.  This feature is controlled by
MICROPY_PY_BUILTINS_ROUND_INT which is disabled by default.
2018-05-22 14:18:16 +10:00
Damien George 1ad0013dec tests: Add some tests for bigint hash, float hash and float parsing.
Following outcome of recent fuzz testing and sanitizing by @jepler.
2018-05-21 13:05:40 +10:00
Damien George 7541be5637 tests/basics/special_methods2: Enable some additional tests that work.
These special methods are all available if MICROPY_PY_ALL_SPECIAL_METHODS
is enabled.
2018-05-11 17:37:16 +10:00
Damien George 3678a6bdc6 py/modbuiltins: Make built-in dir support the __dir__ special method.
If MICROPY_PY_ALL_SPECIAL_METHODS is enabled then dir() will now delegate
to the special method __dir__ if the object it is listing has this method.
2018-05-10 23:14:23 +10:00
Damien George 529860643b py/modbuiltins: Make built-in hasattr work properly for user types.
It now allows __getattr__ in a user type to raise AttributeError when the
attribute does not exist.
2018-05-10 23:03:30 +10:00
Jeff Epler d6cf5c6749 py/objstr: In find/rfind, don't crash when end < start. 2018-04-05 16:14:17 +10:00
Damien George 1bfc774a08 tests/basics/string_compare.py: Add test with string that hashes to 0.
The string "Q+?" is special in that it hashes to zero with the djb2
algorithm (among other strings), and a zero hash should be incremented to a
hash of 1.
2018-04-05 01:04:38 +10:00
Damien George 22161acf47 tests/basics/class_super.py: Add tests for store/delete of super attr. 2018-04-05 01:03:57 +10:00
Damien George 7b7bbd0ee7 tests/basics: Add tests for edge cases of nan-box's 47-bit small int. 2018-04-05 00:59:49 +10:00
Damien George dd48ccb1e3 tests/basics: Add test for subclassing an iterable native type. 2018-04-04 15:26:18 +10:00
Damien George df02f5620a tests/basics/int_big1.py: Add test for big int in mp_obj_get_int_maybe. 2018-04-04 15:23:32 +10:00
Damien George 7d5c753b17 tests/basics: Modify int-big tests to prevent constant folding.
So that these tests test the runtime behaviour, not the compiler (which may
be executed offline).
2018-04-04 13:57:22 +10:00
Damien George f684e9e1ab tests/basics/int_big1.py: Add test converting str with non-print chars. 2018-04-04 13:56:00 +10:00