Commit Graph

784 Commits

Author SHA1 Message Date
Kenny 98aa4b7943 update async tests with less upython workaround and more cpython compatibility 2020-10-10 23:39:32 -07:00
Jeff Epler 54d97251fe modstruct: Improve compliance with python3
While checking whether we can enable -Wimplicit-fallthrough, I encountered
a diagnostic in mp_binary_set_val_array_from_int which led to discovering
the following bug:
```
>>> struct.pack("xb", 3)
b'\x03\x03'
```
That is, the next value (3) was used as the value of a padding byte, while
standard Python always fills "x" bytes with zeros.  I initially thought
this had to do with the unintentional fallthrough, but it doesn't.
Instead, this code would relate to an array.array with a typecode of
padding ('x'), which is ALSO not desktop Python compliant:
```
>>> array.array('x', (1, 2, 3))
array('x', [1, 0, 0])
```
Possibly this is dead code that used to be shared between struct-setting
and array-setting, but it no longer is.

I also discovered that the argument list length for struct.pack
and struct.pack_into were not checked, and that the length of binary data
passed to array.array was not checked to be a multiple of the element
size.

I have corrected all of these to conform more closely to standard Python
and revised some tests where necessary.  Some tests for micropython-specific
behavior that does not conform to standard Python and is not present
in CircuitPython was deleted outright.
2020-09-12 14:07:23 -05:00
Jeff Epler 20c2dd0c08 core: add int.bit_length() when MICROPY_CYPTHON_COMPAT is enabled
This method of integer objects is needed for a port of python3's
decimal.py module.

MICROPY_CPYTHON_COMPAT is enabled by CIRCUITPY_FULL_BUILD.
2020-09-06 09:53:16 -05:00
Jonathan Hogg 901f3dce6e py/compile: Don't await __aiter__ special method in async-for.
MicroPython's original implementation of __aiter__ was correct for an
earlier (provisional) version of PEP492 (CPython 3.5), where __aiter__ was
an async-def function.  But that changed in the final version of PEP492 (in
CPython 3.5.2) where the function was changed to a normal one.  See
https://www.python.org/dev/peps/pep-0492/#why-aiter-does-not-return-an-awaitable
See also the note at the end of this subsection in the docs:
https://docs.python.org/3.5/reference/datamodel.html#asynchronous-iterators
And for completeness the BPO: https://bugs.python.org/issue27243

To be consistent with the Python spec as it stands today (and now that
PEP492 is final) this commit changes MicroPython's behaviour to match
CPython:  __aiter__ should return an async-iterable object, but is not
itself awaitable.

The relevant tests are updated to match.

See #6267.
2020-07-24 22:55:37 -07:00
Kenny 51a79b1af7 add coroutine behavior for generators
coroutines don't have __next__; they also call themselves coroutines.
This does not change the fact that `async def` methods are generators,
but it does make them behave more like CPython.
2020-07-23 20:40:16 -07:00
Diego Elio Pettenò dd5d7c86d2 Fix up end of file and trailing whitespace.
This can be enforced by pre-commit, but correct it separately to make it easier to review.
2020-06-03 10:56:35 +01:00
Damien George aa10e5c334 tests: Add .exp files for basics/parser and import/import_override.
Because CPython 3.8.0 now produces different output:
- basics/parser.py: CPython does not allow '\\\n' as input.
- import/import_override: CPython imports _io.
2020-04-01 16:31:33 -05:00
Damien George 764e65fb11 tests/basics: Provide .exp files for generator tests that fail PEP479.
PEP479 (see https://www.python.org/dev/peps/pep-0479/) prohibited raising
StopIteration from within a generator (it is turned into a RuntimeError).
This behaviour was introduced in Python 3.5 and in 3.7 was made compulsory.
Until uPy implements PEP479, this patch adds .py.exp files for the relevant
tests so they can be run under Python 3.7.
2020-03-31 17:27:10 -05:00
Damien George 8f0147cf00 tests: Modify tests that print repr of an exception with 1 arg.
In Python 3.7 the behaviour of repr() of an exception with one argument
changed: it no longer prints a trailing comma in the argument list.  See
https://bugs.python.org/issue30399

This patch modifies tests that rely on this behaviour to not rely on it.
And the python34.py test is updated to include a test for this behaviour
with a .exp file.
2020-03-31 17:27:10 -05:00
Damien George 4c4f81f8f2 tests/basics/int_big_error.py: Use bytearray to test for int overflow.
In Python 3.7 "1 >> (big int)" is now allowed, it no longer raises an
OverflowError.  So use bytearray to test big-int conversion overflow.
2020-03-31 17:27:10 -05:00
Damien George 8b5fd95897 tests/basics/set_pop.py: Sort set before printing for consistent output. 2020-03-31 17:27:10 -05:00
Jeff Epler ef195d6e1b Update tests
* string_pep498_fstring.py: Not compatible with python3.5
 * cmd_parsetree.py: enumerated constants changed
2020-03-09 21:13:33 -05:00
Jeff Epler 32647cd9b4 lexer: catch concatenation of f'' and '' strings
This turns the "edge case" into a parse-time error.
2020-03-09 09:03:25 -05:00
Josh Klar 40bc05ee1e Address dpgeorge feedback - largely simplifications 2020-03-09 08:16:07 -05:00
Josh Klar 3a7a5ba686 py: Implement partial PEP-498 (f-string) support
This implements (most of) the PEP-498 spec for f-strings, with two
exceptions:

- raw f-strings (`fr` or `rf` prefixes) raise `NotImplementedError`
- one special corner case does not function as specified in the PEP
(more on that in a moment)

This is implemented in the core as a syntax translation, brute-forcing
all f-strings to run through `String.format`. For example, the statement
`x='world'; print(f'hello {x}')` gets translated *at a syntax level*
(injected into the lexer) to `x='world'; print('hello {}'.format(x))`.
While this may lead to weird column results in tracebacks, it seemed
like the fastest, most efficient, and *likely* most RAM-friendly option,
despite being implemented under the hood with a completely separate
`vstr_t`.

Since [string concatenation of adjacent literals is implemented in the
lexer](534b7c368d),
two side effects emerge:

- All strings with at least one f-string portion are concatenated into a
single literal which *must* be run through `String.format()` wholesale,
and:
- Concatenation of a raw string with interpolation characters with an
f-string will cause `IndexError`/`KeyError`, which is both different
from CPython *and* different from the corner case mentioned in the PEP
(which gave an example of the following:)

```python
x = 10
y = 'hi'
assert ('a' 'b' f'{x}' '{c}' f'str<{y:^4}>' 'd' 'e') == 'ab10{c}str< hi >de'
```

The above-linked commit detailed a pretty solid case for leaving string
concatenation in the lexer rather than putting it in the parser, and
undoing that decision would likely be disproportionately costly on
resources for the sake of a probably-low-impact corner case. An
alternative to become complaint with this corner case of the PEP would
be to revert to string concatenation in the parser *only when an
f-string is part of concatenation*, though I've done no investigation on
the difficulty or costs of doing this.

A decent set of tests is included. I've manually tested this on the
`unix` port on Linux and on a Feather M4 Express (`atmel-samd`) and
things seem sane.
2020-03-09 08:16:07 -05:00
Dan Halbert c592bd612a Implement to_bytes(..., signed=True) 2020-02-14 15:12:20 -05:00
Roy Hooper 2cb8f7b2df Add test for issue #2465 - tuple subsclass subscript 2020-01-09 20:13:53 -05:00
Scott Shawcroft 9435e01f9e
Support __bytes
Fixes #1763
2019-10-14 16:05:17 -07:00
Dan Halbert 8664a6574b use approx of original @godlygeek code for smallints; add tests 2019-05-12 11:17:29 -04:00
Matt Wozniski e041df73bb Add tests for overflows converting ints to bytes 2019-05-09 03:22:24 -04:00
Dan Halbert 7c219600a2 WIP: after merge; before testing 2018-07-11 16:45:30 -04:00
Damien George 6a445b60fa py/lexer: Add support for underscores in numeric literals.
This is a very convenient feature introduced in Python 3.6 by PEP 515.
2018-06-12 12:17:43 +10:00
Damien George 36c1052183 py/objtype: Optimise instance get/set/del by skipping special accessors.
This patch is a code optimisation, trading text bytes for speed.  On
pyboard it's an increase of 0.06% in code size for a gain (in pystone
performance) of roughly 6.5%.

The patch optimises load/store/delete of attributes in user defined classes
by not looking up special accessors (@property, __get__, __delete__,
__set__, __setattr__ and __getattr_) if they are guaranteed not to exist in
the class.

Currently, if you do my_obj.foo() then the runtime has to do a few checks
to see if foo is a property or has __get__, and if so delegate the call.
And for stores things like my_obj.foo = 1 has to first check if foo is a
property or has __set__ defined on it.

Doing all those checks each and every time the attribute is accessed has a
performance penalty.  This patch eliminates all those checks for cases when
it's guaranteed that the checks will always fail, ie no attributes are
properties nor have any special accessor methods defined on them.

To make this guarantee it checks all attributes of a user-defined class
when it is first created.  If any of the attributes of the user class are
properties or have special accessors, or any of the base classes of the
user class have them, then it sets a flag in the class to indicate that
special accessors must be checked for.  Then in the load/store/delete code
it checks this flag to see if it can take the shortcut and optimise the
lookup.

It's an optimisation that's pretty widely applicable because it improves
lookup performance for all methods of user defined classes, and stores of
attributes, at least for those that don't have special accessors.  And, it
allows to enable descriptors with minimal additional runtime overhead if
they are not used for a particular user class.

There is one restriction on dynamic class creation that has been introduced
by this patch: a user-defined class cannot go from zero special accessors
to one special accessor (or more) after that class has been subclassed.  If
the script attempts this an AttributeError is raised (see addition to
tests/misc/non_compliant.py for an example of this case).

The cost in code space bytes for the optimisation in this patch is:

   unix x64:  +528
unix nanbox:  +508
      stm32:  +192
     cc3200:  +200
    esp8266:  +332
      esp32:  +244

Performance tests that were done:

- on unix x86-64, pystone improved by about 5%
- on pyboard, pystone improved by about 6.5%, from 1683 up to 1794
- on pyboard, bm_chaos (from CPython benchmark suite) improved by about 5%
- on esp32, pystone improved by about 30% (but there are caching effects)
- on esp32, bm_chaos improved by about 11%
2018-06-08 12:12:08 +10:00
Jeff Epler c60589c02b py/objtype: Fix assertion failures in super_attr by checking type.
Fixes assertion failures and segmentation faults when making calls like:

    super(1, 1).x
2018-05-30 11:14:07 +10:00
Jeff Epler 05b13fd292 py/objtype: Fix assertion failures in mp_obj_new_type by checking types.
Fixes assertion failures when the arguments to type() were not of valid
types, e.g., when making calls like:

    type("", (), 3)
    type("", 3, {})
2018-05-30 11:11:24 +10:00
Damien George dfeaea1441 py/objtype: Remove TODO comment about needing to check for property.
Instance members are always treated as values, even if they are properties.
A test is added to show this is the case.
2018-05-25 10:59:40 +10:00
Damien George 400273a799 py/objgenerator: Protect against reentering a generator.
Generators that are already executing cannot be reexecuted.  This patch
puts in a check for such a case.

Thanks to @jepler for finding the bug.
2018-05-22 16:54:03 +10:00
Jan Klusacek b318ebf101 py/modbuiltins: Add support for rounding integers.
As per CPython semantics.  This feature is controlled by
MICROPY_PY_BUILTINS_ROUND_INT which is disabled by default.
2018-05-22 14:18:16 +10:00
Damien George 1ad0013dec tests: Add some tests for bigint hash, float hash and float parsing.
Following outcome of recent fuzz testing and sanitizing by @jepler.
2018-05-21 13:05:40 +10:00
Dan Halbert 54293397c5
Merge pull request #837 from godlygeek/human_readable_oserror
Human readable OSError messages
2018-05-15 10:12:09 -04:00
Matt Wozniski 086bffb594 Update errno tests to match new expected behavior
Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
2018-05-14 23:16:18 -04:00
bildzeitung bf26ffbf56 Updated tests; removed try/catch for ucollections 2018-05-14 14:43:34 -04:00
Damien George 7541be5637 tests/basics/special_methods2: Enable some additional tests that work.
These special methods are all available if MICROPY_PY_ALL_SPECIAL_METHODS
is enabled.
2018-05-11 17:37:16 +10:00
Damien George 3678a6bdc6 py/modbuiltins: Make built-in dir support the __dir__ special method.
If MICROPY_PY_ALL_SPECIAL_METHODS is enabled then dir() will now delegate
to the special method __dir__ if the object it is listing has this method.
2018-05-10 23:14:23 +10:00
Damien George 529860643b py/modbuiltins: Make built-in hasattr work properly for user types.
It now allows __getattr__ in a user type to raise AttributeError when the
attribute does not exist.
2018-05-10 23:03:30 +10:00
Jeff Epler d6cf5c6749 py/objstr: In find/rfind, don't crash when end < start. 2018-04-05 16:14:17 +10:00
Damien George 1bfc774a08 tests/basics/string_compare.py: Add test with string that hashes to 0.
The string "Q+?" is special in that it hashes to zero with the djb2
algorithm (among other strings), and a zero hash should be incremented to a
hash of 1.
2018-04-05 01:04:38 +10:00
Damien George 22161acf47 tests/basics/class_super.py: Add tests for store/delete of super attr. 2018-04-05 01:03:57 +10:00
Damien George 7b7bbd0ee7 tests/basics: Add tests for edge cases of nan-box's 47-bit small int. 2018-04-05 00:59:49 +10:00
Damien George dd48ccb1e3 tests/basics: Add test for subclassing an iterable native type. 2018-04-04 15:26:18 +10:00
Damien George df02f5620a tests/basics/int_big1.py: Add test for big int in mp_obj_get_int_maybe. 2018-04-04 15:23:32 +10:00
Damien George 7d5c753b17 tests/basics: Modify int-big tests to prevent constant folding.
So that these tests test the runtime behaviour, not the compiler (which may
be executed offline).
2018-04-04 13:57:22 +10:00
Damien George f684e9e1ab tests/basics/int_big1.py: Add test converting str with non-print chars. 2018-04-04 13:56:00 +10:00
Damien George 430efb0444 tests/basics: Add test for use of return within try-except.
The case of a return statement in the try suite of a try-except statement
was previously only tested by builtin_compile.py, and only then in the part
of this test which checked for the existence of the compile builtin.  So
this patch adds an explicit unit test for this case.
2018-04-04 01:43:16 +10:00
Scott Shawcroft e4ae1e3d59
Merge pull request #734 from jepler/str-find-backwards-circuitpython
py/objstr: Don't crash when end < start
2018-04-01 22:41:59 -07:00
Jeff Epler a909007fef py/objgenerator: Check stack before resuming a generator
This turns a hard crash in a recursive generator into
a 'maximum recursion depth exceeded' exception.
2018-04-01 16:40:15 -05:00
Jeff Epler 0041df0c6b py/objstr: Don't crash when end < start
.. and add testcases for the same.

(crash found by afl-fuzz)
2018-03-31 22:17:11 -05:00
Scott Shawcroft 3215b85568
Merge pull request #728 from jepler/double-splat-crash-circuitpython
py/bc: Turn assertion error into exception
2018-03-31 09:57:25 -07:00
Jeff Epler 853f7ac4d0 py/bc: Turn assertion error into exception 2018-03-31 08:44:29 -05:00
Damien George bcfff4fc98 tests/basics/iter1.py: Add more tests for walking a user-defined iter.
Some code in mp_iternext() was only tested by the native emitter, so the
tests added here test this function using just the bytecode emitter.
2018-03-30 14:23:13 +11:00