Commit Graph

2552 Commits

Author SHA1 Message Date
Jeff Epler
27e6623657
Add unicode support to dotenv
Newly passing tests:
```
aa🐍bb=key with emoji
value_with_emoji=aa🐍bb
```
2022-10-05 18:18:07 -05:00
Dan Halbert
91079279da Allow spaces before = in dotenv 2022-10-05 15:50:53 -04:00
Jeff Epler
718b8e706e
disable test e15 for now 2022-10-04 10:54:22 -05:00
Jeff Epler
52bca95208
Refactor dotenv module so that it can be tested on host
.. it needs to operate on a FILE* rather than FIL depending on
the build.

Note that this is comparing output to expected, not to cpython dotenv
package. Because run-tests.py starts the CPython interpreter with the
'-S' (skip site initialization) flag, pip-installed packages are
not available for import inside a test file. Instead, the exp
file is generated manually:
```
circuitpython/tests$ python3 circuitpython/dotenv_test.py > circuitpython/dotenv_test.py.exp
```

Unfortunately, the test fails on test e15:
```diff
FAILURE /home/jepler/src/circuitpython/tests/results/circuitpython_dotenv_test.py
--- /home/jepler/src/circuitpython/tests/results/circuitpython_dotenv_test.py.exp	2022-10-04 09:48:16.307703128 -0500
+++ /home/jepler/src/circuitpython/tests/results/circuitpython_dotenv_test.py.out	2022-10-04 09:48:16.307703128 -0500
@@ -14,7 +14,7 @@
 line
 e13 e13value
 e14 None
-e15 e15value
+e15 None
 e16 #
 e17 def
 e18 #has a hash
```
2022-10-04 09:51:27 -05:00
Dan Halbert
6dc03ae3ce fix some dotenv parsing 2022-10-03 21:56:11 -04:00
Jim Mussared
9d6f474ea4 py/objstr: Don't treat bytes as unicode in str.count.
`b'\xaa \xaa'.count(b'\xaa')` now (correctly) returns 2 instead of 1.

Fixes issue #9404.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-26 00:54:18 +10:00
Angus Gratton
25ff5b52d9 py/parse: Allow const types other than int to optimise as true/false.
Allows optimisation of cases like:

    import micropython
    _DEBUG = micropython.const(False)
    if _DEBUG:
        print('Debugging info')

Previously the 'if' statement was only optimised out if the type of the
const() argument was integer.

The change is implemented in a way that makes the compiler slightly smaller
(-16 bytes on PYBV11) but compilation will also be very slightly slower.

As a bonus, if const support is enabled then the compiler can now optimise
const truthy/falsey expressions of other types, like:

    while "something":
        pass

... unclear if that is useful, but perhaps it could be.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-09-23 16:04:13 +10:00
Angus Gratton
f91ebf6fa9 tests: Allow 'special' tests to output "SKIP" on a single line. 2022-09-23 16:02:59 +10:00
Andrew Leech
13c4470fd0 tests/run-multitests: Make paths more deterministic.
Allows running from a different directory, etc.

This work was funded by Planet Innovation.
2022-09-20 09:07:18 +10:00
Andrew Leech
7589d86b6b tests/run-multitests: Extend usage information. 2022-09-20 09:07:02 +10:00
Jim Mussared
920da9c5e3 unix/variants/coverage: Add test for manifest freeze_mpy().
This uses the frozentest.mpy that is also used by ports/minimal.

Also fixes two bugs that these new tests picked up:
 - File extension matching in manifestfile.py.
 - Handling of freeze_mpy results in makemanifest.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:51:10 +10:00
Jim Mussared
6ecdf1a240 tests/frozen: Move frozentest.mpy from ports/ to tests/.
frozentest.mpy was previously duplicated in ports/minimal and
ports/powerpc.

This needs to be re-generated on every .mpy version increase, so might as
well just have a single copy of it.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:51:10 +10:00
stijn
9ae8d38204 extmod/vfs_posix_file: Implement finaliser for files.
Prevent handle leaks when file objects aren't closed explicitly and
fix some MICROPY_CPYTHON_COMPAT issues: this wasn't properly adhered
to because #ifdef was used so it was always on, and closing files
multiple times should be avoided unconditionally.
2022-09-19 23:44:50 +10:00
Jim Mussared
15d0615d5c py/objmodule: Add support for __dict__.
This matches class `__dict__`, and is similarly gated on
MICROPY_CPYTHON_COMPAT. Unlike class though, because modules's globals are
actually dict instances, the result is a mutable dictionary.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:22:46 +10:00
Jim Mussared
d94141e147 py/persistentcode: Introduce .mpy sub-version.
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.

With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version.  A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION.  This allows 3
additional updates to the native ABI per bytecode revision.

The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-09-19 23:19:55 +10:00
Jim Mussared
6da41b5900 py/obj: Merge getiter and iternext mp_obj_type_t slots.
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.

This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
  type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
  to a mp_getiter_iternext_custom_t instance, which then defines both
  getiter and iternext.

And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.

Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:13 +10:00
Jim Mussared
6c376a9306 tests/extmod/uasyncio_heaplock.py: Force SKIP on stackless.
This is a latent issue that wasn't caught by CI because there was no
configuration that had both stackless+uasyncio.

The previous check to skip with stackless builds only worked when the
bytecode emitter was used by default.  Force the check to use the bytecode
emitter.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:41:02 +10:00
Andrew Leech
4e0964b59f extmod/vfs: Add finaliser to ilistdir to close directory handle.
When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally.  Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.

If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed.  In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.

Fixes issues #6568 and #8506.
2022-09-13 13:00:42 +10:00
Jeff Epler
e90b85cc98 extmod/modure: Convert byte offsets to unicode indices when necessary.
And add a test.

Fixes issue #9202.

Signed-off-by: Jeff Epler <jepler@gmail.com>
2022-09-06 17:08:18 +10:00
Takeo Takahashi
2f2fd36713 tests/renesas-ra: Update pin test to support all boards.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
2022-08-31 12:00:41 +10:00
Jim Mussared
bd4e45fd68 tests/unicode: Add test for invalid utf-8 file contents.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:47:27 +10:00
Jim Mussared
c616721b1a extmod/modframebuf: Improve poly-fill boundary pixels.
Rather than drawing the entire boundary to catch missing pixels, just
detect the cases where boundary pixels are skipped during node calculation
and pre-emptively draw them then.

This adds 72 bytes on PYBV11, but makes filled poly() 20% faster.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-19 23:31:28 +10:00
Mat Booth
04a655c744 extmod/modframebuf: Add polygon drawing methods.
Add method for drawing polygons.

For non-filled polygons, uses the existing line-drawing code to render
arbitrary polygons using the given coords list, at the given x,y position,
in the given colour.

For filled polygons, arbitrary closed polygons are rendered using a fast
point-in-polygon algorithm to determine where the edges of the polygon lie
on each pixel row.

Tests and documentation updates are also included.

Signed-off-by: Mat Booth <mat.booth@gmail.com>
2022-08-19 23:31:28 +10:00
Peter Hinch
42ec9703a0 extmod/modframebuf: Add ellipse drawing method. 2022-08-19 23:31:28 +10:00
Damien George
cbc9f944c4 tests,tools: Update path to unix micropython executable.
These were missed by 47c84286e8

Signed-off-by: Damien George <damien@micropython.org>
2022-08-18 11:47:58 +10:00
Scott Shawcroft
7717ab8e13
Merge pull request #6754 from jepler/check-read-utf8
When reading data from a file into a str, check if it's utf-8
2022-08-15 13:48:43 -07:00
Dan Ellis
6f4d424f46 py/formatfloat: Use pow(10, e) instead of pos/neg_pow lookup tables.
Rework the conversion of floats to decimal strings so it aligns precisely
with the conversion of strings to floats in parsenum.c.  This is to avoid
rendering 1eX as 9.99999eX-1 etc.  This is achieved by removing the power-
of-10 tables and using pow() to compute the exponent directly, and that's
done efficiently by first estimating the power-of-10 exponent from the
power-of-2 exponent in the floating-point representation.

Code size is reduced by roughly 100 to 200 bytes by this commit.

Signed-off-by: Dan Ellis <dan.ellis@gmail.com>
2022-08-12 23:53:34 +10:00
Jeff Epler
606c75ab62
test must read data in binary mode 2022-08-12 08:34:33 -05:00
Ned Konz
5543b2a9cc extmod/uasyncio: Add clear method to ThreadSafeFlag.
This is useful in situations where the ThreadSafeFlag is reused and needs
to be cleared of any previous, unwanted event.

For example, clear the flag at the start of an operation, trigger the
operation (eg an I2C write), then (a)wait for an external event to set the
flag (eg a pin IRQ).  Further events may trigger the flag again but these
are unwanted and should be cleared before the next cycle starts.
2022-08-12 17:06:28 +10:00
Jim Mussared
f694058f2b tests/extmod/ubinascii: Add tests for bytes.hex etc.
Also make the sep test not micropython-specific.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-12 12:44:30 +10:00
Andrew Leech
f7f56d4285 py/objstr: Consolidate methods for str/bytes/bytearray/array.
This commit adds the bytes methods to bytearray, matching CPython.  The
existing implementations of these methods for str/bytes are reused for
bytearray with minor updates to match CPython return types.

For details on the CPython behaviour see
https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations

The work to merge locals tables for str/bytes/bytearray/array was done by
@jimmo.  Because of this merging of locals the change in code size for this
commit is mostly negative:

       bare-arm:    +0 +0.000%
    minimal x86:   +29 +0.018%
       unix x64:  -792 -0.128% standard[incl -448(data)]
    unix nanbox:  -436 -0.078% nanbox[incl -448(data)]
          stm32:   -40 -0.010% PYBV10
         cc3200:   -32 -0.017%
        esp8266:   -28 -0.004% GENERIC
          esp32:   -72 -0.005% GENERIC[incl -200(data)]
         mimxrt:   -40 -0.011% TEENSY40
     renesas-ra:   -40 -0.006% RA6M2_EK
            nrf:   -16 -0.009% pca10040
            rp2:   -64 -0.013% PICO
           samd:  +148 +0.105% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2022-08-11 23:18:02 +10:00
Damien George
d53c3b6ade unix/variants: Remove variant suffix from executable filename.
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-11 13:34:34 +10:00
Daniel Jour
c7aa6a2c73 tests/run-tests.py: Provide better default MPYCROSS value for Windows. 2022-08-11 13:34:04 +10:00
Daniel Jour
47c84286e8 all: Fix paths to mpy-cross and micropython binaries.
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory.  Thus
some paths had to be adjusted.
2022-08-11 13:31:13 +10:00
Dan Ellis
9b5e00fcc5 py/formatfloat: Format all whole-number floats exactly.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place.  This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point.  The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.

Fixes issue #4212.

Signed-off-by: Dan Ellis dan.ellis@gmail.com
2022-08-09 13:43:47 -04:00
David Lechner
6baeded322 py/runtime: Fix crash in star arg unpacking.
The reallocation trigger for unpacking star args with unknown length
did not take into account the number of fixed args remaining. So it was
possible that the unpacked iterators could take up exactly the memory
allocated then nothing would be left for fixed args after the star args.
This causes a segfault crash.

This is fixed by taking into account the remaining number of fixed args
in the check to decide whether to realloc yet or not.

Signed-off-by: David Lechner <david@pybricks.com>
2022-08-06 11:32:58 -05:00
Damien George
963e599ec0 tests/cpydiff: Fix formatting of code snippet to use double quotes.
Signed-off-by: Damien George <damien@micropython.org>
2022-07-29 12:23:05 +10:00
Dan Ellis
f9cbe6bc47 py/formatfloat: Format all whole-number floats exactly.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place.  This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point.  The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.

Fixes issue #4212.

Signed-off-by: Dan Ellis dan.ellis@gmail.com
2022-07-26 22:23:47 +10:00
Jim Mussared
b22abcdbbe extmod/uasyncio: Handle gather with no awaitables.
This previously resulted in gather() yielding but with no way to be
resumed.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-26 18:16:19 +10:00
Jim Mussared
e65d1e69e8 py/modio: Remove FileIO and TextIOWrapper from io module.
On ports with more than one filesystem, the type will be wrong, for example
if using LFS but FAT enabled, then the type will be FAT.  So it's not
possible to use these classes to identify a file object type.

Furthermore, constructing an io.FileIO currently crashes on FAT, and
make_new isn't supported on LFS.

And the io.TextIOWrapper class does not match CPython at all.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-26 17:58:01 +10:00
Damien George
c0fa903d6b py/compile: Support large integers in inline-asm data directive.
Fixes issue #8956.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-26 12:24:50 +10:00
Damien George
4fe3e493b1 py/obj: Make mp_obj_get_complex_maybe call mp_obj_get_float_maybe first.
This commit simplifies mp_obj_get_complex_maybe() by first calling
mp_obj_get_float_maybe() to handle the cases corresponding to floats.
Only if that fails does it attempt to extra a full complex number.

This reduces code size and also means that mp_obj_get_complex_maybe() now
supports user-defined classes defining __float__; in particular this allows
user-defined classes to be used as arguments to cmath-module function.

Furthermore, complex_make_new() can now be simplified to directly call
mp_obj_get_complex(), instead of mp_obj_get_complex_maybe() followed by
mp_obj_get_float().  This also improves error messages from complex with
an invalid argument, it now raises "can't convert <type> to complex" rather
than "can't convert <type> to float".

Signed-off-by: Damien George <damien@micropython.org>
2022-07-25 16:11:26 +10:00
Andrew Leech
1e87b56219 py/obj: Add support for __float__ and __complex__ functions. 2022-07-25 14:23:34 +10:00
Carlosgg
b41cfea02a extmod/modussl_mbedtls: Implement cert_reqs and cadata arguments.
Add cert_reqs and cadata keyword-args to ssl.wrap_socket() and
ssl.CERT_NONE, ssl.CERT_OPTIONAL, ssl.CERT_REQUIRED constants to allow
certificate validation.

CPython doesn't accept cadata in ssl.wrap_socket(), but it does in
SSLContext.load_verify_locations(), so we use this name to at least match
the same name in load_verify_locations().

Add docs for these new arguments, as well as docs for the existing
server_hostname argument which is important for certificate validation.

Tests are added as well.

Signed-off-by: Carlos Gil <carlosgilglez@gmail.com>
2022-07-20 16:46:04 +10:00
Damien George
18ecc29bb8 tests/extmod/ussl_basic: Make test run on axtls and mbedtls.
Fixes issue #4364.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-18 18:12:39 +10:00
Lars Haulin
5bf3765631 py/objnamedtuple: Fix segfault with empty namedtuple.
The empty tuple is usually a constant object, but named tuples must be
allocated to allow modification.  Added explicit allocation to fix this.

Also added a regression test to verify creating an empty named tuple works.

Fixes issue #7870.

Signed-off-by: Lars Haulin <lars.haulin@gmail.com>
2022-07-13 16:25:35 +10:00
Jim Mussared
9714a0ead5 py/emitnative: Fix STORE_ATTR viper code-gen when value is not a pyobj.
There was a missing call to MP_F_CONVERT_NATIVE_TO_OBJ.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-12 17:18:27 +10:00
Jeff Epler
8d816db108
update expected results of help(modules) 2022-07-07 13:42:03 -05:00
Damien George
932556d5fc tests/micropython: Add test for builtin execfile() function.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-29 12:48:42 +10:00
Angus Gratton
e024a4c59c tests: Fix run-perfbench parsing "no matching params" case.
Signed-off-by: Angus Gratton <gus@projectgus.com>
2022-06-28 14:22:06 +10:00
Angus Gratton
ad308bc322 tests: Add an explanation of run-perfbench.py.
Also changes this file to a Markdown file.

Signed-off-by: Angus Gratton <gus@projectgus.com>
2022-06-28 14:21:41 +10:00
Angus Gratton
5568c324ba tests/perf_bench: Add some configurations for N=32, M=10.
For STM32L072 and similar, very low end targets.

The other perf_bench tests run out of memory, crash, or fail on
prerequisite features.

Signed-off-by: Angus Gratton <gus@projectgus.com>
2022-06-28 10:32:18 +10:00
Damien George
e22b7fb4af py/objfun: Support function attributes on native functions.
Native functions can just reuse the bytecode function attribute code.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-25 00:22:15 +10:00
Damien George
268ec1e3eb tests/basics: Add tests for __name__ and __globals__ attrs on closures.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-24 23:55:13 +10:00
Damien George
db7682e02d extmod/uasyncio: Implement stream read(-1) to read all data up to EOF.
Fixes issue #6355.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-24 17:04:57 +10:00
Damien George
2a2589738c tests/extmod: Add heap-lock test for stream writing.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-24 17:00:24 +10:00
Damien George
61ce260ff7 py/parsenum: Fix parsing of complex "j" and also "nanj", "infj".
Prior to this commit, complex("j") would return 0j, and complex("nanj")
would return nan+0j.  This commit makes sure "j" is tested for after
parsing the number (nan, inf or a decimal), and also supports the case of
"j" on its own.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-23 11:46:47 +10:00
Jim Mussared
0172292762 py/parsenum: Support parsing complex numbers of the form "a+bj".
To conform with CPython.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-06-23 11:46:47 +10:00
David Lechner
a565811f23 extmod/modbtree: Use buffer protocol for keys/values.
This changes the btree implementation to use the buffer protocol for
reading key/values in all methods.  `str` and `bytes` objects are not the
only bytes-like objects that could be used.

Documentation and tests are also updated.

Addresses issue #8748.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-21 00:44:49 +10:00
Damien George
a506335524 py/emit: Suppress unreachable bytecode/native code that follows jump.
This new logic tracks when an unconditional jump/raise occurs in the
emitted code stream (bytecode or native machine code) and suppresses all
subsequent code, until a label is assigned.  This eliminates a lot of
cases of dead code, with relatively simple logic.

This commit combined with the previous one (that removed the existing
dead-code finding logic) has the following code size change:

       bare-arm:   -16 -0.028%
    minimal x86:   -60 -0.036%
       unix x64:  -368 -0.070%
    unix nanbox:   -80 -0.017%
          stm32:  -204 -0.052% PYBV10
         cc3200:    +0 +0.000%
        esp8266:  -232 -0.033% GENERIC
          esp32:  -224 -0.015% GENERIC[incl -40(data)]
         mimxrt:  -192 -0.054% TEENSY40
     renesas-ra:  -200 -0.032% RA6M2_EK
            nrf:   +28 +0.015% pca10040
            rp2:  -256 -0.050% PICO
           samd:   -12 -0.009% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 22:28:18 +10:00
Damien George
e85a096302 py/emit: Remove logic to detect last-emit-was-return-value.
This optimisation to remove dead code is not as good as it could be.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 22:28:18 +10:00
Damien George
182256dc13 tests/stress: Adjust bytecode_limit test so it can SKIP if no memory.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-08 15:00:59 +10:00
Damien George
1d143cec63 tests/basics: Add .exp file for sys.tracebacklimit test.
The sys.tracebacklimit feature has changed semantics a bit from CPython 3.7
(in the way it modifies the output), so provide a .exp file for the test so
it doesn't rely on CPython.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George
b37b578214 py/persistentcode: Remove remaining native qstr linking support.
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table.  The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py.  But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.

This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table.  mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:19:55 +10:00
Andrew Leech
b7a39ad2d1 tests/run-multitests.py: Read IP address from boot nic if available.
This works if your network is pre-configured in boot.py as an object called
"nic".  Without this, multitests expects to access the WLAN/LAN class which
isn't always correct.

Signed-off-by: Andrew Leech <andrew@alelec.net>
2022-06-03 14:35:37 +10:00
Andrew Leech
73a1ea8812 tests/net_inet: Remove broken api.telegram.org from tests.
Signed-off-by: Andrew Leech <andrew@alelec.net>
2022-06-03 14:34:29 +10:00
Takeo Takahashi
19c680ff57 test/renesas-ra: Remove unsupported feature test of Pin.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
2022-06-03 10:58:16 +10:00
Damien George
a1afb337d2 extmod/uasyncio: Fix edge case for cancellation of wait_for.
This fixes the cases where the task being waited on finishes just before or
just after the wait_for itself is cancelled.

Fixes issue #8717.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-02 17:14:20 +10:00
Damien George
065df5568c tests: Move native while test from pybnative to micropython.
And make it so this test can run on any target.

LED and time testing has been removed from this test, that can now be
tested using: ./run-tests.py --via-mpy --emit native.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 12:54:43 +10:00
Damien George
20d9f3409a tests/run-tests.py: Add rp2 test target.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 12:54:43 +10:00
Damien George
80a86c48e3 tests/micropython: Make import_mpy_native test run on all architectures.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 12:54:43 +10:00
Damien George
3180113aef tests/micropython: Make import_mpy_native_gc run on ARMv6-M and above.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 12:23:49 +10:00
robert-hh
dd35f76db3 tests/run-multitests.py: Use LAN for IP address if WLAN doesn't exist.
This allows running the test on boards with just a LAN interface.

Fixes issue #8681.
2022-05-24 13:21:05 +10:00
iabdalkader
beeb250d58 tests/multi_net: Fix TCP accept test when using system error numbers.
If a port is not using internal error numbers, which match both lwIP and
Linux error numbers, ENTOCONN from standard libraries errno.h equals 128,
not 107.
2022-05-24 13:15:22 +10:00
Damien George
5fa8ea1b8b tests/extmod: Change expected errno code from 36 to 30 in VfsLfs2 test.
Errno 30 is EROFS, which is now the correct value reported by littlefs 2.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-24 12:52:00 +10:00
David Lechner
d42d35f56d tests/run-tests.py: Enable -X realtime option for macOS tests.
This enables the new `-X realtime` runtime option when running tests on
macOS.  This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers.  This
makes tests that depend on the passage of time more likely to succeed.

CI tests that were disabled because of this are now enabled again.

Signed-off-by: David Lechner <david@pybricks.com>
2022-05-24 00:52:44 +10:00
Damien George
7d3204783a tests/run-tests.py: Handle case where mpy-cross fails to compile script.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 15:45:21 +10:00
Damien George
a8492253c1 tests/basics: Unlock heap if skipping nanbox small-int test.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 15:45:16 +10:00
Damien George
54ab9d23e9 tests/run-perfbench.py: Allow running tests via mpy and native emitter.
The performance benchmark tests now support `--via-mpy` and `--emit native`
on remote targets.  For example:

    $ ./run-perfbench.py -p --via-mpy --emit native 100 100

Signed-off-by: Damien George <damien@micropython.org>
2022-05-19 17:31:56 +10:00
Damien George
1786dacc83 tests/run-tests.py: Allow running tests via mpy-cross on remote targets.
This adds support for the `--via-mpy` and `--emit native` options when
running tests on remote targets (via pyboard.py).  It's now possible to do:

    $ ./run-tests.py --target pyboard --via-mpy
    $ ./run-tests.py --target pyboard --via-mpy --emit native

Signed-off-by: Damien George <damien@micropython.org>
2022-05-19 17:31:56 +10:00
Damien George
079f3e5e5b py/parse: Allow all constant objects to be used in "X = const(o)".
Now that constant tuples are supported in the parser, eg (1, True, "str"),
it's a small step to allow anything that is a constant to be used with the
pattern:

    from micropython import const

    X = const(obj)

This commit makes the required changes to allow the following types of
constants:

    from micropython import const

    _INT = const(123)
    _FLOAT = const(1.2)
    _COMPLEX = const(3.4j)
    _STR = const("str")
    _BYTES = const(b"bytes")
    _TUPLE = const((_INT, _STR, _BYTES))
    _TUPLE2 = const((None, False, True, ..., (), _TUPLE))

Prior to this, only integers could be used in const(...).

Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 16:18:35 +10:00
Damien George
761d2f6741 tests/micropython: Add more test cases for native generators.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 15:23:11 +10:00
Damien George
80938839c1 tests/extmod: Use bytearray instead of bytes for uctypes test.
Because the test modifies the (now) bytearray object, and if it's a bytes
object it's not guaranteed that it can be modified, or that this constant
object isn't used elsewhere.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 15:23:11 +10:00
Damien George
b3d0f5f67c tests/micropython: Fully unlink nested list in extreme exc test.
To make sure there are no dangling references to the lists, and the GC can
reclaim heap memory.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 09:16:33 +10:00
Damien George
90682f43af py/compile: Allow new qstrs to be allocated at all compiler passes.
Prior to this commit, all qstrs were required to be allocated (by calling
mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one).  But
this is an unnecessary restriction, which is lifted by this commit.
Lifting the restriction simplifies the compiler because it can allocate
qstrs in later passes.

This also generates better code, because in some cases (eg when a variable
is closed over) the scope of an identifier is not known until a bit later
and then the identifier no longer needs its qstr allocated in the global
table.

Code size is reduced for all ports with this commit.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 23:39:22 +10:00
Damien George
acfc3bbdf8 tests/multi_net: Skip SSL test if relevant modules aren't available.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George
ab0a8f3086 tests/run-tests.py: Exclude settrace tests when using native emitter.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George
5f650b7b7a tests/thread: Use less resources for stress_aes if settrace enabled.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George
6f68a8c240 tests/run-perfbench.py: Return error code if any test fails on target.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:06:41 +10:00
Damien George
d7cf8a3b9d tests/perf_bench: Update .mpy file header to remove old unicode flag.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:06:41 +10:00
Damien George
c49d5207e9 py/persistentcode: Remove unicode feature flag from .mpy file.
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.

The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.

With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this).  The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.

This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Damien George
8aa254c369 tests: Fix tests to use sys.implementation._mpy.
The field was renamed to _mpy in 59c5d41611

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 09:46:40 +10:00
Damien George
fca5701f74 py/malloc: Introduce m_tracked_calloc, m_tracked_free functions.
Enabled by MICROPY_TRACKED_ALLOC.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 10:31:50 +10:00
Damien George
c90f097519 tests/extmod: Increase timing on uasyncio tests to make more reliable.
Non-real-time systems like Windows, Linux and macOS do not have reliable
timing, so increase the sleep intervals to make these tests more likely to
pass.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-03 22:53:12 +10:00
Damien George
590de399f0 py/emitcommon: Don't implicitly close class vars that are assigned to.
When in a class body or at the module level don't implicitly close over
variables that have been assigned to.

Fixes issue #8603.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-03 16:38:43 +10:00
Takeo Takahashi
3717d599e2 tests/run-tests.py: Update for renesas-ra port.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
2022-04-29 12:29:08 +09:00
Takeo Takahashi
4753913253 tests/renesas-ra: Add tests for renesas-ra port.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
2022-04-29 12:29:07 +09:00
Damien George
6bec5c4da5 tests/cmdline: Add test for REPL auto-indent.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-22 17:00:16 +10:00
Damien George
28e7e15c0a extmod/uasyncio: Fix bug with task ending just after gather is cancel'd.
This fixes a bug where the gather is cancelled externally and then one of
its sub-tasks (that the gather was waiting on) finishes right between the
cancellation being queued and being executed.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-21 14:25:17 +10:00
Damien George
8631753ff4 tests/run-tests.py: Add timeout for running PC-based MicroPython test.
So the test suite runs to completion, even if the interpreter locks up.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-21 14:25:17 +10:00
Damien George
ef1c2cdab0 tests/extmod/uasyncio_gather: Make double-raise gather test reliable.
This double-raise test could fail when task[0] raises and stops the gather
before task[1] raises, then task[1] is left to raise later on and spoil the
test.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-20 19:32:49 +10:00
Jon Bjarni Bjarnason
1ded8a2977 py/objtype: Convert result of user __contains__ method to bool.
Per https://docs.python.org/3/reference/expressions.html#membership-test-operations

    For user-defined classes which define the contains() method, x in y
    returns True if y.contains(x) returns a true value, and False
    otherwise.

Fixes issue #7884.
2022-04-20 15:44:46 +10:00
Damien George
865b61dac2 tests/micropython: Add tests that const tuples don't use the heap.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-15 00:17:02 +10:00
Damien George
999abbb8b5 tests/perf_bench: Update import tests for changes to .mpy consts.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:14 +10:00
Damien George
35c0cff92b py/parse: Add MICROPY_COMP_CONST_TUPLE option to build const tuples.
This commit adds support to the parser so that tuples which contain only
constant elements (bool, int, str, bytes, etc) are immediately converted to
a tuple object.  This makes it more efficient to use tuples containing
constant data because they no longer need to be created at runtime by the
bytecode (or native code).

Furthermore, with this improvement constant tuples that are part of frozen
code are now able to be stored fully in ROM (this will be implemented in
later commits).

Code size is increased by about 400 bytes on Cortex-M4 platforms.

See related issue #722.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 23:52:12 +10:00
Damien George
24bc1f61f9 py/parse: Print const object value in mp_parse_node_print.
To give more information when printing the parse tree.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-14 22:45:42 +10:00
Christian Zietz
d6c59c9d72 tests/inlineasm: Add test for PUSH LR and POP PC. 2022-04-11 15:35:42 +10:00
Jon Bjarni Bjarnason
919f696ad2 extmod/modusocket: Implement optional socket.listen backlog argument.
This follows the CPython change: https://bugs.python.org/issue21455

Socket listen backlog defaults to 2 if not given, based on most bare metal
targets not having many resources for a large backlog.  On UNIX it defaults
to SOMAXCONN or 128, whichever is less.
2022-04-11 15:26:47 +10:00
Mark
8ed7b114cd
Merge branch 'main' into uzlib-module 2022-04-03 11:48:37 -05:00
Damien George
71344c15f4 tests/pyb: Update CAN tests to match revised CAN API.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-02 22:46:31 +11:00
Damien George
1dbf393962 tests/basics/fun_callstardblstar: Add test for large arg allocation.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-01 09:20:42 +11:00
Damien George
bd556b6996 py: Fix compiling and decoding of *args at large arg positions.
There were two issues with the existing code:

1. "1 << i" is computed as a 32-bit number so would overflow when
   executed on 64-bit machines (when mp_uint_t is 64-bit).  This meant that
   *args beyond 32 positions would not be handled correctly.

2. star_args must fit as a positive small int so that it is encoded
   correctly in the emitted code.  MP_SMALL_INT_BITS is too big because it
   overflows a small int by 1 bit.  MP_SMALL_INT_BITS - 1 does not work
   because it produces a signed small int which is then sign extended when
   extracted (even by mp_obj_get_int_truncated), and this sign extension
   means that any position arg after *args is also treated as a star-arg.
   So the maximum bit position is MP_SMALL_INT_BITS - 2.  This means that
   MP_OBJ_SMALL_INT_VALUE() can be used instead of
   mp_obj_get_int_truncated() to get the value of star_args.

These issues are fixed by this commit, and a test added.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-01 09:20:42 +11:00
David Lechner
47685180f0 tests/basics/fun_callstardblstar: Add coverage test.
This fixes code coverage for the case where a *arg without __len__ is
unpacked and uses exactly the amount of memory that was allocated for
kw args. This triggers the code branch where the memory for the kw args
gets reallocated since it was used already by the *arg unpacking.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 17:01:15 +11:00
David Lechner
783b1a868f py/runtime: Allow multiple *args in a function call.
This is a partial implementation of PEP 448 to allow unpacking multiple
star args in a function or method call.

This is implemented by changing the emitted bytecodes so that both
positional args and star args are stored as positional args.  A bitmap is
added to indicate if an argument at a given position is a positional
argument or a star arg.

In the generated code, this new bitmap takes the place of the old star arg.
It is stored as a small int, so this means only the first N arguments can
be star args where N is the number of bits in a small int.

The runtime is modified to interpret this new bytecode format while still
trying to perform as few memory reallocations as possible.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 16:59:30 +11:00
David Lechner
1e99d29f36 py/runtime: Allow multiple **args in a function call.
This is a partial implementation of PEP 448 to allow multiple ** unpackings
when calling a function or method.

The compiler is modified to encode the argument as a None: obj key-value
pair (similar to how regular keyword arguments are encoded as str: obj
pairs).  The extra object that was pushed on the stack to hold a single **
unpacking object is no longer used and is removed.

The runtime is modified to decode this new format.

Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31 16:54:00 +11:00
Damien George
90aaf2dbef extmod/uasyncio: Fix gather cancelling and handling of exceptions.
The following fixes are made:
- cancelling a gather now cancels all sub-tasks of the gather (previously
  it would only cancel the first)
- if any sub-task of a gather raises an exception then the gather finishes
  (previously it would only finish if the first sub-task raised)

Fixes issues #5798, #7807, #7901.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-30 16:07:44 +11:00
Damien George
3e70be8ee9 tests/extmod: Update I2S rate test to work on mimxrt.
Tested on Teensy 4.0.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-30 14:14:57 +11:00
Damien George
7266285845 tests/extmod: Add test for machine.I2S data rate.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-29 11:44:08 +11:00
Damien George
acd2c5c834 py/emitbc: Add check for bytecode jump offset overflow.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 15:41:51 +11:00
Damien George
538c3c0a55 py: Change jump opcodes to emit 1-byte jump offset when possible.
This commit introduces changes:

- All jump opcodes are changed to have variable length arguments, of either
  1 or 2 bytes (previously they were fixed at 2 bytes).  In most cases only
  1 byte is needed to encode the short jump offset, saving bytecode size.

- The bytecode emitter now selects 1 byte jump arguments when the jump
  offset is guaranteed to fit in 1 byte.  This is achieved by checking if
  the code size changed during the last pass and, if it did (if it shrank),
  then requesting that the compiler make another pass to get the correct
  offsets of the now-smaller code.  This can continue multiple times until
  the code stabilises.  The code can only ever shrink so this iteration is
  guaranteed to complete.  In most cases no extra passes are needed, the
  original 4 passes are enough to get it right by the 4th pass (because the
  2nd pass computes roughly the correct labels and the 3rd pass computes
  the correct size for the jump argument).

This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.

The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.

Code size is reduced for builds that include a decent amount of frozen
bytecode.  ARM Cortex-M builds without any frozen code increase by about
350 bytes.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-28 15:41:38 +11:00
gamblor21
0d3f45bef8 Remove move DecompIO files 2022-03-27 16:18:56 -05:00
David Lechner
e7a92c0e69 tests/cmdline/cmd_showbc: Fix spelling of sequence. 2022-03-25 12:11:17 +11:00
foamyguy
255fdf8eba remove unused import 2022-03-19 11:36:23 -05:00
foamyguy
5db7e33237 color index test script for vectorio shapes. 2022-03-19 11:33:55 -05:00
Damien George
63f0e700f4 extmod/modure: Set subject begin_line so ^ doesn't match interior.
Fixes issue #8402.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-16 12:21:00 +11:00
Damien George
3c7cab4e98 py/parse: Put const bytes objects in parse tree as const object.
Instead of as an intermediate qstr, which may unnecessarily intern the data
of the bytes object.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-16 00:41:10 +11:00
Scott Shawcroft
00dcf6bd03
Tweaks from review 2022-03-11 10:51:50 -08:00
Damien George
ac2293161e py/modsys: Add optional mutable attributes sys.ps1/ps2 and use them.
This allows customising the REPL prompt strings.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:58:33 +11:00
Damien George
cac939ddc3 py/modsys: Add optional sys.tracebacklimit attribute.
With behaviour as per CPython.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:43:21 +11:00
Damien George
d470c5a5ba tests/extmod/vfs_posix.py: Only test statvfs if it exists.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 00:41:03 +11:00
Scott Shawcroft
96f5eec2ee
Add Teensy 4.1 power pin and fix SWD for DEBUG=1 2022-03-08 17:17:07 -08:00
Scott Shawcroft
83593a1558
Start of USB host API
This allows you to list and explore connected USB devices. It
only stubs out the methods to communicate to endpoints. That will
come in a follow up once TinyUSB has it. (It's in progress.)

Related to #5986
2022-03-07 18:07:25 -08:00
Damien George
3440201e2e tests/micropython: Switch from set.pop to raise-0 to test exc strings.
To not rely on sets, which are an optional feature.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-07 16:48:35 +11:00
Damien George
9a8ee6a5df tests/run-tests.py: Include test files ending in _set as set tests.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-07 16:48:35 +11:00
Damien George
7cd166ff92 tests/basics: Add test for creating small-ints in nan-box builds.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-07 15:25:11 +11:00
Damien George
c4b8dae438 tests/unix: Add coverage test for freezing various objects.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
Dan Halbert
d70f149390 change module list in extra_coverage test results 2022-02-24 22:44:13 -05:00
Damien George
414b59d39c qemu-arm: Add tests for freezing viper and asm_thumb code.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
0a2895b099 tests/perf_bench: Skip bm_chaos test if random.randrange is unavailable.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
gamblor21
ce4a0806b3 Added test coverage 2022-02-21 10:24:13 -06:00
Jeff Epler
a040b6962f
update module list in extra coverage test 2022-02-19 09:38:56 -06:00
Jeff Epler
34d71097ff
Fix import ulab.foo 2022-02-19 09:32:51 -06:00
stijn
ff9c708507 tests/run-tests.py: Skip repl tests when running windows underneath.
Some versions of Python (for instance: the mingw-w64 version which can be
installed on MSYS2) do include a pty module and claim to be posix-like
(os.name == 'posix'), yet the select.select call used in run-tests.py hangs
forever.  To be on the safe side just exclude anything which might be
running on windows.
2022-02-18 15:14:47 +11:00
Jeff Epler
57f46a177c
remove debug print 2022-02-16 14:28:56 -06:00
Jeff Epler
4c9d14b73f Merge remote-tracking branch 'origin/main' into merge-1.18 2022-02-16 11:43:53 -06:00
Jeff Epler
dd7b48f2b3 blacken 2022-02-16 10:31:34 -06:00
Jeff Epler
01cabb0324 Merge tag 'v1.18'
Boosted performance, board.json metadata, more mimxrt, rp2, samd features

This release of MicroPython sees a boost to the overall performance of the
VM and runtime.  This is achieved by the addition of an optional cache to
speed up general hash table lookups, as well as a fast path in the VM for
the LOAD_ATTR opcode on instance types.  The new configuration options are
MICROPY_OPT_MAP_LOOKUP_CACHE and MICROPY_OPT_LOAD_ATTR_FAST_PATH.  As part
of this improvement the MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE option has
been removed, which provided a similar map caching mechanism but with the
cache stored in the bytecode, which made it not useful on bare metal ports.
The new mechanism is measured to be at least as good as the old one,
applies to more map lookups, has a constant RAM overhead, and applies to
native code as well as bytecode.

These performance options are enabled on the esp32, mimxrt, rp2, stm32 and
unix ports.  For esp32 and mimxrt some code is also moved to RAM to further
boost performance.  On stm32, performance increases by about 20% for
benchmarks that are heavy on name lookups, like misc_pystone.py and
misc_raytrace.py.  On esp32 performance can increase by 2-3x, and on mimxrt
it is up to 6x.

All boards in all ports now have a board.json metadata file, which is used
to automatically build firmware and generate a webpage for that board
(among other possibilities).  Auto-build scripts have been added for this
purpose and they build all esp32, mimxrt, rp2, samd and stm32 boards.  The
generated output is available at https://micropython.org/download.

Support for FROZEN_DIR and FROZEN_MPY_DIR has been deprecated for some time
and was finally removed in this release.  Instead of these, FROZEN_MANIFEST
can be used.  The io.resource_stream() function is also removed, replaced
by the pure Python version in micropython-lib.

The search order for importing frozen Python modules is now controlled by
the ".frozen" entry in sys.path.  This string is added by default in the
second position in sys.path.  User code should adjust sys.path depending on
the desired behaviour.  Putting ".frozen" first in sys.path will speed up
importing frozen modules.

A bug in multiple precision integers with bitwise of -0 was fixed in commit
2c139bbf4e.

The platform module has been added to allow querying the compiler and
underlying SDK/HAL/libc version.  This is enabled on esp32, mimxrt and
stm32 ports.

The mpremote tool now supports seek, flush, mkdir and rmdir on PC-mounted
filesystems.  And a help command has been added.

The documentation has seen many additions and improvements thanks (for a
second time) to the Google Season of Docs project.  The rp2 documentation
now includes a reference for PIO assembly instructions, a PIO quick
reference and a PIO tutorial.  The random and stm modules have been
documented, along with sys.settrace, manifest.py files and mpremote.  There
is also now more detail about the differences between MicroPython and
standard Python 3.5 and above.

The esp32 port sees support for ESP32-S3 SoCs, and new boards GENERIC_S3,
ESP32_S2_WROVER, LOLIN_S2_MINI, LOLIN_S2_PICO and UM_FEATHERS2NEO.  The PWM
driver has been improved and now supports all PWM timers and channels, and
the duty_u16() and duty_ns() methods, and it keeps the duty constant when
changing frequency.  The machine.bitstream() function has been improved to
use RMT, with an option to select the original bit-banging implementation.

The mimxrt port gained new hardware features: SDRAM and SD card support, as
well as network integration with a LAN driver.  The machine.WDT class was
added along with the machine.reset_cause(), machine.soft_reset(),
machine.unique_id() add machine.bitstream() functions.  DHT sensor support
was added, and f-strings were enabled.

The rp2 port now has support for networking, and bluetooth using NimBLE.
The Nina-W10 WiFi/BT driver is fully integrated and supported by the new
Arduino Nano RP2040 connect board.  I2S protocol support is added along
with a machine.bitstream() driver and DHT sensor support.  The PWM driver
had a bug fix with the accuracy of setting/getting the frequency, and the
duty value is now retained when changing the frequency.

On the samd port there is now support for the internal flash being a block
device, and for filesystems and the os module.  Pin and LED classes have
been implemented.  There are more time functions, more Python features
enabled, and the help() function is added.  SEEED_WIO_TERMINAL and
SEEED_XIAO board definitions are now available.

The stm32 port now has support for F427, F479 and H7A3(Q)/H7B3(Q) MCUs, and
new board definitions for VCC_GND_H743VI, OLIMEX_H407, MIKROE_QUAIL,
GARATRONIC_PYBSTICK26_F411, STM32H73B3I_DK.  A bug was fixed in the SPI
driver where a SPI transfer could fail if the CYW43 WiFi driver was also
active at the same time.

On the windows port the help() function has been enabled, and support for
build variants added, to match the unix port.

The zephyr port upgraded Zephyr to v2.7.0.

The change in code size since the previous release for various ports is
(absolute and percentage change in the text section):

       bare-arm:  -1520  -2.605%
    minimal x86:  -2256  -1.531%
       unix x64:   -457  -0.089%
    unix nanbox:   -925  -0.204%
          stm32:   +312  +0.079% PYBV10
         cc3200:   -176  -0.096%
        esp8266:   +532  +0.076% GENERIC
          esp32: +27096  +1.820% GENERIC
            nrf:   -212  -0.121% pca10040
            rp2:  +9904  +2.051% PICO
           samd: +35332 +33.969% ADAFRUIT_ITSYBITSY_M4_EXPRESS

The changes that dominate these numbers are:
- bare-arm, minimal: use of new MICROPY_CONFIG_ROM_LEVEL_MINIMUM option and
  subsequent disabling of remaining optional features
- unix, cc3200, nrf: general code size reductions of the core
- stm32: performance improvements, addition of platform module
- esp8266: enabling f-strings
- esp32: use of -O2 instead of -Os
- rp2: machine.I2S and other new hardware features
- samd: filesystem support and other new hardware features

Thanks to everyone who contributed to this release: Alan Dragomirecký,
Alexey Shvetsov, Andrew Leech, Andrew Scheller, Antoine Aubert, Boris
Vinogradov, Chris Boudacoff, Chris Fiege, Christian Decker, Damien George,
Daniel Gorny, Dave Hylands, David Michieli, Emilie Feral, Frédéric Pierson,
gibbonsc, Henk Vergonet, iabdalkader, Ihor Nehrutsa, Jan Hrudka, Jan Staal,
jc_.kim, Jim Mussared, Jonathan Hogg, Laurens Valk, leo chung, Lorenzo
Cappelletti, Magnus von Wachenfeldt, Matt Trentini, Matt van de Werken,
Maureen Helm, Michael Bentley, Michael Buesch, Mike Causer, Mike Teachman,
Mike Wadsten, Ned Konz, NitiKaur, oli, patrick, Patrick Van Oosterwijck,
Peter Boin, Peter Hinch, Peter van der Burg, Philipp Ebensberger, Pooya
Moradi, retsyo, robert-hh, roland van straten, Scott Armitage, Sebastian
Wicki, Seon Rozenblum, Sergei Silnov, Simon Baatz, Stewart Bonnick, stijn,
Tobias Thyrrestrup, Tomas Vanek, YoungJoon Chun.

What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.

Main components
===============

all:
- remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE
- update Python formatting to latest Black version 21.12b0
- remove support for FROZEN_DIR and FROZEN_MPY_DIR

py core:
- parse: simplify parse nodes representing a list
- emitnative: ensure load_subscr does not clobber existing REG_RET
- mpconfig.h: define initial templates for "feature levels"
- vm: add a fast path for LOAD_ATTR on instance types
- map: add an optional cache of (map+index) to speed up map lookups
- builtinimport: forward all debug printing to MICROPY_DEBUG_PRINTER
- add wrapper macros so hot VM functions can go in fast code location
- runtime: fix crash when exc __new__ doesn't return an exc instance
- mpconfig.h: define the "extra" feature level
- mpconfig.h: revert MICROPY_REPL_INFO to disabled at all levels
- gc: add hook to run code during time consuming GC operations
- showbc: print unary-op string when dumping bytecode
- modsys: replace non-ASCII quote char with ASCII char
- runtime: allow types to use both .attr and .locals_dict
- lexer: support nested [] and {} characters within f-string params
- objfun.h: remove obsolete comments about entries in extra_args
- builtinimport: refactor module importing
- showbc: fix printing of raw bytecode header on nanbox builds
- modio: remove io.resource_stream function
- only search frozen modules when '.frozen' is found in sys.path
- mkrules.cmake: set frozen preprocessor defs early
- runtime: allow initialising sys.path/argv with defaults
- mpstate.h: only include sys.path/argv objects in state when enabled
- mpz: fix bugs with bitwise of -0 by ensuring all 0's are positive
- qstr: reset mpstate.qstr_last_chunk before raising an error
- modbuiltins: add additional macro for extending builtins
- mpconfig.h: define MICROPY_PY_USSL_FINALISER only if not defined

extmod:
- machine_i2c: make SoftI2C configurable via macro option
- machine_spi: make SoftSPI configurable via macro option
- modonewire: make _onewire module configurable via macro option
- machine_pwm: factor out machine.PWM bindings to common code
- move modnetwork and modusocket from stm32 to extmod
- modnetwork: add STA_IF and AP_IF constants
- modnetwork: add extended socket state
- modusocket: add read/write stream methods to socket object
- modnetwork: define network interfaces in port config files
- network_cyw43: make consistent use of STA and AP constants
- modnetwork: remove STM32 references
- modnetwork: remove modnetwork socket u_state member
- mpbthci.h: add mp_bluetooth_hci_uart_any prototype
- nimble: add nimble CMake fragment file
- add platform module
- moduplatform: improve implementation for PC ports
- vfs_posix_file: support MP_STREAM_POLL in vfs_posix_file_ioctl
- modbluetooth: add connection interval to gap_connect
- nimble: update to NimBLE v1.4
- nimble: remove workaround for OS_ENOMEM
- uasyncio: fix gather returning exceptions from a cancelled task
- uplatform: remove unused definitions
- uplatform: use generic custom platform string
- network_ninaw10: fix scan list order to match other NICs
- modbluetooth: support gap_connect(None) to cancel a connection
- modure: redirect regex debug printing to mp_printf
- network_ninaw10: fix config of AP mode
- network_ninaw10: disable active connections before connecting
- network_ninaw10: make NIC state persistent
- network_ninaw10: return -1 on timeout from recv/send
- network_ninaw10: make recv/recvfrom interchangeable
- moduplatform: detect xtensa arch
- modusocket: allow setting timeout on unbound sockets
- modusocket: initialise accepted socket state
- network_ninaw10: use socket timeout preset in modusocket
- modbluetooth: fix conditional compilation of ringbuf_put_uuid
- modbluetooth: put declaration of connect_cancel in correct place

shared:
- libc/string0: don't include string.h, and provide __memcpy_chk
- runtime/pyexec: cleanup EXEC_FLAG flag constants

drivers:
- ninaw10: add ublox Nina-W10 WiFi/BT module driver
- lsm6dsox: add LSM6DSOX driver and examples
- neopixel: avoid heap alloc in fill()
- ninaw10: fix BSSID byte order, and add null byte to ESSID
- ninaw10/nina_wifi_drv: fix DNS resolution

mpy-cross: no changes specific to this component/port

lib:
- mynewt-nimble: switch to the MicroPython fork of NimBLE
- asf4: point submodule to latest commit on circuitpython branch
- update pico-sdk to 1.3.0 and tinyusb to 0.12.0
- stm32lib: update library for L4 v1.17.0, new G4, WL, and MMC fixes
- stm32lib: update library for fix to F7 USB HS

Support components
==================

docs:
- library/os.rst: clarify littlefs requirements for block erase
- library/bluetooth.rst: update incorrect link to gatts_write
- make.bat: change Windows output dir from '_build' to 'build'
- library/machine.I2S.rst: specify that I2S.shift args are kw-only
- esp32: explain ESP32 PWM modes, timers, and channels
- rp2: add reference for PIO assembly instructions, and PIO tutorial
- library/random.rst: document the random module
- reference/mpremote.rst: add docs for mpremote
- reference/manifest.rst: add docs for manifest.py files
- library/stm.rst: document the stm module
- esp32/tutorial: add an example of peripheral control via regs
- rp2/general.rst: fix typo with missing spaces
- library/framebuf.rst: adjust dimensions in example
- library/rp2.rst: update function asm_pio_encode to add sideset_opt
- reference/filesystem.rst: add detail on how to use littlefs fuse
- rp2/quickref.rst: add section on PIO
- library/sys.rst: add docs for sys.settrace
- esp8266/tutorial: fix comments of FrameBuffer examples
- library/uasyncio.rst: detail exception behaviour in cancel/timeout
- library/machine.Timer.rst: document 'id' as positional-only arg
- library/machine.SPI.rst: add example SPI usage
- library/machine.Timer.rst: document `period` and `callback` args
- library/machine.Pin.rst: add Pin.ANALOG mode constant
- remove trailing spaces and convert tabs to spaces
- library/sys.rst: add note about '.frozen' as an entry in sys.path
- differences: document details of new PEPs/features in Python 3.5+
- update copyright year range to include 2022
- esp32: update RMT quickref example to match latest code

examples: no changes specific to this component/port

tests:
- perf_bench: use math.log instead of math.log2
- basics: add tests for type-checking subclassed exc instances
- micropython/const.py: add comment about required config for test
- cpydiff: clarify f-string diffs regarding concatenation
- basics/int_big_cmp.py: add more tests for big-int comparison
- extmod: skip uselect_poll_udp when poll() is not available

tools:
- autobuild: add auto build for GENERIC_C3_USB
- ci.sh: use IDF v4.4 as part of esp32 CI and build GENERIC_S3
- autobuild: add the MIMXRT1010_EVK board to autobuild
- ci.sh: use a specific ESP IDF v4.4 commit
- autobuild: add script to generate website board metadata
- dfu.py: make tool work with python3 when parsing DFU files
- autobuild: automatically build all mimxrt, rp2 and samd boards
- autobuild: automatically build all stm32 boards
- mpremote: implement seek and flush in ioctl method
- autobuild: automatically build all esp32 boards
- upip.py: support == to specify exact package version
- makemanifest.py: make str conversion compatible with Python 2
- makemanifest.py: merge make-frozen.py
- mpremote: add mkdir and rmdir to RemoteFS
- mpremote: add help command
- mpremote: add link to mpremote docs URL in help message
- upip.py: skip '.frozen' entry in sys.path for install path
- autobuild: build esp8266 OTA image with GENERIC_1M board
- ci.sh: upgrade Zephyr docker image to v0.21.0
- ci.sh: build zephyr nucleo_wb55rg to test zephyr bluetooth build

CI:
- workflows: use Python 3.8 for macos workflow
- workflows: add new workflow to build ports download metadata

The ports
=========

all ports:
- add board.json for all boards
- add images, features and urls to board.json
- add '.frozen' as the first entry in sys.path
- move '.frozen' to second entry in sys.path

bare-arm port:
- mpconfigport.h: use MICROPY_CONFIG_ROM_LEVEL_MINIMUM
- mpconfigport.h: disable remaining optional features

cc3200 port: no changes specific to this component/port

esp8266 port:
- boards/GENERIC: enable f-strings
- extract qstr from object when comparing keys in config()
- etshal.h: remove unneeded function declarations
- allow building a board to any dest directory

esp32 port:
- boards: add new FeatherS2-Neo board definition
- machine_timer: use tx_update member for IDF 4.4 and above
- add support for ESP32-S3 SoCs
- boards: add new GENERIC_S3 board definition
- machine_hw_spi: fix hardware SPI DMA channels for S2/S3
- boards: add board definition for ESP32-S2-WROVER module
- boards: add LOLIN_S2_MINI ESP32-S2 board
- machine_pwm: add support for all PWM timers and channels
- README: updated readme with req IDF vers for ESP32-S2, C3 and S3
- usb: add USB host connection detection for CDC serial output
- machine_pin: block out IO16 and IO17 when using SPIRAM on ESP32
- mpthreadport: fix TCB cleanup function so thread_mutex is ready
- main: add option for a board to hook code into startup sequence
- split out WLAN code from modnetwork.c to network_wlan.c
- enable optimisations and move code to iRAM to boost performance
- usb: improve speed of USB CDC output
- add specific deploy_s2.md instructions for esp32-s2
- boards/LOLIN_S2_MINI: add image to board.json
- boards: update board and deploy metadata for UM_xxx boards
- usb: further improve speed of USB CDC output
- boards/LOLIN_S2_PICO: add LOLIN_S2_PICO board definition files
- boards/ESP32_S2_WROVER: link to specific deploy_s2 instructions
- support building with latest IDF v5
- in machine_i2s, send null samples in underflow situations
- in machine_i2s, make object reference arrays root pointers
- add SDCard support for S3, and a GENERIC_S3_SPIRAM board
- boards/GENERIC_S3: enable BLE on ESP32 S3
- machine_pwm: implement duty_u16() and duty_ns() PWM methods
- extract qstr from object when comparing keys in config()
- machine_pin: make GPIO 26 usable for S2,S3 if SPIRAM not config'd
- machine_hw_spi: fix SPI default pins reordering on ESP32-S2/S3
- machine_hw_spi: set proper default SPI(id=1) pins on S2,S3 and C3
- machine_hw_spi: set proper default SPI(id=2) pins on S2 and S3
- boards: remove SPI pin defaults from GENERIC S2/S3 boards
- modnetwork: synchronize WiFi AUTH_xxx constants with IDF values
- machine_pwm: keep duty constant when changing frequency
- machine_bitstream: replace bit-bang code with RMT-based driver
- machine_i2s: add support for ESP-IDF 4.4
- machine_bitstream: fix signal duplication on output pins
- esp32: enable platform module with IDF version
- boards/GENERIC_D2WD: build with -Os optimisation
- esp32_rmt: install RMT driver on core 1
- machine_bitstream: reinstate bitstream bit-bang implementation

javascript port: no changes specific to this component/port

mimxrt port:
- sdcard: implement SDCard driver
- machine_bitstream: add bitstream function to machine module
- rework flash configuration
- sdram: add SDRAM support
- eth: add LAN support and integrate the network module
- modmachine: implement machine.WDT() and machine.reset_cause()
- boards: fix the D14/D15 pin assignment of MIMXRT1050/60/64_EVK
- hal: remove duplicate definitions from flexspi_hyper_flash.h
- dma_channel: fix the DMA channel management
- fix cycle counter for time.ticks_cpu() and machine.bitstream()
- add dht_readinto() to the mimxrt module, and freeze dht.py
- extend the help() message and README.md
- mpconfigport.h: enable f-strings
- modmachine: implement soft_reset() and unique_id() functions
- boards/make-pins.py: allow empty lines and comments in pins.csv
- optimize the runtime speed
- enable the platform module
- boards: add the Seeed ARCH MIX board
- boards: update the board.json files and add deploy_xx.md files
- fix mp_hal_quiet_timing_enter()/exit() so timer still runs
- support PWM using the FLEXPWM and QTMR modules
- define UART 0 on MIMXRT boards
- support selection of PHY type and address
- re-enable eth checksum creation by HW
- fix a tiny unnoticed bug in sdcard.c
- add a driver for the DP83848 PHY device
- refactor the reading of the machine id
- enable ticks_cpu at boot time for NDEBUG builds only
- use -Og instead of -O0 for DEBUG builds
- tidy up the board flash related files
- hal: allow readSampleClkSrc to be configured by a board
- enable MICROPY_PY_USSL_FINALISER

minimal port:
- mpconfigport.h: use MICROPY_CONFIG_ROM_LEVEL_MINIMUM
- Makefile: don't force a 32-bit build
- mpconfigport.h: disable features that are not needed

nrf port:
- Makefile: improve Black Magic Probe commands
- main: use VFS helper function to mount fs and chdir

pic16bit port: no changes specific to this component/port

powerpc port: no changes specific to this component/port

qemu-arm port: no changes specific to this component/port

rp2 port:
- mpconfigport.h: enable heapq module
- add support for bluetooth module using NimBLE
- add framework for networking
- mpconfigport.h: use the "extra" feature level
- enable optimisations (comp goto, map cache, fast attr)
- machine_i2s: add I2S protocol support
- add support for Nina-W10 WiFi/BT module
- boards: add support for Arduino Nano RP2040
- machine_bitstream: implement the machine.bitstream driver
- boards: add neopixel.py to manifest.py
- rp2_pio: support exec with sideset
- boards/PIMORONI_PICOLIPO_16MB: fix 16MB flash size
- boards: add PYBSTICK26 RP2040 board definition
- machine_uart: handle and clear UART RX timeout IRQ
- boards/ARDUINO_NANO_RP2040_CONNECT: set default I2C pins
- machine_pwm: fix PWM frequency setting
- machine_pwm: keep duty value when changing the frequency
- add support for DHT11 and DHT22 sensors
- CMakeLists.txt: allow a board to override PICO_BOARD
- boards/GARATRONIC_PYBSTICK26_RP2040: use correct pico-sdk board cfg

samd port:
- integrate latest asf4, add help, more time funcs and uPy features
- samd_soc: allow a board to configure the low-level MCU config
- add internal flash block device, filesystem and uos support
- add Pin and LED classes, and machine.unique_id
- boards/ADAFRUIT_FEATHER_M0_EXPRESS: update for flash and pins
- boards/ADAFRUIT_ITSYBITSY_M4_EXPRESS: update for flash and pins
- boards/MINISAM_M4: update for flash and pins
- boards/ADAFRUIT_TRINKET_M0: update for flash and pins
- boards/SAMD21_XPLAINED_PRO: update for flash and pins
- boards/SEEED_WIO_TERMINAL: add new board definition
- boards/SEEED_XIAO: add new board definition
- README.md: update README to reflect new features and boards

stm32 port:
- pin: enable GPIO clock of pin if it's constructed without init
- main: don't unconditionally enable GPIO A,B,C,D clocks
- boards/VCC_GND_H743VI: add board definition for VCC_GND_H743VI
- boards/OLIMEX_E407: add Ethernet RMII support
- boards/LEGO_HUB_NO6: remove user paths from cc2564 init file
- boards: remove trailing spaces, and add newline at end of file
- add basic support for STM32H750
- add support for H7A3(Q)/H7B3(Q), and STM32H73B3I_DK board defn
- suggest putting code in main.py not boot.py
- boards/make-pins.py: allow a CPU pin to be hidden
- boards/make-pins.py: allow empty lines and comments in pins.csv
- dma: add functions for external users of DMA to enable clock
- enable LOAD_ATTR fast path, and map lookup caching on >M0
- boards: add OLIMEX H407 board definition
- enable platform module
- extended flash filesystem space to 512K on H743 boards
- boards/NUCLEO_H743ZI: enable VfsLfs2 on NUCLEO_H743ZI(2) boards
- boards: add PF11-BOOT0 to stm32f091_af.csv
- machine_i2c: use hardware I2C for STM32H7
- sdram: enforce gcc opt, and use volatile and DSB in sdram_test
- usbd_cdc_interface: allow a board to hook into USBD CDC RX events
- mpbthciport: allow a board to hook BT HCI poll functions
- pendsv: allow a board to add entries for pendsv_schedule_dispatch
- boards: add images to board.json for Adafruit and VCC_GND boards
- uart: fix race conditions and clearing status in IRQ handler
- mpconfigport.h: use the "extra" feature level
- in machine_i2s, send null samples in underflow situations
- in machine_i2s, make object reference arrays root pointers
- led: support an extra 2 LEDs in board configuration
- boards/MIKROE_CLICKER2_STM32: add more detail to board.json
- boards: add new board MikroElektronika Quail, and F427 support
- main: run optional frozen module at boot
- sdio: don't explicitly disable DMA2 on deinit of SDIO
- dma: make DMA2_Stream3 exclusive to SDIO when CYW43 enabled
- boards: build NUCLEO_WB55 and STM32F769DISC without mboot enabled
- boards: add PYBSTICK26 F411 board definition
- boards/NADHAT_PYBF405: rename board to GARATRONIC_NADHAT_F405
- usb: use a table of allowed values to simplify usb_mode get/set
- boards/NUCLEO_WB55: update rfcore_firmwre for new WS
- flashbdev: support generic flash storage config via link symbols
- boards: convert F413,F439,H743,L4xx,WB55 to new flash FS config
- add support for F479 MCUs
- include HAL MMC code in F4 builds
- boards/make-pins.py: use cpu pins to define static alt-fun macros
- boards/NUCLEO_WB55: fix LED ordering
- boards/LEGO_HUB_NO6: set filesystem label as HUB_NO6
- boards: remove stray '+' characters at start of lines in ld files
- boards: remove unused MICROPY_HW_ENABLE_TIMER config
- boards: enable MICROPY_HW_ENABLE_SERVO on various boards
- update L4 code to build with latest stm32lib and L4 HAL 1.17.0
- main: call sdcard_init when only MICROPY_HW_ENABLE_MMCARD enabled
- sdcard: support 8-bit wide SDIO bus
- sdcard: add config option to force MM card capacity
- factoryreset: init vfs flags before calling pyb_flash_init_vfs
- qspi: fix typo in address comment
- boards/make-pins.py: generate empty ADC table if needed
- boards/OLIMEX_H407: fix typo in OLIMEX H407 board.json
- network_wiznet5k: fix build error with wiznet5k and lwip enabled
- enable MICROPY_PY_USSL_FINALISER

teensy port:
- switch to use manifest.py instead of FROZEN_DIR

unix port:
- enable LOAD_ATTR fast path, and map lookup caching
- modusocket: support MP_STREAM_POLL in unix socket_ioctl
- modos: add support for uos.urandom(n)
- coverage: change remaining printf to mp_printf
- Makefile: use -Og instead of -O0 for debug builds

windows port:
- README: remove unsupported Python instructions for Cygwin
- mpconfigport.h: enable help and help("modules")
- add support for build variants to windows port
- run tests via Makefile
- appveyor: build both standard and dev variants
- appveyor: build mpy-cross only once for mingw-w64
- msvc: run qstr preprocessing phase in parallel

zephyr port:
- mphalport.h: remove unused and unimplemented C-level pin API
- increase minimum CMake version to 3.20.0
- update include path to reboot.h
- get UART console device from devicetree instead of Kconfig
- use CONFIG_USB_DEVICE_STACK for conditional USB device support
- upgrade to Zephyr v2.7.0
- modbluetooth_zephyr: provide dummy connect_cancel function
2022-02-15 12:36:26 -06:00
Jeff Epler
adfa6ef019
Fix all natmod tests to run on x64
My testing sequence is

```
git clean -dxf examples
arch=x64; make -C examples/natmod/features1 ARCH=$arch && make -C examples/natmod/features2 ARCH=$arch && make -C examples/natmod/btree ARCH=$arch && make -C examples/natmod/framebuf ARCH=$arch && make -C examples/natmod/uheapq ARCH=$arch && make -C examples/natmod/urandom ARCH=$arch && make -C examples/natmod/ure ARCH=$arch && make -C examples/natmod/uzlib ARCH=$arch
(cd tests && ./run-natmodtests.py "$@" extmod/{btree*,framebuf*,uheapq*,ure*,uzlib*}.py)
```
2022-02-13 11:18:38 -06:00
Damien George
2ea21abae0 tests/extmod/vfs_fat_finaliser.py: Make finalisation more robust.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-12 09:45:32 +11:00
Damien George
e8bc4a3a5b tests/run-perfbench.py: Use SKIP consistently, and increase print width.
A script will print "SKIP" if it wants to be skipped, so the test runner
must also use uppercase SKIP.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-11 22:19:38 +11:00
Damien George
a434705700 tests/perf_bench: Add perf test for yield-from execution.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-11 13:42:00 +11:00
Damien George
75da124cf8 tests/perf_bench: Add perf tests for qstr interning and importing .mpy.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-10 15:25:33 +11:00
Damien George
b33fdbe535 tests/run-perfbench.py: Allow a test to SKIP, and to have a .exp file.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-10 15:25:33 +11:00
iabdalkader
30a022548f tests/multi_net/udp_data.py: Make UDP test more reliable.
The current test depends on a specific number and order of packets to pass,
which can't be reproduced every run due to the unreliable UDP protocol.
This patch adds simple packets sequencing, retransmits with timeouts, and a
packet loss threshold, to make the test more tolerant to UDP protocol
packet drops and reordering.
2022-02-09 14:05:01 +11:00
Damien George
ab2923dfa1 all: Update Python formatting to latest Black version 22.1.0.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-02 16:49:55 +11:00
Dan Halbert
748834c6ba update to py tests and scripts due to black 22.1.0 2022-01-29 17:14:01 -05:00
Christian Decker
2e3a2785cd extmod/modubinascii: Add newline keyword to b2a_base64 function.
This allows encoding things (eg a Basic-Auth header for a request) without
slicing the \n from the string, which allocates additional memory.

Co-authored-by: David Lechner <david@lechnology.com>
2022-01-23 10:18:01 +11:00
stijn
dd6967202a py/modmath: Add math.tau, math.nan and math.inf constants.
Configurable by the new MICROPY_PY_MATH_CONSTANTS option.
2022-01-23 09:28:33 +11:00
iabdalkader
6e8f4eaa52 tests/multi_net/udp_data.py: Allow reusing port before bind. 2022-01-21 13:34:33 +11:00
iabdalkader
e6ddda29ca tests/multi_net: Close accepted sockets when tests are done.
gc_sweep_all() cleans up sockets via the finaliser, but tests should
cleanly free resources they use.
2022-01-21 13:34:20 +11:00
Jeff Epler
037b2c72a1 py/objstr: Support '{:08}'.format("Jan") like Python 3.10.
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.

See CPython version of the issue at https://bugs.python.org/issue27772

Signed-off-by: Jeff Epler <jepler@gmail.com>
2022-01-19 15:34:32 +11:00
Damien George
2c9dc5742a tests/multi_net: Add testing key/cert to SSL server/client test.
So that this tests works with mbedtls.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 17:35:04 +11:00
Damien George
c54717a78f tests/run-multitests.py: Set HOST_IP so tests work between PC and board.
Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 17:35:04 +11:00
Damien George
5df1d8be6c tests/run-multitests.py: Ignore lld_pdu_get_tx_flush_nb msgs from IDF.
BLE still functions correctly even though these messages are sometimes
printed by the IDF.  Ignoring them allows the multi_bluetooth tests to pass
on an esp32 board.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 14:23:24 +11:00
stijn
19d949a866 tests/extmod: Skip uselect_poll_udp when poll() is not available.
This is the same fix as applied in uselect_poll_basic.py.
2022-01-04 15:07:45 +11:00
Dan Halbert
9d2a32d07e countio: add selectable rise and fall detection, pulls 2021-12-31 16:34:58 -05:00
Damien George
aac5a97d08 ports: Move '.frozen' to second entry in sys.path.
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.

But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.

This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).

Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:55:36 +11:00
Damien George
c768704cfd tests/basics/int_big_cmp.py: Add more tests for big-int comparison.
To improve coverage of mpz_cmp and mpn_cmp.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-21 18:00:11 +11:00
Damien George
2c139bbf4e py/mpz: Fix bugs with bitwise of -0 by ensuring all 0's are positive.
This commit makes sure that the value zero is always encoded in an mpz_t as
neg=0 and len=0 (previously it was just len=0).

This invariant is needed for some of the bitwise operations that operate on
negative numbers, because they cannot handle -0.  For example
(-((1<<100)-(1<<100)))|1 was being computed as -65535, instead of 1.

Fixes issue #8042.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-21 18:00:05 +11:00
Jim Mussared
cc23e99f32 py/modio: Remove io.resource_stream function.
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
2021-12-17 23:53:44 +11:00
microDev
7e46387196
add non-native exception in traceback_test
Co-authored-by: Dan Halbert <halbert@adafruit.com>
2021-12-15 20:48:33 +05:30
Jim Mussared
3770fab334 all: Update Python formatting to latest Black version 21.12b0.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-09 12:09:40 +11:00
Dan Halbert
4a4c5d7ab8 formatting updates for updated black 2021-12-05 21:37:00 -05:00
microDev
69faaa52b8
run test for traceback module 2021-12-02 14:55:02 +05:30
Jeff Epler
1654f5fb58
add an aes test
This combines some test vectors from the implementation & a NIST
standards document, plus the code from the docstring.

The test vectors were eyeball-verified.
2021-11-30 15:26:32 -06:00
Jeff Epler
1f1def54b7
add aesio to unix coverage build 2021-11-30 15:25:05 -06:00
Jim Mussared
e99f7b6d25 tests/cpydiff: Clarify f-string diffs regarding concatenation.
Concatenation of any literals (including f-strings) should be avoided.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-11-25 22:09:59 +11:00
Damien George
11ed94797d py/lexer: Support nested [] and {} characters within f-string params.
Signed-off-by: Damien George <damien@micropython.org>
2021-11-25 21:50:58 +11:00
Jeff Epler
ddf24772c3
tests: move bitmaptools files to their own directory 2021-11-24 09:53:19 -06:00
Jeff Epler
492402f23f
these must be treated as manual tests for now 2021-11-20 21:14:31 -05:00
Jeff Epler
db44df5dbf
blend & dither tests 2021-11-20 21:13:56 -05:00
Jeff Epler
de455b5c4e
Add bitmap loading test 2021-11-20 08:52:18 -05:00
Jeff Epler
dca696dec1
some test images for bitmaptools 2021-11-20 08:31:17 -05:00
Damien George
78ab2eeda3 py/showbc: Print unary-op string when dumping bytecode.
Signed-off-by: Damien George <damien@micropython.org>
2021-11-19 17:05:40 +11:00
jc_.kim
19f09414a6 tests/micropython/const.py: Add comment about required config for test.
Expected result of const.py will be matched only when MICROPY_COMP_CONST is
enabled.  For easy understanding, added description at the first of the
test code.
2021-11-17 14:28:20 +11:00
Damien George
43d08688c3 extmod/uasyncio: Fix gather returning exceptions from a cancelled task.
Fixes issue #5882.
2021-11-17 14:11:31 +11:00
Scott Shawcroft
ba2f32e374
Merge pull request #5578 from jepler/disable-mkfs-fat32
oofatfs: Allow fat32 mkfs to be compiled out.  optimize rainbowio & enable everywhere
2021-11-15 10:57:25 -08:00
Alec Delaney
f25d30087c Updated per pre-commit 2021-11-14 14:06:47 -05:00
Jeff Epler
d4dd06904a
add a test of rainbows 2021-11-12 17:31:14 -06:00
Jeff Epler
1cea871c70
enable rainbowio on unix coverage port 2021-11-12 17:31:14 -06:00
Jeff Epler
010a822b8c
tests: run circuitpython tests during make test
.. there are none that aren't skipped now, but it'll change
2021-11-12 17:31:14 -06:00
Jeff Epler
1c73e2e466
tests: Remove a non-useful test 2021-11-12 17:31:13 -06:00
Jeff Epler
167665f8b7
fix build 2021-11-12 11:03:45 -06:00
Jeff Epler
334a9c85ce Merge remote-tracking branch 'origin/main' into gifio 2021-11-04 20:22:00 -05:00
Kattni Rembor
a63af1d04a Update D13 to LED, add PWM example. 2021-10-28 13:30:51 -04:00
Jeff Epler
b0203381db Update module list in test 2021-10-26 08:54:18 -05:00
Mike Wadsten
c3c2c37fbc tests/basics: Add tests for type-checking subclassed exc instances. 2021-10-21 12:42:48 +11:00
Jeff Epler
8d8f83bc05 Merge tag 'v1.17' into merge-1.17
F-strings, new machine.I2S class, ESP32-C3 support and LEGO_HUB_NO6 board

This release of MicroPython adds support for f-strings (PEP-498), with a
few limitations compared to normal Python.  F-strings are essentially
syntactic sugar for "".format() and make formatting strings a lot more
convenient.  Other improvements to the core runtime include pretty printing
OSError when it has two arguments (an errno code and a string), scheduling
of KeyboardInterrupt on the main thread, and support for a single argument
to the optimised form of StopIteration.

In the machine module a new I2S class has been added, with support for
esp32 and stm32 ports.  This provides a consistent API for transmit and
receive of audio data in blocking, non-blocking and asyncio-based
operation.  Also, the json module has support for the "separators" argument
in the dump and dumps functions, and framebuf now includes a way to blit
between frame buffers of different formats using a palette.  A new,
portable machine.bitstream function is also added which can output a stream
of bits with configurable timing, and is used as the basis for driving
WS2812 LEDs in a common way across ports.

There has been some restructuring of the repository directory layout, with
all third-party code now in the lib/ directory.  And a new top-level
directory shared/ has been added with first-party code that was previously
in lib/ moved there.

The docs have seen further improvement with enhancements and additions to
the rp2 parts, as well as a new quick reference for the zephyr port.
The terms master/slave have been replaced with controller/peripheral,
mainly relating to I2C and SPI usage.  And u-module references have been
replaced with just the module name without the u-prefix to help clear up
the intended usage of modules in MicroPython.

For the esp8266 and esp32 ports, hidden networks are now included in WLAN
scan results.  On the esp32 the RMT class is enhanced with idle_level and
write_pulses modes.  There is initial support for ESP32-C3 chips with
GENERIC_C3 and GENERIC_C3_USB boards.

The javascript port has had its Makefile and garbage collector
implementation reworked so it compiles and runs with latest the Emscripten
using asyncify.

The mimxrt port sees the addition of hardware I2C and SPI support, as well
as some additional methods to the machine module.  There is also support
for Hyperflash chips.

The nrf port now has full VFS storage support, enables source-line on
traceback, and has .mpy features consistent with other ports.

For the rp2 port there is now more configurability for boards, and more
boards added.

The stm32 port has a new LEGO_HUB_NO6 board definition with detailed
information how to get this LEGO Hub running stock MicroPython.  There is
also now support to change the CPU frequency on STM32WB MCUs.  And USBD_xxx
descriptor options have been renamed to MICROPY_HW_USB_xxx.

Thanks to everyone who contributed to this release: Amir Gonnen, Andrew
Scheller, Bryan Tong Minh, Chris Wilson, Damien George, Daniel Mizyrycki,
David Lechner, David P, Fernando, finefoot, Frank Pilhofer, Glenn Ruben
Bakke, iabdalkader, Jeff Epler, Jim Mussared, Jonathan Hogg, Josh Klar,
Josh Lloyd, Julia Hathaway, Krzysztof Adamski, Matúš Olekšák, Michael
Weiss, Michel Bouwmans, Mike Causer, Mike Teachman, Ned Konz, NitiKaur,
oclyke, Patrick Van Oosterwijck, Peter Hinch, Peter Züger, Philipp
Ebensberger, robert-hh, Roberto Colistete Jr, Sashkoiv, Seon Rozenblum,
Tobias Thyrrestrup, Tom McDermott, Will Sowerbutts, Yonatan Goldschmidt.

What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.

Main components
===============

all:
- fix signed shifts and NULL access errors from -fsanitize=undefined
- update to point to files in new shared/ directory

py core:
- mpstate: make exceptions thread-local
- mpstate: schedule KeyboardInterrupt on main thread
- mperrno: add MP_ECANCELED error code
- makeqstrdefs.py: don't include .h files explicitly in preprocessing
- mark unused arguments from bytecode decoding macros
- objexcept: pretty print OSError also when it has 2 arguments
- makeversionhdr: add --tags arg to git describe
- vm: simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode
- objexcept: make mp_obj_exception_get_value support subclassed excs
- support single argument to optimised MP_OBJ_STOP_ITERATION
- introduce and use mp_raise_type_arg helper
- modsys: optimise sys.exit for code size by using exception helpers
- objexcept: make mp_obj_new_exception_arg1 inline
- obj: fix formatting of comment for mp_obj_is_integer
- emitnative: reuse need_reg_all func in need_stack_settled
- emitnative: ensure stack settling is safe mid-branch
- runtime: fix bool unary op for subclasses of native types
- builtinimport: fix condition for including do_execute_raw_code()
- mkrules: automatically build mpy-cross if it doesn't exist
- implement partial PEP-498 (f-string) support
- lexer: clear fstring_args vstr on lexer free
- mkrules.mk: do submodule sync in "make submodules"

extmod:
- btstack: add missing call to mp_bluetooth_hci_uart_deinit
- btstack: check that BLE is active before performing operations
- uasyncio: get addr and bind server socket before creating task
- axtls-include: add axtls_os_port.h to customise axTLS
- update for move of crypto-algorithms, re1.5, uzlib to lib
- moduselect: conditionally compile select()
- nimble: fix leak in l2cap_send if send-while-stalled
- btstack/btstack.mk: use -Wno-implicit-fallthrough, not =0
- utime: always invoke mp_hal_delay_ms when >= to 0ms
- modbluetooth: clamp MTU values to 32->UINT16_MAX
- nimble: allow modbluetooth binding to hook "sent HCI packet"
- nimble: add "memory stalling" mechanism for l2cap_send
- uasyncio: in open_connection use address info in socket creation
- modujson: add support for dump/dumps separators keyword-argument
- modlwip: fix close and clean up of UDP and raw sockets
- modbluetooth: add send_update arg to gatts_write
- add machine.bitstream
- modframebuf: enable blit between different formats via a palette

lib:
- tinyusb: update to version 0.10.1
- pico-sdk: update to version 1.2.0
- utils/stdout_helpers: make mp_hal_stdout_tx_strn_cooked efficient
- axtls: switch to repo at micropython/axtls
- axtls: update to latest axtls 2.1.5 wih additional commits
- re1.5: move re1.5 code from extmod to lib
- uzlib: move uzlib code from extmod to lib
- crypto-algorithms: move crypto-algorithms code from extmod to lib
- update README's based on contents of these dirs

drivers:
- neopixel: add common machine.bitstream-based neopixel module
- neopixel: optimize fill() for speed
- neopixel: reduce code size of driver
- cyw43: fix cyw43_deinit so it can be called many times in a row
- cyw43: make wifi join fail if interface is not active

mpy-cross:
- disable stack check when building with Emscripten

Support components
==================

docs:
- library: document new esp32.RMT features and fix wait_done
- library: warn that ustruct doesn't handle spaces in format strings
- esp8266/tutorial: change flash mode from dio to dout
- replace master/slave with controller/peripheral in I2C and SPI
- rp2: enhance quickref and change image to Pico pinout
- rp2: update general section to give a brief technical overview
- library/utime.rst: clarify behaviour and precision of sleep ms/us
- library/uasyncio.rst: document stream readexactly() method
- library/machine.I2S.rst: fix use of sd pin in examples
- zephyr: add quick reference for the Zephyr port
- library/zephyr: add libraries specific to the Zephyr port
- templates: add unix and zephyr quickref links to top-index
- rename ufoo.rst to foo.rst
- replace ufoo with foo in all docs
- library/index.rst: clarify module naming and purpose
- library/builtins.rst: add module title
- library/network.rst: simplify socket import
- add docs for machine.bitstream and neopixel module
- library: fix usage of :term: for frozen module reference
- esp8266: use monospace for software tools
- reference: mention that slicing a memoryview causes allocation

examples: no changes specific to this component/port

tests:
- extmod: make uasyncio_heaplock test more deterministic
- cpydiff/modules_struct_whitespace_in_format: run black
- extmod/ujson: add tests for dump/dumps separators argument
- run-multitests.py: add broadcast and wait facility
- multi_bluetooth/ble_subscribe.py: add test for subscription
- extmod/vfs_fat_finaliser.py: ensure alloc at never-used GC blocks
- basics: split f-string debug printing to separate file with .exp
- pybnative: make while.py test run on boards without pyb.delay

tools:
- autobuild: add scripts to build release firmware
- remove obsolete build-stm-latest.sh script
- ci.sh: run apt-get update in ci_powerpc_setup
- makemanifest.py: allow passing flags to mpy-tool.py
- autobuild: add mimxrt port to build scripts for nightly builds
- pyboard.py: add cmd-line option to make soft reset configurable
- mpremote: swap order of PID and VID in connect-list output
- ci.sh: build unix dev variant as part of macOS CI
- ci.sh: build GENERIC_C3 board as part of esp32 CI
- autobuild: use separate IDF version to build newer esp32 SoCs
- autobuild: add FeatherS2 and TinyS2 to esp32 auto builds
- mpremote: add seek whence for mounted files
- mpremote: raise OSError on unsupported RemoteFile.seek
- autobuild: add the MIMXRT1050_EVKB board to the daily builds
- ci.sh: add mpy-cross build to nrf port
- codeformat.py: include ports/nrf/modules/nrf in code formatting
- gen-cpydiff.py: don't rename foo to ufoo in diff output
- autobuild: add auto build for Silicognition wESP32
- mpremote: fix connect-list in case VID/PID are None
- mpremote: add "devs" shortcut for "connect list"
- mpremote: remove support for pyb.USB_VCP in/out specialisation
- autobuild: don't use "-B" for make, it's already a fresh build
- pyboard.py: move --no-exclusive/--soft-reset out of mutex group
- pyboard.py: make --no-follow use same variable as --follow
- pyboard.py: add --exclusive to match --no-exclusive
- pyboard.py: make --no-soft-reset consistent with other args
- uncrustify: force 1 newline at end of file
- mpremote: bump version to 0.0.6

CI:
- workflows: add workflow to build and test javascript port
- workflows: switch from Coveralls to Codecov
- workflows: switch from lcov to gcov
- workflows: add workflow to build and test unix dev variant

The ports
=========

all ports:
- use common mp_hal_stdout_tx_strn_cooked instead of custom one
- update for move of crypto-algorithms, uzlib to lib
- rename USBD_VID/PID config macros to MICROPY_HW_USB_VID/PID

bare-arm port: no changes specific to this component/port

cc3200 port: no changes specific to this component/port

esp8266 port:
- add __len__ to NeoPixel driver to support iterating
- Makefile: add more libm files to build
- include hidden networks in WLAN.scan results
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream

esp32 port:
- boards/sdkconfig.base: disable MEMPROT_FEATURE to alloc from IRAM
- add __len__ to NeoPixel driver to support iterating
- main: allow MICROPY_DIR to be overridden
- esp32_rmt: fix RMT looping in newer IDF versions
- esp32_rmt: enhance RMT with idle_level and write_pulses modes
- add new machine.I2S class for I2S protocol support
- machine_spi: calculate actual attained baudrate
- machine_hw_spi: use a 2 item SPI queue for long transfers
- machine_dac: add MICROPY_PY_MACHINE_DAC option, enable by default
- machine_i2s: add MICROPY_PY_MACHINE_I2S option, enable by default
- fix use of mp_int_t, size_t and uintptr_t
- add initial support for ESP32C3 SoCs
- boards/GENERIC_C3: add generic C3-based board
- modmachine: release the GIL in machine.idle()
- mphalport: always yield at least once in delay_ms
- machine_uart: add flow kw-arg to enable hardware flow control
- boards: add Silicognition wESP32 board configuration
- mpconfigport.h: enable reverse and inplace special methods
- include hidden networks in WLAN.scan results
- makeimg.py: get bootloader and partition offset from sdkconfig
- enable MICROPY_PY_FSTRINGS by default
- machine_hw_spi: release GIL during transfers
- machine_pin: make check for non-output pins respect chip variant
- replace esp.neopixel with machine.bitstream
- remove dead code for end_ticks in machine_bitstream
- boards: add GENERIC_C3_USB board with USB serial/JTAG support

javascript port:
- rework Makefile and GC so it works with latest Emscripten
- Makefile: suppress compiler errors from array bounds
- Makefile: change variable to EXPORTED_RUNTIME_METHODS

mimxrt port:
- move calc_weekday helper function to timeutils
- machine_spi: add the SPI class to the machine module
- moduos: seed the PRNG on boot using the TRNG
- boards: set vfs partition start to 1 MBbyte
- main: skip running main.py if boot.py failed
- main: extend the information returned by help()
- mimxrt_flash: remove commented-out code
- modmachine: add a few minor methods to the machine module
- machine_led: use mp_raise_msg_varg helper
- machine_i2c: add hardware-based machine.I2C to machine module
- add support for Hyperflash chips
- boards: add support for the MIMXRT1050_EVKB board
- machine_pin: implement ioctl for Pin

minimal port:
- Makefile: add support for building with user C modules

nrf port:
- modules: replace master/slave with controller/peripheral in SPI
- boards/common.ld: calculate unused flash region
- modules/nrf: add new nrf module with flash block device
- drivers: add support for using flash block device with SoftDevice
- mpconfigport.h: expose nrf module when MICROPY_PY_NRF is set
- README: update README.md to reflect internal file systems
- mpconfigport.h: tune FAT FS configuration
- Makefile: add _fs_size linker script override from make
- modules/uos: allow a board to configure MICROPY_VFS_FAT/LFS1/LFS2
- mpconfigport.h: enable MICROPY_PY_IO_FILEIO when an FS is enabled
- qstrdefsport.h: add entries for in-built FS mount points
- main: add auto mount and auto format hook for internal flash FS
- boards: enable needed features for FAT/LFS1/LFS2
- facilitate use of freeze manifest
- boards: set FROZEN_MANIFEST blank when SD present on nrf51 targets
- modules/scripts: add file system formatting script
- Makefile: set default manifest file for all targets
- mphalport: add dummy function for mp_hal_time_ns()
- boards: enable MICROPY_VFS_LFS2 for all target boards
- modules/uos: add ilistdir to uos module
- modules/nrf: add function to enable/disable DCDC
- enable source line on tracebacks
- set .mpy features consistent with documentation and other ports

pic16bit port: no changes specific to this component/port

powerpc port: no changes specific to this component/port

qemu-arm port: no changes specific to this component/port

rp2 port:
- use 0=Monday datetime convention in RTC
- machine_rtc: in RTC.datetime, compute weekday automatically
- CMakeLists.txt: suppress compiler errors for pico-sdk and tinyusb
- tusb_config.h: set CFG_TUD_CDC_EP_BUFSIZE to 256
- machine_uart: add hardware flow control support
- machine_uart: allow overriding default machine UART pins
- machine_i2c: allow boards to configure I2C pins using new macros
- machine_spi: allow boards to configure SPI pins using new macros
- machine_uart: fix poll ioctl to also check hardware FIFO
- machine_uart: fix read when FIFO has chars but ringbuf doesn't
- tusb_port: allow boards to configure USB VID and PID
- boards/ADAFRUIT_FEATHER_RP2040: configure custom VID/PID
- boards/ADAFRUIT_FEATHER_RP2040: configure I2C/SPI default pins
- boards/SPARKFUN_PROMICRO: configure UART/I2C/SPI default pins
- boards/SPARKFUN_THINGPLUS: configure I2C/SPI default pins
- boards: add Adafruit ItsyBitsy RP2040
- boards: add Adafruit QT Py RP2040
- boards: add Pimoroni Pico LiPo 4MB
- boards: add Pimoroni Pico LiPo 16MB
- boards: add Pimoroni Tiny 2040
- CMakeLists.txt: allow a board's cmake to set the manifest path
- enable MICROPY_PY_FSTRINGS by default
- Makefile: add "submodules" target, to match other ports
- rp2_flash: disable IRQs while calling flash_erase/program
- CMakeLists.txt: add option to enable double tap reset to bootrom
- mpconfigport.h: allow boards to add root pointers

samd port:
- add support for building with user C modules

stm32 port:
- softtimer: add soft_timer_reinsert() helper function
- mpbthciport: change from systick to soft-timer for BT scheduling
- provide a custom BTstack runloop that integrates with soft timer
- usb: make irq's default trigger enable all events
- boardctrl: skip running main.py if boot.py had an error
- sdio: fix undefined reference to DMA stream on H7
- dma: add DMAMUX configuration for H7 to fix dma_nohal_init
- main: call mp_deinit() at end of main
- adc: allow using ADC12 and ADC3 for H7
- adc: define the ADC instance used for internal channels
- adc: simplify and generalise how pin_adcX table is defined
- add new machine.I2S class for I2S protocol support
- boards/NUCLEO_F446RE: fix I2C1 pin assignment to match datasheet
- replace master/slave with controller/peripheral in I2C and SPI
- systick: always POLL_HOOK when delaying for milliseconds
- sdram: make SDRAM test cache aware, and optional failure with msg
- boards/NUCLEO_F446RE: enable CAN bus support
- boards: add support for SparkFun STM32 MicroMod Processor board
- uart: fix LPUART1 baudrate set/get
- uart: support low baudrates on LPUART1
- boards/STM32F429DISC: set correct UART2 pins and add UART3/6
- boards/NUCLEO_F439ZI: add board definition for NUCLEO_F439ZI
- boards/LEGO_HUB_NO6: add board definition for LEGO_HUB_NO6
- Makefile: update to only pull in used Bluetooth library
- README.md: update supported MCUs, and submodule and mboot use
- usbd_desc: rename USBD_xxx descriptor opts to MICROPY_HW_USB_xxx
- usbd_cdc_interface: rename USBD_CDC_xx opts to MICROPY_HW_USB_xx
- powerctrl: support changing frequency on WB MCUs
- boards/NUCLEO_H743ZI2: add modified version of NUCLEO_H743ZI
- mbedtls: fix compile warning about uninitialized val
- enable MICROPY_PY_FSTRINGS by default
- add implementation of machine.bitstream
- Makefile: allow GIT_SUBMODULES and LIBS to be extended
- stm32_it: support TIM17 IRQs on WB MCUs
- disable computed goto on constrained boards
- storage: make extended-block-device more configurable
- boards/LEGO_HUB_NO6: change SPI flash storage to use hardware SPI
- boards/LEGO_HUB_NO6: skip first 1MiB of SPI flash for storage
- boards/LEGO_HUB_NO6: add make commands to backup/restore firmware

teensy port: no changes specific to this component/port

unix port:
- modffi: add option to lock GC in callback, and cfun access
- Makefile: add back LIB_SRC_C to list of object files
- variants: enable help and help("modules") on standard and dev
- Makefile: disable error compression on arm-linux-gnueabi-gcc

windows port:
- Makefile: add .exe extension to executables name
- appveyor: update to VS 2017 and use Python 3.8 for build/test

zephyr port:
- machine_spi: add support for hardware SPI
2021-10-15 08:20:54 -05:00
microDev
a279f7b42c
update extra_coverage.py.exp 2021-10-05 23:48:32 +05:30
Jim Mussared
b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
Damien George
426785a19e py/emitnative: Ensure load_subscr does not clobber existing REG_RET.
Fixes issue #7782, and part of issue #6314.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-13 22:30:24 +10:00
Damien George
c0761d28fc tests/perf_bench: Use math.log instead of math.log2.
So MICROPY_PY_MATH_SPECIAL_FUNCTIONS is not needed for these performance
tests.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-13 18:27:39 +10:00
microDev
7dbf750dd0
add test for traceback module 2021-09-03 20:52:30 +05:30
Damien George
08ff71dfcd tests/pybnative: Make while.py test run on boards without pyb.delay.
Signed-off-by: Damien George <damien@micropython.org>
2021-09-01 00:43:41 +10:00
Scott Shawcroft
558f9cdfc1
Fix meminfo test 2021-08-30 14:23:11 -07:00
Jeff Epler
db3945edfe Make %p include the 0x prefix
.. and modify some messages where 0x was specified "manually".

This involves updating some tests to expect the new 0x to appear.
2021-08-29 11:36:29 -05:00
Damien George
8c4ba575fd tests/basics: Split f-string debug printing to separate file with .exp.
This feature {x=} was introduced in Python 3.8 so needs a separate .exp
file to run on earlier Python versions.

See https://bugs.python.org/issue36817

Signed-off-by: Damien George <damien@micropython.org>
2021-08-26 23:56:02 +10:00
Peter Hinch
2296df0a32 extmod/modframebuf: Enable blit between different formats via a palette.
This achieves a substantial performance improvement when rendering glyphs
to color displays, the benefit increasing proportional to the number of
pixels in the glyph.
2021-08-25 15:31:23 +10:00
microDev
57b4d0176f
Merge pull request #5077 from microDev1/atexit
Add atexit module
2021-08-23 19:49:46 +05:30
microDev
25437dcb33
Merge branch 'main' into atexit 2021-08-20 09:45:54 +05:30
Scott Shawcroft
1d7e24484c
Fix unicode test 2021-08-17 18:18:46 -07:00
Scott Shawcroft
11f1c42bb5
Turn on unicode for FATFS
This also tweaks the repr for unicode strings to only escape a few
utf-8 code points. This makes emoji show in os.listdir() for
example.

Also, enable exfat support on full builds.

Fixes #5146
2021-08-17 17:41:59 -07:00
Jim Mussared
91a99fcf0e tests/extmod/vfs_fat_finaliser.py: Ensure alloc at never-used GC blocks.
Prevents the finaliser from being missed if there's a dangling reference
on the stack to one of the blocks for the files (that this test checks
that they get finalised).

See github.com/micropython/micropython/pull/7659#issuecomment-899479793

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-17 11:20:58 +10:00
microDev
1c4a6c3667
atexit module refinements
- add test for atexit module
- add callback to gc collection
- fix callback memory allocation
- execute callback on both code and repl exit
2021-08-16 21:37:32 +05:30
Jim Mussared
c70930fb24 tests/multi_bluetooth/ble_subscribe.py: Add test for subscription.
This tests both sending indications/notifications from a server to
subscribed clients via gatts_write(...,send_update=True) and subscribing
from a client.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 22:44:47 +10:00
Jim Mussared
692d36d779 py: Implement partial PEP-498 (f-string) support.
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 16:58:40 +10:00
Damien George
8fcdb5490c extmod/modlwip: Fix close and clean up of UDP and raw sockets.
The correct callback-deregister functions must be called dependent on the
socket type, otherwise resources may not be freed correctly.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-13 23:46:11 +10:00
Damien George
90d47ee34d tests/run-multitests.py: Add broadcast and wait facility.
Test instances can now use the following methods to synchronise their
execution:

    multitest.broadcast("sync message")
    multitest.wait("sync message")

Signed-off-by: Damien George <damien@micropython.org>
2021-08-13 23:26:34 +10:00
Peter Züger
d290f369d0 tests/extmod/ujson: Add tests for dump/dumps separators argument.
Basically just copied ujson_dump(s).py and added various valid/invalid
separator tests.

Signed-off-by: Peter Züger <zueger.peter@icloud.com>
2021-08-07 13:52:16 +10:00
Jeff Epler
f9393c9e51 enable qrio in unix coverage build, and add a test 2021-08-05 12:24:07 -05:00
microDev
4938851122
remove legacy sys.atexit() implementation 2021-07-30 10:00:00 +05:30
Jim Mussared
4e39ff221a py/runtime: Fix bool unary op for subclasses of native types.
Previously a subclass of a type that didn't implement unary_op, or didn't
handle MP_UNARY_OP_BOOL, would raise TypeError on bool conversion.

Fixes #5677.
2021-07-23 12:40:00 +10:00
Jim Mussared
0e3752e82a py/emitnative: Ensure stack settling is safe mid-branch.
And add a test for the case where REG_RET could be in use.

Fixes #7523.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-07-19 23:18:59 +10:00
David P
f365025c9c stm32: Replace master/slave with controller/peripheral in I2C and SPI.
Replace "master" with "controller" and "slave" with "peripheral" in
comments, errors, and debug messages.

Add CONTROLLER and PERIPHERAL constants to pyb.SPI and pyb.I2C classes;
retain MASTER and SLAVE constants for backward compatiblity.
2021-07-18 11:23:41 +10:00
Damien George
bb00125aaa py: Support single argument to optimised MP_OBJ_STOP_ITERATION.
The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a
StopIteration() exception object, and means that heap memory does not need
to be allocated for the exception (in cases where it can be used).  This
commit allows this optimised object to take an optional argument (before,
it could only have no argument).

The commit also adds some new tests to cover corner cases with
StopIteration and generators that previously did not work.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George
e3825e28e6 py/objexcept: Make mp_obj_exception_get_value support subclassed excs.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
David Lechner
cd506d6220 tests/cpydiff/modules_struct_whitespace_in_format: Run black.
This test snuck through without proper formatting and is causing CI for
other unrelated changes to fail.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-06 18:19:55 -05:00
Tom McDermott
c1f74b3005 docs/library: Warn that ustruct doesn't handle spaces in format strings.
And also add a test to capture the CPython difference.
2021-07-06 14:59:50 +10:00
David Lechner
58e4d72338 py/objexcept: Pretty print OSError also when it has 2 arguments.
This extends pretty-printing of OSError's to handle two arguments when the
exception name is known.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-01 13:23:54 +10:00
Damien George
7ec95c2768 extmod/uasyncio: Get addr and bind server socket before creating task.
Currently when using uasyncio.start_server() the socket configuration is
done inside a uasyncio.create_task() background function.  If the address
and port are already in use however this throws an OSError which cannot be
cleanly caught behind the create_task().

This commit moves the getaddrinfo and socket binding to the start_server()
function, and only creates the task if that succeeds.  This means that any
OSError from the initial socket configuration is propagated directly up the
call stack, compatible with CPython behaviour.

See #7444.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-26 22:30:22 +10:00
Damien George
180c54d6cc tests/extmod: Make uasyncio_heaplock test more deterministic.
This helps the test pass on systems with an inaccurate sleep time.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-25 11:31:00 +10:00
Dan Halbert
8fd02ffcc1
Merge pull request #4904 from hierophect/alarmtests
Add manual tests for Alarm module
2021-06-24 08:17:01 -04:00
Jeff Epler
680ac9388f review suggestion: remove files 2021-06-21 19:51:03 -05:00
Lucian Copeland
a23c659eda Add alarm tests 2021-06-20 18:09:18 -04:00
Jeff Epler
87d3740c64 Merge tag 'v1.16' 2021-06-18 10:54:19 -05:00
Damien George
adf35cbab0 tests/float: Make bytes/bytearray construct tests work with obj repr C.
2.5 can be represented correctly in object representation C, but 2.3 cannot
(it is slightly truncated).

Signed-off-by: Damien George <damien@micropython.org>
2021-06-18 14:16:07 +10:00
Damien George
514bf1a191 extmod/uasyncio: Fix race with cancelled task waiting on finished task.
This commit fixes a problem with a race between cancellation of task A and
completion of task B, when A waits on B.  If task B completes just before
task A is cancelled then the cancellation of A does not work.  Instead,
the CancelledError meant to cancel A gets passed through to B (that's
expected behaviour) but B handles it as a "Task exception wasn't retrieved"
scenario, printing out such a message (this is because finished tasks point
their "coro" attribute to themselves to indicate they are done, and
implement the throw() method, but that method inadvertently catches the
CancelledError).  The correct behaviour is for B to bounce that
CancelledError back out.

This bug is mainly seen when wait_for() is used, and in that context the
symptoms are:
- occurs when using wait_for(T, S), if the task T being waited on finishes
  at exactly the same time as the wait-for timeout S expires
- task T will have run to completion
- the "Task exception wasn't retrieved message" is printed with
  "<class 'CancelledError'>" as the error (ie no traceback)
- the wait_for(T, S) call never returns (it's never put back on the
  uasyncio run queue) and all tasks waiting on this are blocked forever
  from running
- uasyncio otherwise continues to function and other tasks continue to be
  scheduled as normal

The fix here reworks the "waiting" attribute of Task to be called "state"
and uses it to indicate whether a task is: running and not awaited on,
running and awaited on, finished and not awaited on, or finished and
awaited on.  This means the task does not need to point "coro" to itself to
indicate finished, and also allows removal of the throw() method.

A benefit of this is that "Task exception wasn't retrieved" messages can go
back to being able to print the name of the coroutine function.

Fixes issue #7386.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-16 13:02:37 +10:00
Mike Teachman
b0b8ebc4f6 extmod/uasyncio: Add readinto() method to Stream class.
With docs and a multi-test using TCP server/client.

This method is a MicroPython extension, although there is discussion of
adding it to CPython: https://bugs.python.org/issue41305

Signed-off-by: Mike Teachman <mike.teachman@gmail.com>
2021-06-15 13:13:35 +10:00
Zoltán Vörös
c4ed17ff34 tests/cpydiff: Add test for array constructor with overflowing value. 2021-06-13 10:30:14 +10:00
Damien George
20a8f4f7ec tests/unix: Add ffi test for integer types.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-06 22:52:25 +10:00
Damien George
7842085434 tests/multi_bluetooth/ble_gap_advertise.py: Allow to work without set.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-06 21:58:07 +10:00
Damien George
da8e47da21 tests/run-multitests.py: Allow to work without sys.stdout on target.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-06 21:58:07 +10:00
Jeff Epler
e6fe55349d run_tests: Parallelism improvements, partial sync with micropython
Take changes from https://github.com/micropython/micropython/pull/3694
(expected to be merged soon) as well as other accumulated stuff from
upstream that we want.

Leave our desired differences, including:
 * silencing warnings in python3
 * renaming the file descriptors returned by openpty()
 * adding ulab tests
 * Adding "." to the import path for skip_if

This speeds up `make test_full` and should also reduce the time in CI
a little bit.
2021-06-03 16:17:47 -05:00
Damien George
53519e322a py/builtinimport: Change relative import's ValueError to ImportError.
Following CPython change, see https://bugs.python.org/issue37444.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 19:35:03 +10:00
Damien George
c3199f5649 extmod/modurandom: Support an argument of bits=0 to getrandbits.
This was changed in CPython 3.9; see https://bugs.python.org/issue40282.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 17:05:56 +10:00
Macarthur Inbody
34d4dab683 extmod/modurandom: Add error message when getrandbits has bad value.
The random module's getrandbits() method didn't give a proper error message
when calling it with a value that was outside of the range of 1-32, which
can lead to confusion using this function (which under CPython can accept
numbers larger than 32).  Now instead of simply giving a ValueError it
gives an error message that states that the number of bits is constrained.

Also, since the random module's functions getrandbits() and randint()
differ from CPython, tests have been added to describe these differences.
For getrandbits the relevant documentation is shown and added to the docs.
The same is given for randint method so that the information is more easily
found.

Finally, since the int object lacks the bit_length() method there is a test
for that method also to include within the docs, showing the difference to
CPython.
2021-05-30 16:41:30 +10:00
Damien George
025e4b6fbc tests/basics: Split out literal tests that raise SyntaxWarning on CPy.
Fixes issue #7330.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 13:41:37 +10:00
Jeff Epler
486fe71c6e tests/extmod/btree_gc.py: Close the database to avoid a memory leak.
Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-05-30 11:50:51 +10:00
Jeff Epler
f2dbc91022 py/compile: Raise an error on async with/for outside an async function.
A simple reproducer is:

   async for x in (): x

Before this change, it would cause an assertion error in mpy-cross and
micropython-coverage.
2021-05-30 10:38:48 +10:00
Damien George
6a127810c0 extmod/moduhashlib: Put hash obj in final state after digest is called.
If digest is called then the hash object is put in a "final" state and
calling update() or digest() again will raise a ValueError (instead of
silently producing the wrong result).

See issue #4119.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-26 21:44:46 +10:00
Damien George
dc86e04476 tests: Make float and framebuf tests skip or run on big-endian archs.
Signed-off-by: Damien George <damien@micropython.org>
2021-05-26 16:33:18 +10:00
Damien George
5176a2d732 py/emitnative: Fix x86-64 emitter to generate correct 8/16-bit stores.
Fixes issue #6643.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-20 23:43:25 +10:00
Damien George
6d2680fa36 py/objarray: Fix constructing a memoryview from a memoryview.
Fixes issue #7261.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-18 10:18:56 +10:00
Damien George
47e6c52f0c tests/cpydiff: Add test and workaround for function.__module__ attr.
MicroPython does not store any reference from a function object to the
module it was defined in, but there is a way to use function.__globals__ to
indirectly get the module.

See issue #7259.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-16 11:06:46 +10:00
Jeff Epler
94a3f8a4b0 tests/run-tests.py: Parallelize running tests by default.
This significantly reduces the time taken to run the test suite (on the
unix port).  Use `-j1` to disable this feature.

Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-05-14 11:33:31 +10:00
stijn
09be0c083c py/objarray: Implement more/less comparisons for array. 2021-05-13 22:16:14 +10:00
stijn
57365d8557 py/objarray: Prohibit comparison of mismatching types.
Array equality is defined as each element being equal but to keep
code size down MicroPython implements a binary comparison.  This
can only be used correctly for elements with the same binary layout
though so turn it into an NotImplementedError when comparing types
for which the binary comparison yielded incorrect results: types
with different sizes, and floating point numbers because nan != nan.
2021-05-13 22:16:14 +10:00
Damien George
6affcb0104 tests/run-multitests.py: Flush stdout for each line of trace output.
Signed-off-by: Damien George <damien@micropython.org>
2021-05-13 16:26:07 +10:00
Scott Shawcroft
966d25c6a5
Merge MicroPython v1.15 into CircuitPython 2021-05-12 17:51:42 -07:00
Scott Shawcroft
e02a26453c
Merge MicroPython 1.14 into CircuitPython 2021-05-11 15:07:40 -07:00
Damien George
18d984c8b2 tests/run-perfbench.py: Fix native feature check.
This was broken by 8459f538eb

Signed-off-by: Damien George <damien@micropython.org>
2021-05-11 23:45:36 +10:00
Damien George
4cdcbdb753 tests/thread: Make exc1,exit1,exit2,stacksize1,start1 tests run on rp2.
The RP2040 has 2 cores and supports running at most 2 Python threads (the
main one plus another), and will raise OSError if a thread cannot be
created because core1 is already in use.  This commit adjusts some thread
tests to be robust against such OSError's.  These tests now pass on rp2
boards.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-10 13:07:16 +10:00
Damien George
b6b39bff47 py/gc: Make gc_lock_depth have a count per thread.
This commit makes gc_lock_depth have one counter per thread, instead of one
global counter.  This makes threads properly independent with respect to
the GC, in particular threads can now independently lock the GC for
themselves without locking it for other threads.  It also means a given
thread can run a hard IRQ without temporarily locking the GC for all other
threads and potentially making them have MemoryError exceptions at random
locations (this really only occurs on MCUs with multiple cores and no GIL,
eg on the rp2 port).

The commit also removes protection of the GC lock/unlock functions, which
is no longer needed when the counter is per thread (and this also fixes the
cas where a hard IRQ calling gc_lock() may stall waiting for the mutex).

It also puts the check for `gc_lock_depth > 0` outside the GC mutex in
gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from
waiting on a mutex if it does attempt to allocate heap memory (and putting
the check outside the GC mutex is now safe now that there is a
gc_lock_depth per thread).

Signed-off-by: Damien George <damien@micropython.org>
2021-05-10 13:07:16 +10:00
Jeff Epler
aca9d5bc40 Fix up all natmod examples
* modframebuf: _mp_framebuf_p_t is not "really" a protocol, but the
   QSTR assignment caused problems when building as a dynamic module
 * modure: str_index_to_ptr is not in the natmod API, disable URE match
   spans when dynamic.  mp_obj_len() is a bugfix, we should throw here
   if the object is not string-like
 * moduzlib: Correct paths to uzlib headers & sources.  this relative
   path (from moduzlib.c to the referenced file) works in all cases,
   the other only worked from ports/PORTNAME.
 * dynruntime: Handle 2-arg m_malloc, assert_native_inited, add a
   micropythonish mp_arg_check_num_mp, fix mp_raise_msg to use dumb
   strings, add mp_raise_arg1
 * nativeglue: ad assert_native_inited
 * translate: MP_ERROR_TEXT evaluates to its argument for DYNRUNTIME
 * mpy-tool: A straggling magic number change
 * mpy_ld: Have to renumber manually after dynruntime change
 * import_mpy_native_gc.py: Update copy of features0 baked into this test
2021-05-08 20:17:16 -05:00
Damien George
7b923d6c72 tests/thread: Make stress_aes.py test run on bare-metal ports.
This is a long-running test, so make it run in reasonable time on slower,
bare-metal ports.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-08 22:47:03 +10:00
Damien George
9340cfe774 tests/thread: Make stress_create.py test run on esp32.
The esp32 port needs to be idle for finished threads and their resources to
be freed up.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-08 22:47:03 +10:00
Damien George
864e4ecc47 esp32/mpthreadport: Use binary semaphore instead of mutex.
So a lock can be acquired on one Python thread and then released on
another.  A test for this is added.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-08 22:47:03 +10:00
Scott Shawcroft
cebb10ddae
Fixes from review 2021-05-07 09:31:59 -07:00
Scott Shawcroft
b19c1a310e
mpy M -> C 2021-05-06 13:31:20 -07:00
Scott Shawcroft
42f4065c8a
Merge remote-tracking branch 'adafruit/main' into merge_1.13 2021-05-06 11:17:53 -07:00
Damien George
47583d8cbd extmod/moductypes: Fix size and offset calculation for ARRAY of FLOAT32.
uctypes.FLOAT32 has a special value representation and
uctypes_struct_scalar_size() should be used instead of GET_SCALAR_SIZE().

Signed-off-by: Damien George <damien@micropython.org>
2021-05-06 13:11:33 +10:00
Damien George
9e29217c73 unix/modffi: Use a union for passing/returning FFI values.
This fixes a bug where double arguments on a 32-bit architecture would not
be passed correctly because they only had 4 bytes of storage (not 8).  It
also fixes a compiler warning/error in return_ffi_value on certian
architectures: array subscript 'double[0]' is partly outside array bounds
of 'ffi_arg[1]' {aka 'long unsigned int[1]'}.

Fixes issue #7064.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-06 12:17:10 +10:00
Scott Shawcroft
7f2f8e2d58
Turn off CPython warnings in tests 2021-05-05 17:26:39 -07:00
Jeff Epler
ef3ec93c8b Change the first byte of CircuitPython 'mpy' files to "C"
.. and also distinguish CircuitPython better in `mpy-cross --version`
2021-05-05 18:06:06 -05:00
Jeff Epler
f21c2bd25a Update native skips to match micropython 2021-05-05 11:00:39 -05:00
Jeff Epler
c174bc42fa Update to match our bytecode style 2021-05-05 09:21:42 -05:00
Jeff Epler
d0f6b323bb Update bytecode 2021-05-05 09:05:44 -05:00
Jeff Epler
92d63b40cc tests: Restore memoryview1 test to micropython version
we changed this way back at fab634e3ee for Rosie Pi and the merge
mangled it.  I think at this point it makes sense just to take upstream's
version again.
2021-05-05 08:46:37 -05:00
Scott Shawcroft
f0bb26d70f
Merge MicroPython 1.13 into CircuitPython 2021-05-04 18:06:33 -07:00
Scott Shawcroft
b35fa44c8a
Merge MicroPython 1.12 into CircuitPython 2021-05-03 14:01:18 -07:00
Artyom Skrobov
ca35c0059c py/repl: Autocomplete builtin modules.
Doing "import <tab>" will now complete/list built-in modules.

Originally at adafruit#4548 and adafruit#4608

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2021-05-02 23:11:14 +10:00
Kathryn Lingel
1f1a54d0b1 py/repl: Filter private methods from tab completion.
Anything beginning with "_" will now only be tab-completed if there is
already a partial match for such an entry.  In other words, entering
foo.<tab> will no longer complete/list anything beginning with "_".

Originally at adafruit#1850

Signed-off-by: Kathryn Lingel <kathryn@lingel.net>
2021-05-02 23:11:03 +10:00
Damien George
1d9528210b tests/multi_bluetooth: Add performance test for gatt char writes.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 16:14:48 +10:00
Damien George
76dab3bf31 tests/run-multitests.py: Provide some convenient serial device shorcuts.
It's now possible to specify a device serial port using shorcuts like:

    $ ./run-multitests.py -i pyb:a0 -i pyb:u1 multi_bluetooth/*.py

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:47:22 +10:00
Scott Shawcroft
76033d5115
Merge MicroPython v1.11 into CircuitPython 2021-04-26 15:47:41 -07:00
Damien George
3123f6918b tests: Use .errno instead of .args[0] for OSError exceptions.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-23 22:03:46 +10:00
Damien George
3c4bfd1dec py/objexcept: Support errno attribute on OSError exceptions.
This commit adds the errno attribute to exceptions, so code can retrieve
errno codes from an OSError using exc.errno.

The implementation here simply lets `errno` (and the existing `value`)
attributes work on any exception instance (they both alias args[0]).  This
is for efficiency and to keep code size down.  The pros and cons of this
are:

Pros:
- more compatible with CPython, less difference to document and learn
- OSError().errno will correctly return None, whereas the current way of
  doing it via OSError().args[0] will raise an IndexError
- it reduces code size on most bare-metal ports (because they already have
  the errno qstr)
- for Python code that uses exc.errno the generated bytecode is 2 bytes
  smaller and more efficient to execute (compared with exc.args[0]); so
  bytecode loaded to RAM saves 2 bytes RAM for each use of this attribute,
  and bytecode that is frozen saves 2 bytes flash/ROM for each use
- it's easier/shorter to type, and saves 2 bytes of space in .py files that
  use it (for each use)

Cons:
- increases code size by 4-8 bytes on minimal ports that don't already have
  the `errno` qstr
- all exceptions now have .errno and .value attributes (a cpydiff test is
  added to address this)

See also #2407.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-23 22:03:46 +10:00
Scott Shawcroft
09f7b43c64
Merge MicroPython 1.10 into CircuitPython 2021-04-21 15:59:17 -07:00
Scott Shawcroft
b057fb8a4b
codeformat 2021-04-19 22:22:44 -07:00
Damien George
7d911d2069 tests/net_inet: Add 'Strict-Transport-Security' to exp file.
Because micropython.org now adds this to the headers.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-18 23:20:26 +10:00
Scott Shawcroft
5ccd5dc08b
Fix struct.pack with padding bytes
It used to validate the following arg could fit in a single byte.
Now, it always uses zero to pad.
2021-04-16 12:39:23 -07:00
Damien George
8459f538eb tests/feature_check: Check for lack of pass result rather than failure.
Commit cb68a5741a broke automatic Python
feature detection when running tests, because some detection relied on a
crash of a feature script returning exactly b"CRASH".

This commit fixes this and improves the situation by testing for the lack
of a known pass result, rather than an exact failure result.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-15 00:52:56 +10:00
Artyom Skrobov
c6ee471b52 Merge branch 'main' of https://github.com/adafruit/circuitpython into patch-3 2021-04-13 00:32:19 -04:00
stijn
a66286f3a0 unix: Improve command line argument processing.
Per CPython everything which comes after the command, module or file
argument is not an option for the interpreter itself.  Hence the processing
of options should stop when encountering those, and the remainder be passed
as sys.argv.  Note the latter was already the case for a module or file but
not for a command.

This fixes issues like 'micropython myfile.py -h' showing the help and
exiting instead of passing '-h' as sys.argv[1], likewise for
'-X <something>' being treated as a special option no matter where it
occurs on the command line.
2021-04-07 12:41:25 +10:00
Jeff Epler
2fb5eb3b11 extmod/re1.5: Check and report byte overflow errors in _compilecode.
The generated regex code is limited in the range of jumps and counts, and
this commit checks all cases which can overflow given the right kind of
input regex, and returns an error in such a case.

This change assumes that the results that overflow an int8_t do not
overflow a platform int.

Closes: #7078

Signed-off-by: Jeff Epler <jepler@gmail.com>

# Conflicts:
#	extmod/re1.5/compilecode.c
2021-04-06 19:11:40 -05:00
Jeff Epler
172fb5230a extmod/re1.5: Check and report byte overflow errors in _compilecode.
The generated regex code is limited in the range of jumps and counts, and
this commit checks all cases which can overflow given the right kind of
input regex, and returns an error in such a case.

This change assumes that the results that overflow an int8_t do not
overflow a platform int.

Closes: #7078

Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-04-06 13:36:42 +10:00
Artyom Skrobov
8056af8648 [synthio] add a simple MidiTrack implementation 2021-03-26 16:38:10 -04:00
Damien George
cb68a5741a tests/run-tests.py: Provide more info if script run via pyboard crashes.
Signed-off-by: Damien George <damien@micropython.org>
2021-03-16 14:49:57 +11:00
Damien George
a79d97cb76 tests/extmod/vfs_fat_fileio2.py: Close test file at end of test.
Otherwise it can lead to inconsistent results running subsequent tests.

Signed-off-by: Damien George <damien@micropython.org>
2021-03-16 14:49:57 +11:00