Make tests run in an isolated environment (i.e. `import io` would
otherwise get the `tests/io` directory).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
If a multitest calls `multitest.output_metric(...)` then that output will
be collected separately, not considered as part of the test verification
output, and instead be printed at the end. This is useful for tests that
want to output performance/timing metrics that may change from one run to
the next.
Signed-off-by: Damien George <damien@micropython.org>
The device-under-test should use `multitest.expect_reboot()` to indicate
that it will reboot.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
This works if your network is pre-configured in boot.py as an object called
"nic". Without this, multitests expects to access the WLAN/LAN class which
isn't always correct.
Signed-off-by: Andrew Leech <andrew@alelec.net>
BLE still functions correctly even though these messages are sometimes
printed by the IDF. Ignoring them allows the multi_bluetooth tests to pass
on an esp32 board.
Signed-off-by: Damien George <damien@micropython.org>
Test instances can now use the following methods to synchronise their
execution:
multitest.broadcast("sync message")
multitest.wait("sync message")
Signed-off-by: Damien George <damien@micropython.org>
It's now possible to specify a device serial port using shorcuts like:
$ ./run-multitests.py -i pyb:a0 -i pyb:u1 multi_bluetooth/*.py
Signed-off-by: Damien George <damien@micropython.org>
For example, to run the BLE multitests entirely with the unix port:
env MICROPY_MICROPYTHON=../ports/unix/micropython-dev ./run-multitests.py \
-i micropython,MICROPYBTUSB=01 \
-i micropython,MICROPYBTUSB=02:02 \
multi_bluetooth/ble_*.py
One can now use `-i micropython` and `-i cpython` to add instances using
the `MICROPYTHON` and `CPYTHON3` variables (which can be overridden by env
vars).
This adds the Python files in the tests/ directory to be formatted with
./tools/codeformat.py. The basics/ subdirectory is excluded for now so we
aren't changing too much at once.
In a few places `# fmt: off`/`# fmt: on` was used where the code had
special formatting for readability or where the test was actually testing
the specific formatting.
This commit adds a test runner and initial test scripts which run multiple
Python/MicroPython instances (eg executables, target boards) in parallel.
This is useful for testing, eg, network and Bluetooth functionality.
Each test file has a set of functions called instanceX(), where X ranges
from 0 up to the maximum number of instances that are needed, N-1. Then
run-multitests.py will execute this script on N separate instances (eg
micropython executables, or attached boards via pyboard.py) at the same
time, synchronising their start in the right order, possibly passing IP
address (or other address like bluetooth MAC) from the "server" instance to
the "client" instances so they can connect to each other. It then runs
them to completion, collects the output, and then tests against what
CPython gives (or what's in a provided .py.exp file).
The tests will be run using the standard unix executable for all instances
by default, eg:
$ ./run-multitests.py multi_net/*.py
Or they can be run with a board and unix executable via:
$ ./run-multitests.py --instance pyb:/dev/ttyACM0 --instance exec:micropython multi_net/*.py
Jim Mussared
Damien George
Andrew Leech
stijn
Andrew Leech
robert-hh
Damien George
David Lechner