circuitpython/tests
Damien George df9a949891 tests/run-multitests.py: Add new test runner for multiple Py instances.
This commit adds a test runner and initial test scripts which run multiple
Python/MicroPython instances (eg executables, target boards) in parallel.
This is useful for testing, eg, network and Bluetooth functionality.

Each test file has a set of functions called instanceX(), where X ranges
from 0 up to the maximum number of instances that are needed, N-1.  Then
run-multitests.py will execute this script on N separate instances (eg
micropython executables, or attached boards via pyboard.py) at the same
time, synchronising their start in the right order, possibly passing IP
address (or other address like bluetooth MAC) from the "server" instance to
the "client" instances so they can connect to each other.  It then runs
them to completion, collects the output, and then tests against what
CPython gives (or what's in a provided .py.exp file).

The tests will be run using the standard unix executable for all instances
by default, eg:

    $ ./run-multitests.py multi_net/*.py

Or they can be run with a board and unix executable via:

    $ ./run-multitests.py --instance pyb:/dev/ttyACM0 --instance exec:micropython multi_net/*.py
2020-03-10 02:22:34 +11:00
..
basics py/builtinevex: Support passing in a bytearray/buffer to eval/exec. 2020-02-28 12:45:36 +11:00
cmdline tests/cmdline: Add test for MICROPYINSPECT environment variable. 2020-02-04 17:54:31 +11:00
cpydiff tests: Move CPy diff test to real test now that subclass equality works. 2020-02-04 18:25:34 +11:00
extmod extmod/modframebuf: Allow blit source to be a subclass of FrameBuffer. 2020-02-21 13:32:48 +11:00
feature_check lib/mp-readline: Add word-based move/delete EMACS key sequences. 2020-01-12 13:09:27 +11:00
float tests/basics: Add tests for equality between bool and int/float/complex. 2020-02-11 11:06:17 +11:00
import tests: Add .exp files for basics/parser and import/import_override. 2019-12-13 14:20:47 +11:00
inlineasm tests: Rename "array" module to "uarray". 2019-10-22 19:16:54 +11:00
internal_bench tests: Rename "bench" tests to "internal_bench" and run-internalbench.py 2019-06-28 16:28:59 +10:00
io tests/io: Update tests to use uos.remove() instead of uos.unlink(). 2018-11-26 23:27:28 +11:00
jni tests: Convert remaining "sys.exit()" to "raise SystemExit". 2017-06-10 20:34:38 +03:00
micropython tests/micropython: Add test for yield-from while heap is locked. 2019-12-27 12:34:22 +11:00
misc tests: Rename "array" module to "uarray". 2019-10-22 19:16:54 +11:00
net_hosted extmod/modlwip: Unconditionally return POLLHUP when polling new socket. 2019-10-31 12:54:37 +11:00
net_inet tests/net_inet: Update tls test to work with CPython and incl new site. 2017-10-26 12:29:24 +11:00
perf_bench tests/perf_bench: Add bm_fft test. 2019-10-15 16:38:11 +11:00
pyb tests/pyb: Adjust UART and Timer tests to work on PYBD_SF6. 2019-12-20 16:42:38 +11:00
pybnative py: Fix stack access in thumb native emitter. 2014-05-07 23:27:45 +01:00
qemu-arm qemu-arm: Add testing of frozen native modules. 2019-08-20 15:14:08 +10:00
stress tests/stress/qstr_limit: Tune params to run with stm32 port. 2019-12-13 15:58:28 +11:00
thread py/modthread: Raise RuntimeError in release() if lock is not acquired. 2017-06-14 14:43:50 +10:00
unicode py/objslice: Add support for indices() method on slice objects. 2019-12-28 23:55:15 +11:00
unix tests/unix: Add coverage tests for kbd-intr and scheduler. 2020-02-07 16:08:29 +11:00
wipy various: Spelling fixes 2017-05-29 11:36:05 +03:00
README tests: Make run-tests help and README be more descriptive of behaviour. 2020-01-22 16:35:24 +11:00
run-internalbench.py tests: Rename "bench" tests to "internal_bench" and run-internalbench.py 2019-06-28 16:28:59 +10:00
run-multitests.py tests/run-multitests.py: Add new test runner for multiple Py instances. 2020-03-10 02:22:34 +11:00
run-natmodtests.py unix: Rename unix binaries to micropython-variant (not _variant). 2020-01-12 10:37:40 +11:00
run-perfbench.py tests/run-perfbench.py: Skip complex tests if target doesn't enable it. 2019-10-15 16:46:06 +11:00
run-tests tests/run-tests: Auto-skip extmod/ticks_diff, extmod/time_ms_us tests. 2020-02-11 10:56:49 +11:00
run-tests-exp.py unix: Rename "_os" module to "uos" for consistency with baremetal ports. 2015-12-12 00:04:35 +02:00
run-tests-exp.sh run-tests-exp.sh: Typo fix in comment. 2015-02-21 03:22:33 +02:00

This directory contains tests for various functionality areas of MicroPython.
To run all stable tests, run "run-tests" script in this directory.

Tests of capabilities not supported on all platforms should be written
to check for the capability being present. If it is not, the test
should merely output 'SKIP' followed by the line terminator, and call
sys.exit() to raise SystemExit, instead of attempting to test the
missing capability. The testing framework (run-tests in this
directory, test_main.c in qemu_arm) recognizes this as a skipped test.

There are a few features for which this mechanism cannot be used to
condition a test. The run-tests script uses small scripts in the
feature_check directory to check whether each such feature is present,
and skips the relevant tests if not.

Tests are generally verified by running the test both in MicroPython and
in CPython and comparing the outputs. If the output differs the test fails
and the outputs are saved in a .out and a .exp file respectively.
For tests that cannot be run in CPython, for example because they use
the machine module, a .exp file can be provided next to the test's .py
file. A convenient way to generate that is to run the test, let it fail
(because CPython cannot run it) and then copy the .out file (but not
before checking it manually!)

When creating new tests, anything that relies on float support should go in the
float/ subdirectory.  Anything that relies on import x, where x is not a built-in
module, should go in the import/ subdirectory.