circuitpython/tests
Damien George 7c1584aef1 py/compile: Fix scope of assignment expression target in comprehensions.
When := is used in a comprehension the target variable is bound to the
parent scope, so it's either a global or a nonlocal.  Prior to this commit
that was handled by simply using the parent scope's id_info for the
target variable.  That's completely wrong because it uses the slot number
for the parent's Python stack to store the variable, rather than the slot
number for the comprehension.  This will in most cases lead to incorrect
behaviour or memory faults.

This commit fixes the scoping of the target variable by explicitly
declaring it a global or nonlocal, depending on whether the parent is the
global scope or not.  Then the id_info of the comprehension can be used to
access the target variable.  This fixes a lot of cases of using := in a
comprehension.

Code size change for this commit:

       bare-arm:    +0 +0.000%
    minimal x86:    +0 +0.000%
       unix x64:  +152 +0.019% standard
          stm32:   +96 +0.024% PYBV10
         cc3200:   +96 +0.052%
        esp8266:  +196 +0.028% GENERIC
          esp32:  +156 +0.010% GENERIC[incl +8(data)]
         mimxrt:   +96 +0.027% TEENSY40
     renesas-ra:   +88 +0.014% RA6M2_EK
            nrf:   +88 +0.048% pca10040
            rp2:  +104 +0.020% PICO
           samd:   +88 +0.033% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Fixes issue #10895.

Signed-off-by: Damien George <damien@micropython.org>
2023-03-09 12:13:12 +11:00
..
basics py/compile: Fix scope of assignment expression target in comprehensions. 2023-03-09 12:13:12 +11:00
cmdline py/parse: Allow const types other than int to optimise as true/false. 2022-09-23 16:04:13 +10:00
cpydiff unix/moduos: Implement 2-arg version of os.getenv(). 2022-12-14 13:38:39 +11:00
esp32 top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
extmod tests/extmod: Skip vfs tests if target doesn't have enough memory. 2023-02-09 15:32:06 +11:00
feature_check py: Implement partial PEP-498 (f-string) support. 2021-08-14 16:58:40 +10:00
float py/modmath: Fix two-argument math function domain check. 2023-02-24 15:55:12 +11:00
frozen tests/frozen: Move frozentest.mpy from ports/ to tests/. 2022-09-19 23:51:10 +10:00
import top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
inlineasm top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
internal_bench tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
io py/modio: Remove io.resource_stream function. 2021-12-17 23:53:44 +11:00
jni tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
micropython tests/micropython: Split viper_misc test into two files. 2023-02-09 16:12:25 +11:00
misc top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
multi_bluetooth tests/multi_bluetooth: Add bluetooth multi-test for deepsleep. 2023-03-08 14:04:14 +11:00
multi_net top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
net_hosted extmod/uasyncio: Get addr and bind server socket before creating task. 2021-06-26 22:30:22 +10:00
net_inet extmod/modussl_mbedtls: Implement cert_reqs and cadata arguments. 2022-07-20 16:46:04 +10:00
perf_bench top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
pyb top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
qemu-arm qemu-arm: Add tests for freezing viper and asm_thumb code. 2022-02-24 18:29:02 +11:00
renesas-ra tests/renesas-ra: Update pin test to support all boards. 2022-08-31 12:00:41 +10:00
stress top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
thread top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
unicode top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
unix tests/unix/mod_os: Add test for os module. 2022-12-14 13:38:51 +11:00
wipy tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
README.md tests: Add an explanation of run-perfbench.py. 2022-06-28 14:21:41 +10:00
run-internalbench.py top: Update Python formatting to black "2023 stable style". 2023-02-02 12:51:03 +11:00
run-multitests.py tests/run-multitests.py: Add ability to test instance over reboot. 2023-03-08 14:04:14 +11:00
run-natmodtests.py unix/variants: Remove variant suffix from executable filename. 2022-08-11 13:34:34 +10:00
run-perfbench.py all: Keep msvc build output in build/ directories. 2022-12-13 17:18:53 +11:00
run-tests-exp.py tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
run-tests-exp.sh tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
run-tests.py tests/float: Skip new complex tests if complex unavailable. 2023-02-09 15:20:00 +11:00

README.md

MicroPython Test Suite

This directory contains tests for various functionality areas of MicroPython. To run all stable tests, run "run-tests.py" script in this directory.

Tests of capabilities not supported on all platforms should be written to check for the capability being present. If it is not, the test should merely output 'SKIP' followed by the line terminator, and call sys.exit() to raise SystemExit, instead of attempting to test the missing capability. The testing framework (run-tests.py in this directory, test_main.c in qemu_arm) recognizes this as a skipped test.

There are a few features for which this mechanism cannot be used to condition a test. The run-tests.py script uses small scripts in the feature_check directory to check whether each such feature is present, and skips the relevant tests if not.

Tests are generally verified by running the test both in MicroPython and in CPython and comparing the outputs. If the output differs the test fails and the outputs are saved in a .out and a .exp file respectively. For tests that cannot be run in CPython, for example because they use the machine module, a .exp file can be provided next to the test's .py file. A convenient way to generate that is to run the test, let it fail (because CPython cannot run it) and then copy the .out file (but not before checking it manually!)

When creating new tests, anything that relies on float support should go in the float/ subdirectory. Anything that relies on import x, where x is not a built-in module, should go in the import/ subdirectory.

perf_bench

The perf_bench directory contains some performance benchmarks that can be used to benchmark different MicroPython firmwares or host ports.

The runner utility is run-perfbench,py. Execute ./run-perfbench.py --help for a full list of command line options.

Benchmarking a target

To run tests on a firmware target using pyboard.py, run the command line like this:

./run-perfbench.py -p -d /dev/ttyACM0 168 100
  • -p indicates running on a remote target via pyboard.py, not the host.
  • -d PORTNAME is the serial port, /dev/ttyACM0 is the default if not provided.
  • 168 is value N, the approximate CPU frequency in MHz (in this case Pyboard V1.1 is 168MHz). It's possible to choose other values as well: lower values like 10 will run much the tests much quicker, higher values like 1000 will run much longer.
  • 100 is value M, the approximate heap size in kilobytes (can get this from import micropython; micropython.mem_info() or estimate it). It's possible to choose other values here too: lower values like 10 will run shorter/smaller tests, and higher values will run bigger tests. The maximum value of M is limited by available heap, and the tests are written so the "recommended" value is approximately the upper limit.

Benchmarking the host

To benchmark the host build (unix/Windows), run like this:

./run-perfbench.py 2000 10000

The output of perfbench is a list of tests and times/scores, like this:

N=2000 M=10000 n_average=8
perf_bench/bm_chaos.py: SKIP
perf_bench/bm_fannkuch.py: 94550.38 2.9145 84.68 2.8499
perf_bench/bm_fft.py: 79920.38 10.0771 129269.74 8.8205
perf_bench/bm_float.py: 43844.62 17.8229 353219.64 17.7693
perf_bench/bm_hexiom.py: 32959.12 15.0243 775.77 14.8893
perf_bench/bm_nqueens.py: 40855.00 10.7297 247776.15 11.3647
perf_bench/bm_pidigits.py: 64547.75 2.5609 7751.36 2.5996
perf_bench/core_import_mpy_multi.py: 15433.38 14.2733 33065.45 14.2368
perf_bench/core_import_mpy_single.py: 263.00 11.3910 3858.35 12.9021
perf_bench/core_qstr.py: 4929.12 1.8434 8117.71 1.7921
perf_bench/core_yield_from.py: 16274.25 6.2584 12334.13 5.8125
perf_bench/misc_aes.py: 57425.25 5.5226 17888.60 5.7482
perf_bench/misc_mandel.py: 40809.25 8.2007 158107.00 9.8864
perf_bench/misc_pystone.py: 39821.75 6.4145 100867.62 6.5043
perf_bench/misc_raytrace.py: 36293.75 6.8501 26906.93 6.8402
perf_bench/viper_call0.py: 15573.00 14.9931 19644.99 13.1550
perf_bench/viper_call1a.py: 16725.75 9.8205 18099.96 9.2752
perf_bench/viper_call1b.py: 20752.62 8.3372 14565.60 9.0663
perf_bench/viper_call1c.py: 20849.88 5.8783 14444.80 6.6295
perf_bench/viper_call2a.py: 16156.25 11.2956 18818.59 11.7959
perf_bench/viper_call2b.py: 22047.38 8.9484 13725.73 9.6800

The numbers across each line are times and scores for the test:

  • Runtime average (microseconds, lower is better)
  • Runtime standard deviation as a percentage
  • Score average (units depend on the benchmark, higher is better)
  • Score standard deviation as a percentage

Comparing performance

Usually you want to know if something is faster or slower than a reference. To do this, copy the output of each run-perfbench.py run to a text file.

This can be done multiple ways, but one way on Linux/macOS is with the tee utility: ./run-perfbench.py -p 168 100 | tee pyb-run1.txt

Once you have two files with output from two different runs (maybe with different code or configuration), compare the runtimes with ./run-perfbench.py -t pybv-run1.txt pybv-run2.txt or compare scores with ./run-perfbench.py -s pybv-run1.txt pybv-run2.txt:

> ./run-perfbench.py -s pyb-run1.txt pyb-run2.txt
diff of scores (higher is better)
N=168 M=100                pyb-run1.txt -> pyb-run2.txt         diff      diff% (error%)
bm_chaos.py                    352.90 ->     352.63 :      -0.27 =  -0.077% (+/-0.00%)
bm_fannkuch.py                  77.52 ->      77.45 :      -0.07 =  -0.090% (+/-0.01%)
bm_fft.py                     2516.80 ->    2519.74 :      +2.94 =  +0.117% (+/-0.00%)
bm_float.py                   5749.27 ->    5749.65 :      +0.38 =  +0.007% (+/-0.00%)
bm_hexiom.py                    42.22 ->      42.30 :      +0.08 =  +0.189% (+/-0.00%)
bm_nqueens.py                 4407.55 ->    4414.44 :      +6.89 =  +0.156% (+/-0.00%)
bm_pidigits.py                 638.09 ->     632.14 :      -5.95 =  -0.932% (+/-0.25%)
core_import_mpy_multi.py       477.74 ->     477.57 :      -0.17 =  -0.036% (+/-0.00%)
core_import_mpy_single.py       58.74 ->      58.72 :      -0.02 =  -0.034% (+/-0.00%)
core_qstr.py                    63.11 ->      63.11 :      +0.00 =  +0.000% (+/-0.01%)
core_yield_from.py             357.57 ->     357.57 :      +0.00 =  +0.000% (+/-0.00%)
misc_aes.py                    397.27 ->     396.47 :      -0.80 =  -0.201% (+/-0.00%)
misc_mandel.py                3375.70 ->    3375.84 :      +0.14 =  +0.004% (+/-0.00%)
misc_pystone.py               2265.36 ->    2265.97 :      +0.61 =  +0.027% (+/-0.01%)
misc_raytrace.py               367.61 ->     368.15 :      +0.54 =  +0.147% (+/-0.01%)
viper_call0.py                 605.92 ->     605.92 :      +0.00 =  +0.000% (+/-0.00%)
viper_call1a.py                576.78 ->     576.78 :      +0.00 =  +0.000% (+/-0.00%)
viper_call1b.py                452.45 ->     452.46 :      +0.01 =  +0.002% (+/-0.01%)
viper_call1c.py                457.39 ->     457.39 :      +0.00 =  +0.000% (+/-0.00%)
viper_call2a.py                561.37 ->     561.37 :      +0.00 =  +0.000% (+/-0.00%)
viper_call2b.py                389.49 ->     389.50 :      +0.01 =  +0.003% (+/-0.01%)

Note in particular the error percentages at the end of each line. If these are high relative to the percentage difference then it indicates high variability in the test runs, and the absolute difference value is unreliable. High error percentages are particularly common on PC builds, where the host OS may influence test run times. Increasing the N value may help average this out by running each test longer.