tests: Add an explanation of run-perfbench.py.
Also changes this file to a Markdown file. Signed-off-by: Angus Gratton <gus@projectgus.com>
This commit is contained in:
Angus Gratton
parent
ccaf197807
commit
ad308bc322
27
tests/README
27
tests/README
|
|
@ -1,27 +0,0 @@
|
|||
This directory contains tests for various functionality areas of MicroPython.
|
||||
To run all stable tests, run "run-tests.py" script in this directory.
|
||||
|
||||
Tests of capabilities not supported on all platforms should be written
|
||||
to check for the capability being present. If it is not, the test
|
||||
should merely output 'SKIP' followed by the line terminator, and call
|
||||
sys.exit() to raise SystemExit, instead of attempting to test the
|
||||
missing capability. The testing framework (run-tests.py in this
|
||||
directory, test_main.c in qemu_arm) recognizes this as a skipped test.
|
||||
|
||||
There are a few features for which this mechanism cannot be used to
|
||||
condition a test. The run-tests.py script uses small scripts in the
|
||||
feature_check directory to check whether each such feature is present,
|
||||
and skips the relevant tests if not.
|
||||
|
||||
Tests are generally verified by running the test both in MicroPython and
|
||||
in CPython and comparing the outputs. If the output differs the test fails
|
||||
and the outputs are saved in a .out and a .exp file respectively.
|
||||
For tests that cannot be run in CPython, for example because they use
|
||||
the machine module, a .exp file can be provided next to the test's .py
|
||||
file. A convenient way to generate that is to run the test, let it fail
|
||||
(because CPython cannot run it) and then copy the .out file (but not
|
||||
before checking it manually!)
|
||||
|
||||
When creating new tests, anything that relies on float support should go in the
|
||||
float/ subdirectory. Anything that relies on import x, where x is not a built-in
|
||||
module, should go in the import/ subdirectory.
|
||||
|
|
@ -0,0 +1,149 @@
|
|||
# MicroPython Test Suite
|
||||
|
||||
This directory contains tests for various functionality areas of MicroPython.
|
||||
To run all stable tests, run "run-tests.py" script in this directory.
|
||||
|
||||
Tests of capabilities not supported on all platforms should be written
|
||||
to check for the capability being present. If it is not, the test
|
||||
should merely output 'SKIP' followed by the line terminator, and call
|
||||
sys.exit() to raise SystemExit, instead of attempting to test the
|
||||
missing capability. The testing framework (run-tests.py in this
|
||||
directory, test_main.c in qemu_arm) recognizes this as a skipped test.
|
||||
|
||||
There are a few features for which this mechanism cannot be used to
|
||||
condition a test. The run-tests.py script uses small scripts in the
|
||||
feature_check directory to check whether each such feature is present,
|
||||
and skips the relevant tests if not.
|
||||
|
||||
Tests are generally verified by running the test both in MicroPython and
|
||||
in CPython and comparing the outputs. If the output differs the test fails
|
||||
and the outputs are saved in a .out and a .exp file respectively.
|
||||
For tests that cannot be run in CPython, for example because they use
|
||||
the machine module, a .exp file can be provided next to the test's .py
|
||||
file. A convenient way to generate that is to run the test, let it fail
|
||||
(because CPython cannot run it) and then copy the .out file (but not
|
||||
before checking it manually!)
|
||||
|
||||
When creating new tests, anything that relies on float support should go in the
|
||||
float/ subdirectory. Anything that relies on import x, where x is not a built-in
|
||||
module, should go in the import/ subdirectory.
|
||||
|
||||
## perf_bench
|
||||
|
||||
The `perf_bench` directory contains some performance benchmarks that can be used
|
||||
to benchmark different MicroPython firmwares or host ports.
|
||||
|
||||
The runner utility is `run-perfbench,py`. Execute `./run-perfbench.py --help`
|
||||
for a full list of command line options.
|
||||
|
||||
### Benchmarking a target
|
||||
|
||||
To run tests on a firmware target using `pyboard.py`, run the command line like
|
||||
this:
|
||||
|
||||
```
|
||||
./run-perfbench.py -p -d /dev/ttyACM0 168 100
|
||||
```
|
||||
|
||||
* `-p` indicates running on a remote target via pyboard.py, not the host.
|
||||
* `-d PORTNAME` is the serial port, `/dev/ttyACM0` is the default if not
|
||||
provided.
|
||||
* `168` is value `N`, the approximate CPU frequency in MHz (in this case Pyboard
|
||||
V1.1 is 168MHz). It's possible to choose other values as well: lower values
|
||||
like `10` will run much the tests much quicker, higher values like `1000` will
|
||||
run much longer.
|
||||
* `100` is value `M`, the approximate heap size in kilobytes (can get this from
|
||||
`import micropython; micropython.mem_info()` or estimate it). It's possible to
|
||||
choose other values here too: lower values like `10` will run shorter/smaller
|
||||
tests, and higher values will run bigger tests. The maximum value of `M` is
|
||||
limited by available heap, and the tests are written so the "recommended"
|
||||
value is approximately the upper limit.
|
||||
|
||||
### Benchmarking the host
|
||||
|
||||
To benchmark the host build (unix/Windows), run like this:
|
||||
|
||||
```
|
||||
./run-perfbench.py 2000 10000
|
||||
```
|
||||
|
||||
The output of perfbench is a list of tests and times/scores, like this:
|
||||
|
||||
```
|
||||
N=2000 M=10000 n_average=8
|
||||
perf_bench/bm_chaos.py: SKIP
|
||||
perf_bench/bm_fannkuch.py: 94550.38 2.9145 84.68 2.8499
|
||||
perf_bench/bm_fft.py: 79920.38 10.0771 129269.74 8.8205
|
||||
perf_bench/bm_float.py: 43844.62 17.8229 353219.64 17.7693
|
||||
perf_bench/bm_hexiom.py: 32959.12 15.0243 775.77 14.8893
|
||||
perf_bench/bm_nqueens.py: 40855.00 10.7297 247776.15 11.3647
|
||||
perf_bench/bm_pidigits.py: 64547.75 2.5609 7751.36 2.5996
|
||||
perf_bench/core_import_mpy_multi.py: 15433.38 14.2733 33065.45 14.2368
|
||||
perf_bench/core_import_mpy_single.py: 263.00 11.3910 3858.35 12.9021
|
||||
perf_bench/core_qstr.py: 4929.12 1.8434 8117.71 1.7921
|
||||
perf_bench/core_yield_from.py: 16274.25 6.2584 12334.13 5.8125
|
||||
perf_bench/misc_aes.py: 57425.25 5.5226 17888.60 5.7482
|
||||
perf_bench/misc_mandel.py: 40809.25 8.2007 158107.00 9.8864
|
||||
perf_bench/misc_pystone.py: 39821.75 6.4145 100867.62 6.5043
|
||||
perf_bench/misc_raytrace.py: 36293.75 6.8501 26906.93 6.8402
|
||||
perf_bench/viper_call0.py: 15573.00 14.9931 19644.99 13.1550
|
||||
perf_bench/viper_call1a.py: 16725.75 9.8205 18099.96 9.2752
|
||||
perf_bench/viper_call1b.py: 20752.62 8.3372 14565.60 9.0663
|
||||
perf_bench/viper_call1c.py: 20849.88 5.8783 14444.80 6.6295
|
||||
perf_bench/viper_call2a.py: 16156.25 11.2956 18818.59 11.7959
|
||||
perf_bench/viper_call2b.py: 22047.38 8.9484 13725.73 9.6800
|
||||
```
|
||||
|
||||
The numbers across each line are times and scores for the test:
|
||||
|
||||
* Runtime average (microseconds, lower is better)
|
||||
* Runtime standard deviation as a percentage
|
||||
* Score average (units depend on the benchmark, higher is better)
|
||||
* Score standard deviation as a percentage
|
||||
|
||||
### Comparing performance
|
||||
|
||||
Usually you want to know if something is faster or slower than a reference. To
|
||||
do this, copy the output of each `run-perfbench.py` run to a text file.
|
||||
|
||||
This can be done multiple ways, but one way on Linux/macOS is with the `tee`
|
||||
utility: `./run-perfbench.py -p 168 100 | tee pyb-run1.txt`
|
||||
|
||||
Once you have two files with output from two different runs (maybe with
|
||||
different code or configuration), compare the runtimes with `./run-perfbench.py
|
||||
-t pybv-run1.txt pybv-run2.txt` or compare scores with `./run-perfbench.py -s
|
||||
pybv-run1.txt pybv-run2.txt`:
|
||||
|
||||
```
|
||||
> ./run-perfbench.py -s pyb-run1.txt pyb-run2.txt
|
||||
diff of scores (higher is better)
|
||||
N=168 M=100 pyb-run1.txt -> pyb-run2.txt diff diff% (error%)
|
||||
bm_chaos.py 352.90 -> 352.63 : -0.27 = -0.077% (+/-0.00%)
|
||||
bm_fannkuch.py 77.52 -> 77.45 : -0.07 = -0.090% (+/-0.01%)
|
||||
bm_fft.py 2516.80 -> 2519.74 : +2.94 = +0.117% (+/-0.00%)
|
||||
bm_float.py 5749.27 -> 5749.65 : +0.38 = +0.007% (+/-0.00%)
|
||||
bm_hexiom.py 42.22 -> 42.30 : +0.08 = +0.189% (+/-0.00%)
|
||||
bm_nqueens.py 4407.55 -> 4414.44 : +6.89 = +0.156% (+/-0.00%)
|
||||
bm_pidigits.py 638.09 -> 632.14 : -5.95 = -0.932% (+/-0.25%)
|
||||
core_import_mpy_multi.py 477.74 -> 477.57 : -0.17 = -0.036% (+/-0.00%)
|
||||
core_import_mpy_single.py 58.74 -> 58.72 : -0.02 = -0.034% (+/-0.00%)
|
||||
core_qstr.py 63.11 -> 63.11 : +0.00 = +0.000% (+/-0.01%)
|
||||
core_yield_from.py 357.57 -> 357.57 : +0.00 = +0.000% (+/-0.00%)
|
||||
misc_aes.py 397.27 -> 396.47 : -0.80 = -0.201% (+/-0.00%)
|
||||
misc_mandel.py 3375.70 -> 3375.84 : +0.14 = +0.004% (+/-0.00%)
|
||||
misc_pystone.py 2265.36 -> 2265.97 : +0.61 = +0.027% (+/-0.01%)
|
||||
misc_raytrace.py 367.61 -> 368.15 : +0.54 = +0.147% (+/-0.01%)
|
||||
viper_call0.py 605.92 -> 605.92 : +0.00 = +0.000% (+/-0.00%)
|
||||
viper_call1a.py 576.78 -> 576.78 : +0.00 = +0.000% (+/-0.00%)
|
||||
viper_call1b.py 452.45 -> 452.46 : +0.01 = +0.002% (+/-0.01%)
|
||||
viper_call1c.py 457.39 -> 457.39 : +0.00 = +0.000% (+/-0.00%)
|
||||
viper_call2a.py 561.37 -> 561.37 : +0.00 = +0.000% (+/-0.00%)
|
||||
viper_call2b.py 389.49 -> 389.50 : +0.01 = +0.003% (+/-0.01%)
|
||||
```
|
||||
|
||||
Note in particular the error percentages at the end of each line. If these are
|
||||
high relative to the percentage difference then it indicates high variability in
|
||||
the test runs, and the absolute difference value is unreliable. High error
|
||||
percentages are particularly common on PC builds, where the host OS may
|
||||
influence test run times. Increasing the `N` value may help average this out by
|
||||
running each test longer.
|
||||
Loading…
Reference in New Issue