Commit Graph

256 Commits

Author SHA1 Message Date
Damien George 3c2aa5ff93 unix/.gitignore: Remove *.py from ignore list.
.py files are valid source files and shouldn't be ignored.  This line was
from the early days when .py files in the unix directory were used for
testing.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
Damien George c4b8dae438 tests/unix: Add coverage test for freezing various objects.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
Damien George f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
stijn 5f50f4a130 unix: Show compiler info in REPL banner.
The unix port's main.c gets used by unix and windows ports, and with a
variety of compilers, so it's convenient to see which version is actually
being used immediately when starting micropython.  This is similar to what
CPython does.
2022-02-22 00:59:31 +11:00
stijn 8bb50c6301 unix/Makefile: Remove explicit addition of -std=c++ flag.
This was added merely for building the C++ user module example, so it's a
better fit to add it in the corresponding micropython.mk.
2022-02-18 14:40:16 +11:00
Damien George fbd47fc46c ports: Consolidate inclusion of umachine module in built-ins.
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c.  Enabling MICROPY_PY_MACHINE will include this
module.

As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).

Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 10:08:54 +11:00
stijn dd6967202a py/modmath: Add math.tau, math.nan and math.inf constants.
Configurable by the new MICROPY_PY_MATH_CONSTANTS option.
2022-01-23 09:28:33 +11:00
Damien George aac5a97d08 ports: Move '.frozen' to second entry in sys.path.
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.

But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.

This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).

Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:55:36 +11:00
Damien George de43b500bd py/runtime: Allow initialising sys.path/argv with defaults.
If MICROPY_PY_SYS_PATH_ARGV_DEFAULTS is enabled (which it is by default)
then sys.path and sys.argv will be initialised and populated with default
values.  This keeps all bare-metal ports aligned.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-18 00:08:07 +11:00
Jim Mussared 86ce442607 ports: Add '.frozen' as the first entry in sys.path.
Frozen modules will be searched preferentially, but gives the user the
ability to override this behavior.

This matches the previous behavior where "" was implicitly the frozen
search path, but the frozen list was checked before the filesystem.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-18 00:08:07 +11:00
Jim Mussared 92353c2911 all: Remove support for FROZEN_DIR and FROZEN_MPY_DIR.
These have been deprecated for over two years in favour of FROZEN_MANIFEST
and manifest.py.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-17 23:54:05 +11:00
Jim Mussared cc23e99f32 py/modio: Remove io.resource_stream function.
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
2021-12-17 23:53:44 +11:00
Damien George 9ffb1ad2f8 unix/Makefile: Use -Og instead of -O0 for debug builds.
For the coverage build this reduces the binary size to about 1/4 of its
size, and seems to help gcov/lcov coverage analysis so that it doesn't miss
lines.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-09 12:53:09 +11:00
Damien George 71168ec55c unix/coverage: Change remaining printf to mp_printf.
For consistency with all other prints in this file, so that the ordering
of output is correct.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-09 12:38:50 +11:00
Jim Mussared a7fa18c203 py/builtinimport: Refactor module importing.
Simplify and document/comment the handling of builtin import for:
- already-loaded modules
- built-in modules
- built-in umodules (formerly weak links)
- filesystem modules

Retains existing functionality with smaller code size but should also
facilitate potential new features (built-in packages, controlling the
frozen path).

Also makes the (unix-only) -m behavior a bit more obvious and configurable.

Code size change with this commit:

   bare-arm:    +0 +0.000%
minimal x86:   -64 -0.039%
   unix x64:   -32 -0.006%
unix nanbox:    -4 -0.001%
      stm32:  -184 -0.047% PYBV10
     cc3200:  -120 -0.065%
    esp8266:  -228 -0.033% GENERIC
      esp32:  -268 -0.018% GENERIC[incl +16(data)]
        nrf:  -152 -0.087% pca10040
        rp2:  -256 -0.052% PICO
       samd:   -80 -0.057% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2021-12-01 13:23:34 +11:00
Henk Vergonet d11ff0499f unix/modos: Add support for uos.urandom(n).
Use getrandom function if available, otherwise read from /dev/urandom.

Signed-off-by: Henk.Vergonet@gmail.com
2021-11-17 12:59:38 +11:00
Andrew Leech cc42b7c88b unix/modusocket: Support MP_STREAM_POLL in unix socket_ioctl.
Allows asyncio reading of network sockets when MICROPY_PY_USELECT is used
in the build configuration.
2021-10-19 22:48:10 +11:00
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
Jim Mussared 60c6d5594f unix: Enable LOAD_ATTR fast path, and map lookup caching.
Enabled for all variants except minimal.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:02:19 +10:00
Jim Mussared 0c2cadf1c8 unix/Makefile: Disable error compression on arm-linux-gnueabi-gcc.
This compiler is unable to optimise out the giant strcmp match generated
by MP_MATCH_COMPRESSED.

See github.com/micropython/micropython/pull/7659#issuecomment-899479793

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-17 11:20:58 +10:00
Jim Mussared 692d36d779 py: Implement partial PEP-498 (f-string) support.
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 16:58:40 +10:00
Damien George 38a204ed96 py: Introduce and use mp_raise_type_arg helper.
To reduce code size.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George 6430cd3e02 unix/variants: Enable help and help("modules") on standard and dev.
See related #1354, #2906, #3436.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-13 16:18:15 +10:00
Damien George 240888a0d2 unix/Makefile: Add back LIB_SRC_C to list of object files.
This fixes the dev build (it needs LIB_SRC_C for Bluetooth) which was
broken by 136369d72f.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-13 15:58:39 +10:00
Damien George 136369d72f all: Update to point to files in new shared/ directory.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-12 17:08:10 +10:00
Amir Gonnen cb332ddae8 unix/modffi: Add option to lock GC in callback, and cfun access.
Add an optional 'lock' kwarg to callback that locks GC and scheduler.  This
allows the callback to be invoked asynchronously in 'interrupt context',
for example as a signal handler.

Also add the 'cfun' member function to callback, that allows retrieving the
C callback function address.  This is needed when the callback should be
set to a struct field.

See related #7373.

Signed-off-by: Amir Gonnen <amirgonnen@gmail.com>
2021-06-24 23:12:26 +10:00
Damien George 38bc5a9f67 stm32: Provide a custom BTstack runloop that integrates with soft timer.
It reschedules the BT HCI poll soft timer so that it is called exactly when
the next timer expires.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-23 13:14:26 +10:00
David Lechner 259d9b69fe py/mpstate: Schedule KeyboardInterrupt on main thread.
This introduces a new macro to get the main thread and uses it to ensure
that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only
scheduled on the main thread. This is more deterministic than being
scheduled on a random thread and is more in line with CPython that only
allow signal handlers to run on the main thread.

Fixes issue #7026.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-19 09:49:00 +10:00
David Lechner ca920f7218 py/mpstate: Make exceptions thread-local.
This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t.
This allows exceptions to be scheduled on a specific thread.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-19 09:43:44 +10:00
Pavol Rusnak 95048129b1 unix: Fix build on arm64-darwin due to integer cast.
This fixes error: cast to smaller integer type 'int' from 'pthread_t'.
pthread_t is defined as long, not as int.

Signed-off-by: Pavol Rusnak <pavol@rusnak.io>
2021-06-15 00:08:24 +10:00
Abilio Marques 525a920ca5 unix/modffi: Fix conversion between Python integers and ffi types.
This commit fixes the following problems converting to/from Python integers
and ffi types:
- integers of 8 and 16 bits not working on big endian
- integers of 64 bits not working on 32 bits architectures
- unsigned returns were converted to signed Python integers

Fixes issue #7269.
2021-06-08 13:06:17 +10:00
Damien George 36cb365cad unix/main: Increase stack limit on ARM architectures.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-05 11:03:09 +10:00
Damien George 9e29217c73 unix/modffi: Use a union for passing/returning FFI values.
This fixes a bug where double arguments on a 32-bit architecture would not
be passed correctly because they only had 4 bytes of storage (not 8).  It
also fixes a compiler warning/error in return_ffi_value on certian
architectures: array subscript 'double[0]' is partly outside array bounds
of 'ffi_arg[1]' {aka 'long unsigned int[1]'}.

Fixes issue #7064.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-06 12:17:10 +10:00
Artyom Skrobov ca35c0059c py/repl: Autocomplete builtin modules.
Doing "import <tab>" will now complete/list built-in modules.

Originally at adafruit#4548 and adafruit#4608

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2021-05-02 23:11:14 +10:00
Damien George e9e9c76ddf all: Rename mp_keyboard_interrupt to mp_sched_keyboard_interrupt.
To match mp_sched_exception() and mp_sched_schedule().

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:13:43 +10:00
Marian Buschsieweke 9c9bfe1968 unix/main: Make static variable that's potentially clobbered by longjmp.
This fixes `error: variable 'subpkg_tried' might be clobbered by 'longjmp'
or 'vfork' [-Werror=clobbered]` when compiling on ppc64le and aarch64 (and
possibly other architectures/toolchains).
2021-04-14 10:45:26 +10:00
stijn a66286f3a0 unix: Improve command line argument processing.
Per CPython everything which comes after the command, module or file
argument is not an option for the interpreter itself.  Hence the processing
of options should stop when encountering those, and the remainder be passed
as sys.argv.  Note the latter was already the case for a module or file but
not for a command.

This fixes issues like 'micropython myfile.py -h' showing the help and
exiting instead of passing '-h' as sys.argv[1], likewise for
'-X <something>' being treated as a special option no matter where it
occurs on the command line.
2021-04-07 12:41:25 +10:00
Damien George d87f42b0e5 examples/usercmodules: Simplify user C module enabling.
It's a bit of a pitfall with user C modules that including them in the
build does not automatically enable them.  This commit changes the docs and
examples for user C modules to encourage writers of user C modules to
enable them unconditionally.  This makes things simpler and covers most use
cases.

See discussion in issue #6960, and also #7086.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-01 16:27:38 +11:00
Damien George 6129b8e401 tests: Rename run-tests to run-tests.py for consistency.
Signed-off-by: Damien George <damien@micropython.org>
2021-03-12 19:56:09 +11:00
Jim Mussared 4c54012373 unix/moduselect: Don't allow both posix and non-posix configurations.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-02-16 14:59:52 +11:00
Damien George 7815dd2cc5 unix/mpbtstackport_common: Implement mp_bluetooth_hci_active.
So that BTSTACK can be enabled with SYNC_EVENTS.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-12 12:07:05 +11:00
Damien George ad4656b861 all: Rename BYTES_PER_WORD to MP_BYTES_PER_OBJ_WORD.
The "word" referred to by BYTES_PER_WORD is actually the size of mp_obj_t
which is not always the same as the size of a pointer on the target
architecture.  So rename this config value to better reflect what it
measures, and also prefix it with MP_.

For uses of BYTES_PER_WORD in setting the stack limit this has been
changed to sizeof(void *), because the stack usually grows with
machine-word sized values (eg an nlr_buf_t has many machine words in it).

Signed-off-by: Damien George <damien@micropython.org>
2021-02-04 22:46:42 +11:00
stijn b9a35bebf7 py/qstr.h: Remove QSTR_FROM_STR_STATIC macro.
It practically does the same as qstr_from_str and was only used in one
place, which should actually use the compile-time MP_QSTR_XXX form for
consistency; qstr_from_str is for runtime strings only.
2021-01-30 13:40:48 +11:00
Oliver Joos 290dc1d5ee unix/modtime: Fix time() precision on unix ports with non-double floats.
With MICROPY_FLOAT_IMPL_FLOAT the results of utime.time(), gmtime() and
localtime() change only every 129 seconds.  As one consequence
tests/extmod/vfs_lfs_mtime.py will fail on a unix port with LFS support.

With this patch these functions only return floats if
MICROPY_FLOAT_IMPL_DOUBLE is used.  Otherwise they return integers.
2021-01-23 16:55:24 +11:00
Joris Peeraer 5020b14d54 py/mpprint: Fix length calculation for strings with precision-modifier.
Two issues are tackled:

1. The calculation of the correct length to print is fixed to treat the
   precision as a maximum length instead as the exact length.
   This is done for both qstr (%q) and for regular str (%s).

2. Fix the incorrect use of mp_printf("%.*s") to mp_print_strn().

   Because of the fix of above issue, some testcases that would print
   an embedded null-byte (^@ in test-output) would now fail.
   The bug here is that "%s" was used to print null-bytes. Instead,
   mp_print_strn is used to make sure all bytes are outputted and the
   exact length is respected.

Test-cases are added for both %s and %q with a combination of precision
and padding specifiers.
2020-12-07 23:32:06 +11:00
Jim Mussared 0e8af2b370 extmod/modbluetooth: Add API for L2CAP channels.
Also known as L2CAP "connection oriented channels". This provides a
socket-like data transfer mechanism for BLE.

Currently only implemented for NimBLE on STM32 / Unix.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-24 01:07:17 +11:00
Jim Mussared 61d1e4b01b extmod/nimble: Make stm32 and unix NimBLE ports use synchronous events.
This changes stm32 from using PENDSV to run NimBLE to use the MicroPython
scheduler instead.  This allows Python BLE callbacks to be invoked directly
(and therefore synchronously) rather than via the ringbuffer.

The NimBLE UART HCI and event processing now happens in a scheduled task
every 128ms.  When RX IRQ idle events arrive, it will also schedule this
task to improve latency.

There is a similar change for the unix port where the background thread now
queues the scheduled task.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-13 17:19:05 +11:00
Jim Mussared 81e92d3d6e extmod/modbluetooth: Re-instate optional no-ringbuf modbluetooth.
This requires that the event handlers are called from non-interrupt context
(i.e. the MicroPython scheduler).

This will allow the BLE stack (e.g. NimBLE) to run from the scheduler
rather than an IRQ like PENDSV, and therefore be able to invoke Python
callbacks directly/synchronously.  This allows writing Python BLE handlers
for events that require immediate response such as _IRQ_READ_REQUEST (which
was previous a hard IRQ) and future events relating to pairing/bonding.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-13 17:19:05 +11:00
Jim Mussared 4559bcb467 unix: Make mp_hal_delay_ms run MICROPY_EVENT_POLL_HOOK.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-13 17:19:05 +11:00
Andrew Leech de60aa7d6b unix: Handle pending events/scheduler in MICROPY_EVENT_POLL_HOOK.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-13 17:19:05 +11:00