Commit Graph

4097 Commits

Author SHA1 Message Date
Damien George 0149cd6b8b windows: Switch to VFS subsystem and use VfsPosix.
Following the unix port.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 00:26:36 +11:00
Damien George 94077c6402 samd/moduos: Convert module to use extmod version.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-09 10:03:23 +11:00
Damien George 926b554daf extmod/moduos: Create general uos module to be used by all ports.
Based on the rp2 port version, with the rp2 port converted to use this
module.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-09 10:03:23 +11:00
stijn 795370ca23 py/bc.h: Fix C++ compilation of public API.
Casts between unrelated types must be explicit.  Regression in
f2040bfc7e
2022-03-01 16:17:30 +11:00
Damien George f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
Damien George 18acd0318f py/gc: Update debug code to compile with changes to qstr pool types.
Following on from 18b1ba086c and
f46a7140f5.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-17 11:17:21 +11:00
Artyom Skrobov f46a7140f5 py/qstr: Use `const` consistently to avoid a cast.
Originally at adafruit#4707

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2022-02-11 22:55:02 +11:00
Artyom Skrobov 18b1ba086c py/qstr: Separate hash and len from string data.
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.

No functional change.

The size reduction depends on the number of qstrs in the build.  The change
this commit brings is:

   bare-arm:    -4 -0.007%
minimal x86:  +150 +0.092% [incl +48(data)]
   unix x64:  -608 -0.118%
unix nanbox:  -572 -0.126% [incl +32(data)]
      stm32: -1392 -0.352% PYBV10
     cc3200:  -448 -0.244%
    esp8266: -1208 -0.173% GENERIC
      esp32: -1028 -0.068% GENERIC[incl -1020(data)]
        nrf:  -440 -0.252% pca10040
        rp2: -1072 -0.217% PICO
       samd:  -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.

Originally at adafruit#4583

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2022-02-11 22:52:32 +11:00
Damien George fbd47fc46c ports: Consolidate inclusion of umachine module in built-ins.
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c.  Enabling MICROPY_PY_MACHINE will include this
module.

As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).

Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 10:08:54 +11:00
stijn dd6967202a py/modmath: Add math.tau, math.nan and math.inf constants.
Configurable by the new MICROPY_PY_MATH_CONSTANTS option.
2022-01-23 09:28:33 +11:00
stijn a9448c0a86 all: Fix MICROPY_OBJ_REPR_D compilation with msvc. 2022-01-23 09:25:01 +11:00
Jeff Epler 037b2c72a1 py/objstr: Support '{:08}'.format("Jan") like Python 3.10.
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.

See CPython version of the issue at https://bugs.python.org/issue27772

Signed-off-by: Jeff Epler <jepler@gmail.com>
2022-01-19 15:34:32 +11:00
Damien George da4b38e756 all: Bump version to 1.18.
Signed-off-by: Damien George <damien@micropython.org>
2022-01-17 09:50:31 +11:00
Damien George 772058a6bd py/mpconfig.h: Define MICROPY_PY_USSL_FINALISER only if not defined.
So a port can define it even if MICROPY_PY_USSL is not defined.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-07 23:59:17 +11:00
stijn 22cf0940e1 py/modbuiltins: Add additional macro for extending builtins.
Mainly useful for defining additional globals in boards and variants.
2022-01-07 11:36:52 +11:00
Emilie Feral 58b56b91c4 py/qstr: Reset mpstate.qstr_last_chunk before raising an error.
The qstr_last_chunk is not collected by the garbage collector.  This relies
on the assertion that qstr_pool_t also references the qstr_last_chunk.  If
an exception is raised while allocating the qstr_pool_t, qstr_last_chunk
has to be invalidated not to become a dangling reference at the next
garbage collection.

Signed-off-by: Emilie Feral <emilie.feral@numworks.com>
2022-01-06 13:00:43 +11:00
Damien George aac5a97d08 ports: Move '.frozen' to second entry in sys.path.
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.

But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.

This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).

Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:55:36 +11:00
Damien George 2c139bbf4e py/mpz: Fix bugs with bitwise of -0 by ensuring all 0's are positive.
This commit makes sure that the value zero is always encoded in an mpz_t as
neg=0 and len=0 (previously it was just len=0).

This invariant is needed for some of the bitwise operations that operate on
negative numbers, because they cannot handle -0.  For example
(-((1<<100)-(1<<100)))|1 was being computed as -65535, instead of 1.

Fixes issue #8042.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-21 18:00:05 +11:00
Damien George fe9ffff9c0 py/mpstate.h: Only include sys.path/argv objects in state when enabled.
The mp_sys_path_obj and mp_sys_argv_obj objects are only used by the
runtime and accessible from Python if MICROPY_PY_SYS is enabled.  So
exclude them from the runtime state if this option is disabled.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-19 08:55:40 +11:00
Damien George de43b500bd py/runtime: Allow initialising sys.path/argv with defaults.
If MICROPY_PY_SYS_PATH_ARGV_DEFAULTS is enabled (which it is by default)
then sys.path and sys.argv will be initialised and populated with default
values.  This keeps all bare-metal ports aligned.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-18 00:08:07 +11:00
Jim Mussared d6d4a5819b py/mkrules.cmake: Set frozen preprocessor defs early.
This ensures MICROPY_QSTR_EXTRA_POOL and MICROPY_MODULE_FROZEN_MPY are set
if necessary before the CFLAGS are extracted for QSTR generation.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-18 00:05:18 +11:00
Jim Mussared e0bf4611c3 py: Only search frozen modules when '.frozen' is found in sys.path.
This changes makemanifest.py & mpy-tool.py to merge string and mpy names
into the same list (now mp_frozen_names).

The various paths for loading a frozen module (mp_find_frozen_module) and
checking existence of a frozen module (mp_frozen_stat) use a common
function that searches this list.

In addition, the frozen lookup will now only take place if the path starts
with ".frozen", which needs to be added to sys.path.

This fixes issues #1804, #2322, #3509, #6419.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-18 00:01:59 +11:00
Jim Mussared 92353c2911 all: Remove support for FROZEN_DIR and FROZEN_MPY_DIR.
These have been deprecated for over two years in favour of FROZEN_MANIFEST
and manifest.py.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-17 23:54:05 +11:00
Jim Mussared cc23e99f32 py/modio: Remove io.resource_stream function.
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
2021-12-17 23:53:44 +11:00
Damien George d6dc4cb65a py/showbc: Fix printing of raw bytecode header on nanbox builds.
Signed-off-by: Damien George <damien@micropython.org>
2021-12-15 16:54:47 +11:00
Jim Mussared a7fa18c203 py/builtinimport: Refactor module importing.
Simplify and document/comment the handling of builtin import for:
- already-loaded modules
- built-in modules
- built-in umodules (formerly weak links)
- filesystem modules

Retains existing functionality with smaller code size but should also
facilitate potential new features (built-in packages, controlling the
frozen path).

Also makes the (unix-only) -m behavior a bit more obvious and configurable.

Code size change with this commit:

   bare-arm:    +0 +0.000%
minimal x86:   -64 -0.039%
   unix x64:   -32 -0.006%
unix nanbox:    -4 -0.001%
      stm32:  -184 -0.047% PYBV10
     cc3200:  -120 -0.065%
    esp8266:  -228 -0.033% GENERIC
      esp32:  -268 -0.018% GENERIC[incl +16(data)]
        nrf:  -152 -0.087% pca10040
        rp2:  -256 -0.052% PICO
       samd:   -80 -0.057% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2021-12-01 13:23:34 +11:00
Damien George a0890983ea py/objfun.h: Remove obsolete comments about entries in extra_args.
These two entries were removed in 049a7a8153

Signed-off-by: Damien George <damien@micropython.org>
2021-11-25 23:24:40 +11:00
Damien George 11ed94797d py/lexer: Support nested [] and {} characters within f-string params.
Signed-off-by: Damien George <damien@micropython.org>
2021-11-25 21:50:58 +11:00
Laurens Valk e2ca8ab8fc py/runtime: Allow types to use both .attr and .locals_dict.
Make it possible to proceed to a regular lookup in locals_dict if the
custom type->attr fails.  This allows type->attr to extend rather than
completely replace the lookup in locals_dict.

This is useful for custom builtin classes that have mostly regular methods
but just a few special attributes/properties.  This way, type->attr needs
to deal with the special cases only and the default lookup will be used for
generic methods.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2021-11-22 12:10:35 +11:00
Damien George 123dcdb8e5 py/modsys: Replace non-ASCII quote char with ASCII char.
The source code should stay 7-bit ASCII clean.

Signed-off-by: Damien George <damien@micropython.org>
2021-11-19 17:26:04 +11:00
Damien George 78ab2eeda3 py/showbc: Print unary-op string when dumping bytecode.
Signed-off-by: Damien George <damien@micropython.org>
2021-11-19 17:05:40 +11:00
Laurens Valk fe120484b6 py/gc: Add hook to run code during time consuming GC operations.
This makes it possible for cooperative multitasking systems to keep running
event loops during garbage collector operations.

For example, this can be used to ensure that a motor control loop runs
approximately each 5 ms.  Without this hook, the loop time can jump to
about 15 ms.

Addresses #3475.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2021-11-01 15:39:37 +11:00
Damien George c62351fbd6 py/mpconfig.h: Revert MICROPY_REPL_INFO to disabled at all levels.
This is an stm32-specific feature that's accessed via the pyb module, so
not something that will be widely enabled.

Signed-off-by: Damien George <damien@micropython.org>
2021-11-01 15:18:22 +11:00
Jim Mussared 0e236eef08 py/mpconfig.h: Define the "extra" feature level.
Some of these will later be moved to CORE or BASIC, but EXTRA is a good
starting point based on what stm32 uses.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-11-01 14:57:28 +11:00
Mike Wadsten fe2bc92b4d py/runtime: Fix crash when exc __new__ doesn't return an exc instance.
See CPython bug https://bugs.python.org/issue39091 for more details.
2021-10-21 12:32:16 +11:00
Damien George 8412568e7b py: Add wrapper macros so hot VM functions can go in fast code location.
For example, on esp32 they can go in iRAM to improve performance.

Signed-off-by: Damien George <damien@micropython.org>
2021-10-15 23:31:19 +11:00
stijn ea880d5674 py/builtinimport: Forward all debug printing to MICROPY_DEBUG_PRINTER. 2021-09-24 13:17:19 +10:00
iabdalkader 2c5e9bbdfa extmod: Add platform module.
It contains the compiler version, and underlying system HAL/SDK version.
2021-09-19 23:35:10 +10:00
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
Jim Mussared 11ef8f22fe py/map: Add an optional cache of (map+index) to speed up map lookups.
The existing inline bytecode caching optimisation, selected by
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, reserves an extra byte in the
bytecode after certain opcodes, which at runtime stores a map index of the
likely location of this field when looking up the qstr.  This scheme is
incompatible with bytecode-in-ROM, and doesn't work with native generated
code.  It also stores bytecode in .mpy files which is of a different format
to when the feature is disabled, making generation of .mpy files more
complex.

This commit provides an alternative optimisation via an approach that adds
a global cache for map offsets, then all mp_map_lookup operations use it.
It's less precise than bytecode caching, but allows the cache to be
independent and external to the bytecode that is executing.  It also works
for the native emitter and adds a similar performance boost on top of the
gain already provided by the native emitter.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:02:19 +10:00
Jim Mussared 7b89ad8dbf py/vm: Add a fast path for LOAD_ATTR on instance types.
When the LOAD_ATTR opcode is executed there are quite a few different cases
that have to be handled, but the common case is accessing a member on an
instance type.  Typically, built-in types provide methods which is why this
is common.

Fortunately, for this specific case, if the member is found in the member
map then there's no further processing.

This optimisation does a relatively cheap check (type is instance) and then
forwards directly to the member map lookup, falling back to the regular
path if necessary.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:02:15 +10:00
Jim Mussared 01374d941f py/mpconfig.h: Define initial templates for "feature levels".
This is the beginning of a set of changes to simplify enabling/disabling
features.  The goals are:
- Remove redundancy from mpconfigport.h (never set a value to the default
  -- make it clear exactly what's being enabled).
- Improve consistency between ports.  All "similar" ports (i.e. approx same
  flash size) should get the same features.
- Simplify mpconfigport.h -- just get default/sensible options for the size
  of the port.
- Make it easy for defining constrained boards (e.g. STM32F0/L0), they can
  just set a lower level.

This commit makes a step towards this and defines the "core" level as the
current default feature set, and a "minimal" level to turn off everything.
And a few placeholder levels are added for where the other ports will
roughly land.

This is a no-op change for all ports.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 13:19:11 +10:00
Damien George 426785a19e py/emitnative: Ensure load_subscr does not clobber existing REG_RET.
Fixes issue #7782, and part of issue #6314.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-13 22:30:24 +10:00
Damien George e6850838cd py/parse: Simplify parse nodes representing a list.
This commit simplifies and optimises the parse tree in-memory
representation of lists of expressions, for tuples and lists, and when
tuples are used on the left-hand-side of assignments and within del
statements.  This reduces memory usage of the parse tree when such code is
compiled, and also reduces the size of the compiler.

For example, (1,) was previously the following parse tree:

    expr_stmt(5) (n=2)
      atom_paren(45) (n=1)
        testlist_comp(146) (n=2)
          int(1)
          testlist_comp_3b(149) (n=1)
            NULL
      NULL

and with this commit is now:

    expr_stmt(5) (n=2)
      atom_paren(45) (n=1)
        testlist_comp(146) (n=1)
          int(1)
      NULL

Similarly, (1, 2, 3) was previously:

    expr_stmt(5) (n=2)
      atom_paren(45) (n=1)
        testlist_comp(146) (n=2)
          int(1)
          testlist_comp_3c(150) (n=2)
            int(2)
            int(3)
      NULL

and is now:

    expr_stmt(5) (n=2)
      atom_paren(45) (n=1)
        testlist_comp(146) (n=3)
          int(1)
          int(2)
          int(3)
      NULL

Signed-off-by: Damien George <damien@micropython.org>
2021-09-10 14:09:44 +10:00
Damien George af64c2ddbd extmod/machine_pwm: Factor out machine.PWM bindings to common code.
This commit refactors machine.PWM and creates extmod/machine_pwm.c.  The
esp8266, esp32 and rp2 ports all use this and provide implementations of
the required PWM functionality.  This helps to reduce code duplication and
keep the same Python API across ports.

This commit does not make any functional changes.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-04 16:31:17 +10:00
Damien George d41f6dde56 extmod/modonewire: Make _onewire module configurable via macro option.
Signed-off-by: Damien George <damien@micropython.org>
2021-09-02 13:11:23 +10:00
Damien George afe0634c98 extmod/machine_spi: Make SoftSPI configurable via macro option.
Signed-off-by: Damien George <damien@micropython.org>
2021-09-02 13:11:23 +10:00
Damien George 122d901ef1 extmod/machine_i2c: Make SoftI2C configurable via macro option.
The zephyr port doesn't support SoftI2C so it's not enabled, and the legacy
I2C constructor check can be removed.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-02 13:11:23 +10:00
Damien George 7c54b64280 all: Bump version to 1.17.
Signed-off-by: Damien George <damien@micropython.org>
2021-09-02 00:07:13 +10:00
Jim Mussared b51e7e9d01 stm32: Disable computed goto on constrained boards.
Saves ~1kiB.  Add comment to this effect to mpconfig.h.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-20 20:18:52 +10:00
Jim Mussared 85adc25558 py/mkrules.mk: Do submodule sync in "make submodules".
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-19 22:53:44 +10:00
Jim Mussared e64cda5295 stm32: Add implementation of machine.bitstream.
Hand-written version for M0, and cycle-counter version for everything else.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-19 22:50:32 +10:00
Jim Mussared 870000f35b extmod: Add machine.bitstream.
This is a generic API for synchronously bit-banging data on a pin.

Initially this adds a single supported encoding, which supports controlling
WS2812 LEDs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-19 22:50:11 +10:00
Jim Mussared 5555f147df py/lexer: Clear fstring_args vstr on lexer free.
This was missed in 692d36d779.  It's not
strictly necessary as the GC will clean it anyway, but it's good to
pre-emptively gc_free() all the blocks used in lexing/parsing.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-19 17:31:02 +10:00
Jim Mussared 692d36d779 py: Implement partial PEP-498 (f-string) support.
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 16:58:40 +10:00
Damien George 78718fffb1 py/mkrules: Automatically build mpy-cross if it doesn't exist.
Commit 4173950658 removed automatic building
of mpy-cross, which rebuilt it whenever any of its dependent source files
changed.

But needing to build mpy-cross, and not knowing how, is a frequent issue.
This commit aims to help by automatically building mpy-cross only if it
doesn't exist.  For Makefiles it uses an order-only prerequisite, while
for CMake it uses a custom command.

If MICROPY_MPYCROSS (which is what makemanifest.py uses to locate the
mpy-cross executable) is defined in the environment then automatic build
will not be attempted, allowing a way to prevent this auto-build if needed.

Thanks to Trammell Hudson aka @osresearch for the original idea; see #5760.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-07 20:25:32 +10:00
Peter Züger ffc854f17f extmod/modujson: Add support for dump/dumps separators keyword-argument.
Optionally enabled via MICROPY_PY_UJSON_SEPARATORS.  Enabled by default.

For dump, make sure mp_get_stream_raise is called after
mod_ujson_separators since CPython does it in this order (if both
separators and stream are invalid, separators will raise an exception
first).

Add separators argument in the docs as well.

Signed-off-by: Peter Züger <zueger.peter@icloud.com>
Signed-off-by: Damien George <damien@micropython.org>
2021-08-07 13:52:16 +10:00
David Lechner afcc77cebc py/builtinimport: Fix condition for including do_execute_raw_code().
Commit e33bc597 ("py: Remove calls to file reader functions when these
are disabled.") changed the condition for one caller of
do_execute_raw_code() from

    MICROPY_PERSISTENT_CODE_LOAD

to

    MICROPY_HAS_FILE_READER && MICROPY_PERSISTENT_CODE_LOAD

The condition that enables compiling the function itself needs to be
changed to match.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-31 16:51:58 +10:00
Jim Mussared 4e39ff221a py/runtime: Fix bool unary op for subclasses of native types.
Previously a subclass of a type that didn't implement unary_op, or didn't
handle MP_UNARY_OP_BOOL, would raise TypeError on bool conversion.

Fixes #5677.
2021-07-23 12:40:00 +10:00
Jim Mussared 0e3752e82a py/emitnative: Ensure stack settling is safe mid-branch.
And add a test for the case where REG_RET could be in use.

Fixes #7523.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-07-19 23:18:59 +10:00
Damien George d0227d5862 py/emitnative: Reuse need_reg_all func in need_stack_settled.
To reduce code size and code duplication.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-19 23:18:13 +10:00
David Lechner 8758504f0f extmod/moduselect: Conditionally compile select().
This adds #if MICROPY_PY_USELECT_SELECT around the uselect.select()
function. According to the docs, this function is only for CPython
compatibility and should not normally be used. So we can disable it
and save a few bytes of flash space where possible.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-17 23:32:39 +10:00
Damien George 70b8e1d1f5 py/obj: Fix formatting of comment for mp_obj_is_integer.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George 022b8a7fea py/objexcept: Make mp_obj_new_exception_arg1 inline.
This function is rarely used so making it inline reduces code size.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George 74085f167e py/modsys: Optimise sys.exit for code size by using exception helpers.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George 38a204ed96 py: Introduce and use mp_raise_type_arg helper.
To reduce code size.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George bb00125aaa py: Support single argument to optimised MP_OBJ_STOP_ITERATION.
The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a
StopIteration() exception object, and means that heap memory does not need
to be allocated for the exception (in cases where it can be used).  This
commit allows this optimised object to take an optional argument (before,
it could only have no argument).

The commit also adds some new tests to cover corner cases with
StopIteration and generators that previously did not work.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George e3825e28e6 py/objexcept: Make mp_obj_exception_get_value support subclassed excs.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George b8255dd2e0 py/vm: Simplify handling of MP_OBJ_STOP_ITERATION in yield-from opcode.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George 136369d72f all: Update to point to files in new shared/ directory.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-12 17:08:10 +10:00
Bryan Tong Minh 3d9af87721 windows/Makefile: Add .exe extension to executables name.
Uses the same logic applied in 5b57ae985f
to determine when to add .exe.

See related: #3310, #3361, #3370, #4143, #5727.
2021-07-08 12:35:08 +10:00
David Lechner d934f8c8a8 py/makeversionhdr: Add --tags arg to git describe.
This adds the --tags argument to the git describe command that is used
to define the MICROPY_GIT_TAG macro. This makes it match non-annotated
tags. This is useful for MicroPython derivatives that don't use
annotated tags.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-05 10:41:31 -05:00
David Lechner 58e4d72338 py/objexcept: Pretty print OSError also when it has 2 arguments.
This extends pretty-printing of OSError's to handle two arguments when the
exception name is known.

Signed-off-by: David Lechner <david@pybricks.com>
2021-07-01 13:23:54 +10:00
Yonatan Goldschmidt 4ada56d4cb tools/makemanifest.py: Allow passing flags to mpy-tool.py. 2021-06-28 01:50:00 +03:00
Damien George cfd08448a1 py: Mark unused arguments from bytecode decoding macros.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-25 10:58:22 +10:00
Damien George 08e0e065f4 py/makeqstrdefs.py: Don't include .h files explicitly in preprocessing.
Only include .c and .cpp files explicitly in the list of files passed to
the preprocessor for QSTR extraction.  All relevant .h files will be
included in this process by "#include" from the .c(pp) files.  In
particular for moduledefs.h, this is included by py/objmodule.c (and
doesn't actually contain any extractable MP_QSTR_xxx, but rather defines
macros with MP_QSTR_xxx's in them which are then part of py/objmodule.c).

The main reason for this change is to simplify the preprocessing step on
the javascript port, which tries to compile .h files as C++ precompiled
headers if they are passed with -E to clang.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-25 10:50:54 +10:00
David Lechner b51ae20c07 py/mperrno: Add MP_ECANCELED error code.
This is useful when binding asynchronous functions in C.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-24 23:14:01 +10:00
Jeff Epler 413f34cd8f all: Fix signed shifts and NULL access errors from -fsanitize=undefined.
Fixes the following (the line numbers match commit 0e87459e2b):

../../extmod/crypto-algorithms/sha256.c:49:19: runtime error: left shif...
../../extmod/moduasyncio.c:106:35: runtime error: member access within ...
../../py/binary.c:210:13: runtime error: left shift of negative value -...
../../py/mpz.c:744:16: runtime error: negation of -9223372036854775808 ...
../../py/objint.c:109:22: runtime error: left shift of 1 by 31 places c...
../../py/objint_mpz.c:374:9: runtime error: left shift of 4611686018427...
../../py/objint_mpz.c:374:9: runtime error: left shift of negative valu...
../../py/parsenum.c:106:14: runtime error: left shift of 46116860184273...
../../py/runtime.c:395:33: runtime error: left shift of negative value ...
../../py/showbc.c:177:28: runtime error: left shift of negative value -...
../../py/vm.c:321:36: runtime error: left shift of negative value -1```

Testing was done on an amd64 Debian Buster system using gcc-8.3 and these
settings:

    CFLAGS += -g3 -Og -fsanitize=undefined
    LDFLAGS += -fsanitize=undefined

The introduced TASK_PAIRHEAP macro's conditional (x ? &x->i : NULL)
assembles (under amd64 gcc 8.3 -Os) to the same as &x->i, since i is the
initial field of the struct.  However, for the purposes of undefined
behavior analysis the conditional is needed.

Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-06-24 23:01:04 +10:00
David Lechner 259d9b69fe py/mpstate: Schedule KeyboardInterrupt on main thread.
This introduces a new macro to get the main thread and uses it to ensure
that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only
scheduled on the main thread. This is more deterministic than being
scheduled on a random thread and is more in line with CPython that only
allow signal handlers to run on the main thread.

Fixes issue #7026.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-19 09:49:00 +10:00
David Lechner ca920f7218 py/mpstate: Make exceptions thread-local.
This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t.
This allows exceptions to be scheduled on a specific thread.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-19 09:43:44 +10:00
Damien George 7c51cb2307 all: Bump version to 1.16.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-18 16:38:06 +10:00
Damien George bc89cdeb45 py/gc: Only use no_sanitize_address attribute for GCC 4.8 and above.
It's not supported on older GCC versions.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-18 14:15:37 +10:00
Damien George 5e1d3c8b5d py/stackctrl: Prevent unused-var warning when stack checking disabled.
Signed-off-by: Damien George <damien@micropython.org>
2021-06-05 11:03:09 +10:00
Damien George a70a4e6688 py/emitglue: Always flush caches when assigning native ARM code.
Prior to this commit, cache flushing for ARM native code was done only in
the assembler code asm_thumb_end_pass()/asm_arm_end_pass(), at the last
pass of the assembler.  But this misses flushing the cache when loading
native code from an .mpy file, ie in persistentcode.c.

The change here makes sure the cache is always flushed/cleaned/invalidated
when assigning native code on ARM architectures.

This problem was found running tests/micropython/import_mpy_native_gc.py on
the mimxrt port.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-05 11:03:04 +10:00
Damien George 53519e322a py/builtinimport: Change relative import's ValueError to ImportError.
Following CPython change, see https://bugs.python.org/issue37444.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-30 19:35:03 +10:00
Jeff Epler d67f4115b4 py/repl: Don't read past the end of import_str.
asan considers that memcmp(p, q, N) is permitted to access N bytes at each
of p and q, even for values of p and q that have a difference earlier.
Accessing additional values is frequently done in practice, reading 4 or
more bytes from each input at a time for efficiency, so when completing
"non_exist<TAB>" in the repl, this causes a diagnostic:

    ==16938==ERROR: AddressSanitizer: global-buffer-overflow on
    address 0x555555cd8dc8 at pc 0x7ffff726457b bp 0x7fffffffda20 sp 0x7fff
    READ of size 9 at 0x555555cd8dc8 thread T0
        #0 0x7ffff726457a  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xb857a)
        #1 0x555555b0e82a in mp_repl_autocomplete ../../py/repl.c:301
        #2 0x555555c89585 in readline_process_char ../../lib/mp-readline/re
        #3 0x555555c8ac6e in readline ../../lib/mp-readline/readline.c:513
        #4 0x555555b8dcbd in do_repl /home/jepler/src/micropython/ports/uni
        #5 0x555555b90859 in main_ /home/jepler/src/micropython/ports/unix/
        #6 0x555555b90a3a in main /home/jepler/src/micropython/ports/unix/m
        #7 0x7ffff619a09a in __libc_start_main ../csu/libc-start.c:308
        #8 0x55555595fd69 in _start (/home/jepler/src/micropython/ports/uni

    0x555555cd8dc8 is located 0 bytes to the right of global variable
    'import_str' defined in '../../py/repl.c:285:23' (0x555555cd8dc0) of
    size 8
      'import_str' is ascii string 'import '

Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-05-30 11:50:51 +10:00
Jeff Epler 9a74546f8d py/gc: Access the list of root pointers in an asan-compatible way.
Signed-off-by: Jeff Epler <jepler@gmail.com>
2021-05-30 11:50:51 +10:00
Jeff Epler f2dbc91022 py/compile: Raise an error on async with/for outside an async function.
A simple reproducer is:

   async for x in (): x

Before this change, it would cause an assertion error in mpy-cross and
micropython-coverage.
2021-05-30 10:38:48 +10:00
Damien George 4ee8ec6931 py/asmarm: Use builtin func to flush I- and D-cache on ARM 7 archs.
The inline assembler code does not work for __ARM_ARCH == 7.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-26 16:24:00 +10:00
Damien George e61ac453dc py/mkrules.cmake: Add MPY_LIB_DIR and BOARD_DIR to makemanifest call.
So that the FROZEN_MANIFEST option in cmake works the same as make.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-23 00:00:39 +10:00
Damien George 5176a2d732 py/emitnative: Fix x86-64 emitter to generate correct 8/16-bit stores.
Fixes issue #6643.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-20 23:43:25 +10:00
Damien George f49d47c167 py/asmx64: Support use of top 8 regs in src_r64 argument.
Signed-off-by: Damien George <damien@micropython.org>
2021-05-20 23:43:25 +10:00
Bob Abeles 7ceccad4e2 py/nlrx64: Correct the detection of Darwin ABI.
__APPLE__ tests for an Apple OS and __MACH__ tests that it is based on CMU
Mach.  Using both tests ensures that just Darwin is recognized.
2021-05-18 11:52:00 +10:00
Bob Abeles 126b1c7271 py/nlraarch64: Add underscore prefix to function symbols for Darwin ABI.
The proper way to do this is to test for __APPLE__ and __MACH__, where
__APPLE__ tests for an Apple OS and __MACH__ tests that it is based on CMU
Mach.  Using both tests ensures that just Darwin (Apple's open source base
for MacOS, iOS, etc.) is recognized. __APPLE__ by itself will test for any
Apple OS, which can include older OS 7-9 and any future Apple OS. __MACH__
tests for any OS based on CMU Mach, including Darwin and GNU Hurd.

Fixes #7232.
2021-05-18 11:46:30 +10:00
Damien George 6d2680fa36 py/objarray: Fix constructing a memoryview from a memoryview.
Fixes issue #7261.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-18 10:18:56 +10:00
Damien George 1446107b4d py/objarray: Use mp_obj_memoryview_init helper in mp_obj_new_memoryview.
Signed-off-by: Damien George <damien@micropython.org>
2021-05-18 00:22:53 +10:00
stijn 09be0c083c py/objarray: Implement more/less comparisons for array. 2021-05-13 22:16:14 +10:00
stijn 57365d8557 py/objarray: Prohibit comparison of mismatching types.
Array equality is defined as each element being equal but to keep
code size down MicroPython implements a binary comparison.  This
can only be used correctly for elements with the same binary layout
though so turn it into an NotImplementedError when comparing types
for which the binary comparison yielded incorrect results: types
with different sizes, and floating point numbers because nan != nan.
2021-05-13 22:16:14 +10:00
Damien George 300fc842ce py/mkenv.mk: Don't emit info about BUILD_VERBOSE if it's set.
If the user sets V or BUILD_VERBOSE then they don't need to see this
message.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-12 13:22:13 +10:00
Damien George b6b39bff47 py/gc: Make gc_lock_depth have a count per thread.
This commit makes gc_lock_depth have one counter per thread, instead of one
global counter.  This makes threads properly independent with respect to
the GC, in particular threads can now independently lock the GC for
themselves without locking it for other threads.  It also means a given
thread can run a hard IRQ without temporarily locking the GC for all other
threads and potentially making them have MemoryError exceptions at random
locations (this really only occurs on MCUs with multiple cores and no GIL,
eg on the rp2 port).

The commit also removes protection of the GC lock/unlock functions, which
is no longer needed when the counter is per thread (and this also fixes the
cas where a hard IRQ calling gc_lock() may stall waiting for the mutex).

It also puts the check for `gc_lock_depth > 0` outside the GC mutex in
gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from
waiting on a mutex if it does attempt to allocate heap memory (and putting
the check outside the GC mutex is now safe now that there is a
gc_lock_depth per thread).

Signed-off-by: Damien George <damien@micropython.org>
2021-05-10 13:07:16 +10:00
Artyom Skrobov ca35c0059c py/repl: Autocomplete builtin modules.
Doing "import <tab>" will now complete/list built-in modules.

Originally at adafruit#4548 and adafruit#4608

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2021-05-02 23:11:14 +10:00
Artyom Skrobov 7556e01f14 py/repl: Refactor autocomplete, extracting reusable parts.
Originally at adafruit#4548

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2021-05-02 23:11:12 +10:00
Artyom Skrobov f85ea8d4fe py/repl: Refactor autocomplete to reduce nesting.
Originally at adafruit#4548

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2021-05-02 23:11:10 +10:00
scottbelden befbff31b7 py/repl: Enter four spaces when there are no matches.
Originally at adafruit#1859

Signed-off-by: scottbelden <scottabelden@gmail.com>
2021-05-02 23:11:07 +10:00
Kathryn Lingel 1f1a54d0b1 py/repl: Filter private methods from tab completion.
Anything beginning with "_" will now only be tab-completed if there is
already a partial match for such an entry.  In other words, entering
foo.<tab> will no longer complete/list anything beginning with "_".

Originally at adafruit#1850

Signed-off-by: Kathryn Lingel <kathryn@lingel.net>
2021-05-02 23:11:03 +10:00
Damien George aa061ae391 py/scheduler: Add missing MICROPY_WRAP_MP_SCHED_EXCEPTION usage.
This was missed in commit 7cbf826a95.

Signed-off-by: Damien George <damien@micropython.org>
2021-05-01 23:10:26 +10:00
Damien George 6b7c8d3e72 py/runtime: Remove commented-out code from mp_deinit().
These commented-out lines of code have been unused for a long time, so
remove them to avoid confusion as to why they are there.

mp_obj_dict_free() never existed, this line was converted from
mp_map_deinit() and commented out as soon as it was added.  The call to
mp_map_deinit(mp_loaded_modules_map) was commented in
1a1d11fa32.

Fixes issue #3507.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 16:35:14 +10:00
Damien George 916c3fd23f py/scheduler: Add optional port hook for when something is scheduled.
So that a port can "wake up" when there is work to do.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:32:16 +10:00
Damien George e9e9c76ddf all: Rename mp_keyboard_interrupt to mp_sched_keyboard_interrupt.
To match mp_sched_exception() and mp_sched_schedule().

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:13:43 +10:00
Damien George 7cbf826a95 py/scheduler: Add mp_sched_exception() to schedule a pending exception.
This helper is added to properly set a pending exception, to mirror
mp_sched_schedule(), which schedules a function.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:13:43 +10:00
Damien George 7e549b6718 py/profile: Use mp_handle_pending() to raise pending exception.
If MICROPY_ENABLE_SCHEDULER is enabled then MP_STATE_VM(sched_state) must
be updated after handling the pending exception, which is done by the
mp_handle_pending() function.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:13:43 +10:00
Damien George c5cbfd545a py/dynruntime.h: Add mp_obj_get_array() function.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-28 00:06:46 +10:00
Damien George d4b706c4d0 py: Add option to compile without any error messages at all.
This introduces a new option, MICROPY_ERROR_REPORTING_NONE, which
completely disables all error messages.  To be used in cases where
MicroPython needs to fit in very limited systems.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-27 23:51:52 +10:00
Damien George 3c4bfd1dec py/objexcept: Support errno attribute on OSError exceptions.
This commit adds the errno attribute to exceptions, so code can retrieve
errno codes from an OSError using exc.errno.

The implementation here simply lets `errno` (and the existing `value`)
attributes work on any exception instance (they both alias args[0]).  This
is for efficiency and to keep code size down.  The pros and cons of this
are:

Pros:
- more compatible with CPython, less difference to document and learn
- OSError().errno will correctly return None, whereas the current way of
  doing it via OSError().args[0] will raise an IndexError
- it reduces code size on most bare-metal ports (because they already have
  the errno qstr)
- for Python code that uses exc.errno the generated bytecode is 2 bytes
  smaller and more efficient to execute (compared with exc.args[0]); so
  bytecode loaded to RAM saves 2 bytes RAM for each use of this attribute,
  and bytecode that is frozen saves 2 bytes flash/ROM for each use
- it's easier/shorter to type, and saves 2 bytes of space in .py files that
  use it (for each use)

Cons:
- increases code size by 4-8 bytes on minimal ports that don't already have
  the `errno` qstr
- all exceptions now have .errno and .value attributes (a cpydiff test is
  added to address this)

See also #2407.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-23 22:03:46 +10:00
Damien George 321d1897c3 all: Bump version to 1.15.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-19 00:11:51 +10:00
matejcik 1a2ffda175 py/runtime: Make sys.modules preallocate to a configurable size.
This allows configuring the pre-allocated size of sys.modules dict, in
order to prevent unwanted reallocations at run-time (3 sys-modules is
really not quite enough for a larger project).
2021-04-12 22:36:16 +10:00
matejcik b26def0644 py/profile: Resolve name collision with STATIC unset.
When building with STATIC undefined (e.g., -DSTATIC=), there are two
instances of mp_type_code that collide at link time: in profile.c and in
builtinevex.c.  This patch resolves the collision by renaming one of them.
2021-04-12 22:31:42 +10:00
Damien George 5dcc9b3b16 py/py.cmake: Introduce MICROPY_INC_CORE as a list with core includes.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-09 13:08:35 +10:00
Damien George 0fabda31de py/py.cmake: Move qstr helper code to micropy_gather_target_properties.
Signed-off-by: Damien George <damien@micropython.org>
2021-04-09 13:08:35 +10:00
Phil Howard 0cf12dd59c rp2: Add support for USER_C_MODULES to CMake build system.
The parts that are generic are added to py/ so they can be used by other
ports that use CMake.

py/usermod.cmake:

* Creates a usermod target to hang user C/CXX modules from.
* Gathers sources from user C/CXX modules and libs for QSTR scan.

ports/rp2/CMakeLists.txt:

* Includes py/usermod.cmake.
* Links the resulting usermod library to the MicroPython target.

py/mkrules.cmake:

Add cxxflags to qstr.i.last custom command for CXX modules:

* MICROPY_CPP_FLAGS so CXX modules will find includes.
* -DNO_QSTR to fix fatal error missing "genhdr/qstrdefs.generated.h".

Usage:

The rp2 port can be linked against user C modules by running:

make USER_C_MODULES=/path/to/module/micropython.cmake

CMake will print a list of included modules.

Co-authored-by: Graham Sanderson <graham.sanderson@raspberrypi.org>
Co-authored-by: Michael O'Cleirigh <michael.ocleirigh@rivulet.ca>
Signed-off-by: Phil Howard <phil@pimoroni.com>
2021-03-31 00:26:01 +11:00
Damien George 9fef1c0bde py: Rename remaining object types to be of the form mp_type_xxx.
For consistency with all other object types in the core.

Signed-off-by: Damien George <damien@micropython.org>
2021-03-26 13:48:34 +11:00
Damien George 42cf77f48b py/vm: For tracing use mp_printf, and print state when thread enabled.
mp_printf should be used to print the prefix because it's also used in
mp_bytecode_print2 (otherwise, depending on the system, different output
streams may be used).

Also print the current thread state when threading is enabled to easily see
which thread executes what opcode.

Signed-off-by: Damien George <damien@micropython.org>
2021-03-17 12:13:53 +11:00
Yonatan Goldschmidt e196cb762e py/nlrx64: Fix typo in comment. 2021-03-11 12:51:10 +11:00
Yonatan Goldschmidt 2d5cece5ac py/nlr: Implement NLR for AArch64. 2021-03-11 12:51:10 +11:00
Damien George 2adf20c5f2 py/mkrules.cmake: Add MICROPY_QSTRDEFS_PORT to qstr build process.
This allows a port to specify a custom qstrdefsport.h file, the same as the
QSTR_DEFS variable in a Makefile.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-21 15:56:54 +11:00
Damien George d867d20d9a py/mkrules.cmake: Rename QSTR_DEFS variables to QSTRDEFS.
And also MICROPY_PY_QSTRDEFS to MICROPY_QSTRDEFS_PY.  These variables are
all related.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-21 15:56:54 +11:00
Maureen Helm dff6fc64d2 py: Expand lists in core cmake custom commands.
The core cmake rules use custom commands to invoke qstr processing
scripts. For the zephyr port, it's possible that list arguments to these
commands may contain generator expressions, therefore we need to expand
them properly.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
2021-02-16 18:38:13 +11:00
Damien George 66098c0985 py,extmod: Add core cmake rule files.
These allow a port to use cmake natively instead of make.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-15 12:48:18 +11:00
Damien George 0a59938574 py/mpz: Fix overflow of borrow in mpn_div.
For certain operands to mpn_div, the existing code path for
`DIG_SIZE == MPZ_DBL_DIG_SIZE / 2` had a bug in it where borrow could still
overflow in the `(x >= *n || *n - x <= borrow)` branch, ie
`borrow + x - (mpz_dbl_dig_t)*n` overflows the borrow variable.  In such
cases the subsequent right-shift of borrow would not bring in the overflow
bit, leading to an error in the result.  An example division that had
overflow when MPZ_DIG_SIZE = 16 is `(2 ** 48 - 1) ** 2 // (2 ** 48 - 1)`.

This is fixed in this commit by simplifying the code and handling the low
digits of borrow first, and then the upper bits (to shift down) separately.
There is no longer a distinction between `DIG_SIZE < MPZ_DBL_DIG_SIZE / 2`
and `DIG_SIZE == MPZ_DBL_DIG_SIZE / 2`.

This commit also simplifies the second part of the calculation so that
borrow does not need to be negated (instead the code just works knowing
that borrow is negative and using + instead of - in calculations involving
borrow).

Fixes #6777.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-08 11:50:05 +11:00
Damien George 9dedcf122d py/gc: Change include of stdint.h to stddef.h.
No std-int types are used in gc.h, but size_t is which needs stddef.h.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-05 15:46:56 +11:00
Damien George c891190c69 py: Rename WORD_MSBIT_HIGH to MP_OBJ_WORD_MSBIT_HIGH.
To make it clear it is for mp_obj_t/mp_uint_t "word" types, and to prefix
this macro with MP_.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-04 22:46:42 +11:00
Damien George ad4656b861 all: Rename BYTES_PER_WORD to MP_BYTES_PER_OBJ_WORD.
The "word" referred to by BYTES_PER_WORD is actually the size of mp_obj_t
which is not always the same as the size of a pointer on the target
architecture.  So rename this config value to better reflect what it
measures, and also prefix it with MP_.

For uses of BYTES_PER_WORD in setting the stack limit this has been
changed to sizeof(void *), because the stack usually grows with
machine-word sized values (eg an nlr_buf_t has many machine words in it).

Signed-off-by: Damien George <damien@micropython.org>
2021-02-04 22:46:42 +11:00
Damien George 7e956fae28 py: Rename BITS_PER_BYTE to MP_BITS_PER_BYTE.
To give this macro a standard MP_ prefix.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-04 22:46:42 +11:00
Damien George 8a41ee19c2 py: Remove BITS_PER_WORD definition.
It's only used in one location, to test if << or >> will overflow when
shifting mp_uint_t.  For such a test it's clearer to use sizeof(lhs_val),
which will be valid even if the type of lhs_val changes.

Signed-off-by: Damien George <damien@micropython.org>
2021-02-04 22:46:42 +11:00
Xiang Xiao 5fdf351178 py/gc: Don't include mpconfig.h and misc.h in gc.h.
Because gc.h doesn't reference any symbol from these header files.

Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
2021-02-04 22:37:26 +11:00
Damien George 78b23c3a1f all: Bump version to 1.14.
Signed-off-by: Damien George <damien@micropython.org>
2021-02-03 00:59:07 +11:00
iTitou 4fb5f012c3 py/makeversionhdr: Honor SOURCE_DATE_EPOCH if present.
This environment variable, if defined during the build process,
indicates a fixed time that should be used in place of "now" when
such a time is explicitely referenced.

This allows for reproducible builds of micropython.
See https://reproducible-builds.org/specs/source-date-epoch/

Signed-off-by: iTitou <moiandme@gmail.com>
2021-01-31 17:48:59 +01:00
David CARLIER cb30928ac8 py/persistentcode: Introduce MICROPY_PERSISTENT_CODE_SAVE_FILE option.
This should be enabled when the mp_raw_code_save_file function is needed.

It is enabled for mpy-cross, and a check for defined(__APPLE__) is added to
cover Mac M1 systems.
2021-01-30 15:13:24 +11:00
stijn cb8e2f02ab py/gc: Fix debug printing of pointer.
When DEBUG_printf is the standard printf, compilers require the value for
%p to be an actual pointer instead of an integer.
2021-01-30 14:41:29 +11:00
stijn b9a35bebf7 py/qstr.h: Remove QSTR_FROM_STR_STATIC macro.
It practically does the same as qstr_from_str and was only used in one
place, which should actually use the compile-time MP_QSTR_XXX form for
consistency; qstr_from_str is for runtime strings only.
2021-01-30 13:40:48 +11:00
graham sanderson 794df0f1d5 py/emitnative: Support binary ops on ARMv6M without use of ite instr. 2021-01-29 23:57:10 +11:00
Damien George 75fea330bf py/emitinlinethumb: Exclude code using #if when ARMV7M disabled.
So there are no references to undeclared asm_thumb_mov_reg_i16().

Signed-off-by: Damien George <damien@micropython.org>
2021-01-29 23:57:10 +11:00
Damien George c9f4c5acd6 py/emitnative: Ensure encoding to load prelude_offset doesn't change sz.
Based on change made by Graham Sanderson.

Signed-off-by: Damien George <damien@micropython.org>
2021-01-29 23:57:10 +11:00
graham sanderson 40d2010882 py/asmthumb: Add support for ARMv6M in native emitter.
Adds a new compile-time option MICROPY_EMIT_THUMB_ARMV7M which is enabled
by default (to get existing behaviour) and which should be disabled (set to
0) when building native emitter support (@micropython.native) on ARMv6M
targets.
2021-01-29 23:57:10 +11:00
Damien George 925bd67cfb py/objfun: Support fun.__globals__ attribute.
This returns a reference to the globals dict associated with the function,
ie the global scope that the function was defined in.  This attribute is
read-only but the dict itself is modifiable, per CPython behaviour.

Signed-off-by: Damien George <damien@micropython.org>
2021-01-29 23:57:10 +11:00
Damien George 246b2e016a py/mkrules.mk: Remove stray vpath and unused -Itmp, add $(Q) for $(AR).
Signed-off-by: Damien George <damien@micropython.org>
2020-12-14 13:57:15 +11:00
Damien George 0091041f5a py/modmath: Simplify handling of positional args to reduce code size.
As a general pattern, required positional arguments that are not named do
not need to be parsed using mp_arg_parse_all().

Signed-off-by: Damien George <damien@micropython.org>
2020-12-14 13:30:56 +11:00
Joris Peeraer 5020b14d54 py/mpprint: Fix length calculation for strings with precision-modifier.
Two issues are tackled:

1. The calculation of the correct length to print is fixed to treat the
   precision as a maximum length instead as the exact length.
   This is done for both qstr (%q) and for regular str (%s).

2. Fix the incorrect use of mp_printf("%.*s") to mp_print_strn().

   Because of the fix of above issue, some testcases that would print
   an embedded null-byte (^@ in test-output) would now fail.
   The bug here is that "%s" was used to print null-bytes. Instead,
   mp_print_strn is used to make sure all bytes are outputted and the
   exact length is respected.

Test-cases are added for both %s and %q with a combination of precision
and padding specifiers.
2020-12-07 23:32:06 +11:00
Damien George be24e6a53f py/mpprint: Prevent case fall-through when assert is disabled.
Signed-off-by: Damien George <damien@micropython.org>
2020-11-29 17:31:24 +11:00
Jim Mussared 0e8af2b370 extmod/modbluetooth: Add API for L2CAP channels.
Also known as L2CAP "connection oriented channels". This provides a
socket-like data transfer mechanism for BLE.

Currently only implemented for NimBLE on STM32 / Unix.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-24 01:07:17 +11:00
Jim Mussared a7932ae4e6 tools/makeqstrdefs.py: Run qstr preprocessing in parallel.
This gives a substantial speedup of the preprocessing step, i.e. the
generation of qstr.i.last.  For example on a clean build, making
qstr.i.last:

    21s -> 4s on STM32 (WB55)
    8.9 -> 1.8s on Unix (dev).

Done in collaboration with @stinos.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-12 15:04:53 +11:00
Damien George d7e1526593 py/binary: Fix sign extension setting wide integer on 32-bit archs.
Signed-off-by: Damien George <damien@micropython.org>
2020-11-11 22:18:24 +11:00
Damien George 1fef5662ab py/mpz: Do sign extension in mpz_as_bytes for negative values.
Signed-off-by: Damien George <damien@micropython.org>
2020-11-11 22:18:24 +11:00
stijn 0153148fd2 py/py.mk: Support C++ code for user C modules.
Support C++ code in .cpp files by providing CXX counterparts of the
_USERMOD_ flags we have for C already.  This merely enables the Makefile of
user C modules to use variables specific to C++ compilation, it is still up
to each port's main Makefile to also include these in the build.
2020-10-29 15:29:20 +11:00
stijn e498a8bd13 py: Workaround clang error when building misc.h with C++ compiler. 2020-10-29 15:28:14 +11:00
stijn 8e94fa0d2e py/makeqstrdefs.py: Support preprocessing C++ files for QSTR generation.
When SCR_QSTR contains C++ files they should be preprocessed with the same
compiler flags (CXXFLAGS) as they will be compiled with, to make sure code
scanned for QSTR occurrences is effectively the code used in the rest of
the build.  The 'split SCR_QSTR in .c and .cpp files and process each with
different flags' logic isn't trivial to express in a Makefile and the
existing principle for deciding which files to preprocess was already
rather complicated, so the actual preprocessing is moved into
makeqstrdefs.py completely.
2020-10-29 15:27:30 +11:00
stijn f1666419a8 py/mkrules.mk: Add target for compiling C++ files.
Move the target from the ESP32 Makefile since that does what is needed
already, but also include files from user C modules as is done for the C
files.
2020-10-29 15:27:18 +11:00
stijn 2b9f0586e7 py/makeqstrdefs.py: Process C++ files as well.
Preprocessed C++ code isn't different from C code when it comes to QSTR
instances so process it as well.
2020-10-29 15:27:11 +11:00
stijn 1b723937e3 py/makeqstrdefs.py: Fix beaviour when scanning non-C preprocessed files.
When process_file() is passed a preprocessed C++ file for instance it won't
find any lines containing .c files and the last_fname variable remains
None, so handle that gracefully.
2020-10-29 15:26:35 +11:00
Emil Renner Berthing ccd92335a1 py, extmod: Introduce and use MP_FALLTHROUGH macro.
Newer GCC versions are able to warn about switch cases that fall
through.  This is usually a sign of a forgotten break statement, but in
the few cases where a fall through is intended we annotate it with this
macro to avoid the warning.
2020-10-22 11:53:16 +02:00
Emil Renner Berthing f1f6ef7b17 py/vmentrytable: Ignore GCC -Woverride-init.
Like Clang, GCC warns about this file, but only with -Woverride-init
which is enabled by -Wextra. Disable the warnings for this file just
like we do for Clang to make -Wextra happy.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing 9aa58cf8ba py, extmod: Add explicit initializers for default values.
When compiling with -Wextra which includes -Wmissing-field-initializers
GCC will warn that the defval field of mp_arg_val_t is not initialized.
This is just a warning as it is defined to be zero initialized, but since
it is a union it makes sense to be explicit about which member we're
going to use, so add the explicit initializers and get rid of the
warning.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing fdd6fa389e py: Use unsigned comparison of chars.
On x86 chars are signed, but we're comparing a char to '0' + unsigned int,
which is promoted to an unsigned int. Let's promote the char to unsigned
before doing the comparison to avoid weird corner cases.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing 6d3aa16443 py/objexcept: Compare mp_emergency_exception_buf_size signed.
mp_emergency_exception_buf_size is signed, so let's make sure we compare
it as such.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing 6324c3e054 py/scope: Name and use id_kind_type_t.
The function scope_find_or_add_id used to take a scope_kind_t enum and
save it in an uint8_t. Saving an enum in a uint8_t is fine, but
everywhere this function is called it is not actually given a
scope_kind_t but an anonymous enum instead. Let's give this enum a name
and use that as the argument type.

This doesn't change the generated code, but is a C type mismatch that
unfortunately doesn't show up unless you enable -Wenum-conversion.
2020-10-22 11:40:56 +02:00
Jim Mussared b137d064e9 py/objtype: Handle __dict__ attribute when type has no locals. 2020-10-10 00:16:32 +11:00
Jim Mussared 880875bea1 py/objdict: Add mp_const_empty_dict_obj, use it for mp_const_empty_map. 2020-10-10 00:16:26 +11:00
Damien George 843dcd4f85 py/parse: Expose rule-name printing as MICROPY_DEBUG_PARSE_RULE_NAME.
So it can be enabled without modifying the source.

Signed-off-by: Damien George <damien@micropython.org>
2020-10-01 15:26:43 +10:00
Mike Wadsten c711c0049e py/makeversionhdr.py: Match only git tags which look like versions.
Some downstream projects may use tags in their repositories for more than
just designating MicroPython releases.  In those cases, the
makeversionhdr.py script would end up using a different tag than intended.
So tell `git describe` to only match tags that look like a MicroPython
version tag, such as `v1.12` or `v2.0`.
2020-10-01 11:01:43 +10:00
Damien George 50e34f979c py/objarray.h: Add mp_obj_memoryview_init() helper function.
Signed-off-by: Damien George <damien@micropython.org>
2020-09-25 12:23:11 +10:00
Iyassou Shimels ca017841d6 py/objstr: Make bytes(bytes_obj) return bytes_obj.
Calling the bytes constructor on a bytes object returns the original bytes
object.  This saves allocating a new instance, and matches CPython.

Signed-off-by: Iyassou Shimels <s.iyassou@gmail.com>
2020-09-24 11:04:58 +10:00
Jim Mussared 9d1983f078 py/dynruntime.h: Add mp_import_* and mp_load/store_*.
These functions already exist in the fun table, and this commit just adds
convenience macros for them.
2020-09-18 18:34:02 +10:00
Damien George 8f20cdc353 all: Rename absolute time-based functions to include "epoch".
For time-based functions that work with absolute time there is the need for
an Epoch, to set the zero-point at which the absolute time starts counting.
Such functions include time.time() and filesystem stat return values.  And
different ports may use a different Epoch.

To make it clearer what functions use the Epoch (whatever it may be), and
make the ports more consistent with their use of the Epoch, this commit
renames all Epoch related functions to include the word "epoch" in their
name (and remove references to "2000").

Along with this rename, the following things have changed:

- mp_hal_time_ns() is now specified to return the number of nanoseconds
  since the Epoch, rather than since 1970 (but since this is an internal
  function it doesn't change anything for the user).

- littlefs timestamps on the esp8266 have been fixed (they were previously
  off by 30 years in nanoseconds).

Otherwise, there is no functional change made by this commit.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-18 17:20:34 +10:00
Damien George acdb0608b7 py/parse: Pass in an mp_print_t to mp_parse_node_print.
So the output can be redirected if needed.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-11 23:00:03 +10:00
Damien George 85f2b239d8 py/showbc: Pass in an mp_print_t struct to all bytecode-print functions.
So the output can be redirected if needed.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-11 17:22:28 +10:00
stijn 2e54d9d146 py: Fix handling of NaN in certain pow implementations.
Adds a new compile-time option MICROPY_PY_MATH_POW_FIX_NAN for use with
toolchains that don't handle pow-of-NaN correctly.
2020-09-11 10:04:57 +10:00
Damien George 8d5a40c86e py/objfloat: Fix handling of negative float to power of nan.
Prior to this commit, pow(-2, float('nan')) would return (nan+nanj), or
raise an exception on targets that don't support complex numbers.  This is
fixed to return simply nan, as CPython does.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-11 10:03:57 +10:00
stijn 40ad8f1666 all: Rename "sys" module to "usys".
This is consistent with the other 'micro' modules and allows implementing
additional features in Python via e.g. micropython-lib's sys.

Note this is a breaking change (not backwards compatible) for ports which
do not enable weak links, as "import sys" must now be replaced with
"import usys".
2020-09-04 00:10:24 +10:00
Damien George b0932fcf2e all: Bump version to 1.13.
Signed-off-by: Damien George <damien@micropython.org>
2020-09-02 12:01:26 +10:00
Damien George 06659077a8 all: Update Python code to conform to latest black formatting.
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:

- If there is a trailing comma in a list (eg [], () or function call) then
  that list is now written out with one line per element.  So remove such
  trailing commas where the list should stay on one line.

- Spaces at the start of """ doc strings are removed.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 15:18:01 +10:00
Damien George ee50a6effe py/mphal.h: Introduce mp_hal_time_ns and implement on various ports.
This should return a 64-bit value being the number of nanoseconds since
1970/1/1.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 16:13:44 +10:00
Damien George 5f9b105244 py/runtime: Fix builtin compile() in "single" mode so it prints exprs.
As per CPython behaviour, compile(stmt, "file", "single") should create
code which prints to stdout (via __repl_print__) the results of any
expressions in stmt.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 11:38:46 +10:00
Damien George 9883d8e818 py/persistentcode: Maintain root ptr list of imported native .mpy code.
On ports where normal heap memory can contain executable code (eg ARM-based
ports such as stm32), native code loaded from an .mpy file may be reclaimed
by the GC because there's no reference to the very start of the native
machine code block that is reachable from root pointers (only pointers to
internal parts of the machine code block are reachable, but that doesn't
help the GC find the memory).

This commit fixes this issue by maintaining an explicit list of root
pointers pointing to native code that is loaded from an .mpy file.  This
is not needed for all ports so is selectable by the new configuration
option MICROPY_PERSISTENT_CODE_TRACK_RELOC_CODE.  It's enabled by default
if a port does not specify any special functions to allocate or commit
executable memory.

A test is included to test that native code loaded from an .mpy file does
not get reclaimed by the GC.

Fixes #6045.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-02 22:34:09 +10:00
Jonathan Hogg 37e1b5c891 py/compile: Don't await __aiter__ special method in async-for.
MicroPython's original implementation of __aiter__ was correct for an
earlier (provisional) version of PEP492 (CPython 3.5), where __aiter__ was
an async-def function.  But that changed in the final version of PEP492 (in
CPython 3.5.2) where the function was changed to a normal one.  See
https://www.python.org/dev/peps/pep-0492/#why-aiter-does-not-return-an-awaitable
See also the note at the end of this subsection in the docs:
https://docs.python.org/3.5/reference/datamodel.html#asynchronous-iterators
And for completeness the BPO: https://bugs.python.org/issue27243

To be consistent with the Python spec as it stands today (and now that
PEP492 is final) this commit changes MicroPython's behaviour to match
CPython:  __aiter__ should return an async-iterable object, but is not
itself awaitable.

The relevant tests are updated to match.

See #6267.
2020-07-25 00:58:18 +10:00
Damien George a853fff838 py/obj.h: Fix mp_seq_replace_slice_no_grow to use memmove not memcpy.
Because the argument arrays may overlap, as show by the new tests in this
commit.

Also remove the debugging comments for these macros, add a new comment
about overlapping regions, and separate the macros by blank lines to make
them easier to read.

Fixes issue #6244.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-21 01:05:59 +10:00
Damien George 332d83343f py: Rework mp_convert_member_lookup to properly handle built-ins.
This commit fixes lookups of class members to make it so that built-in
functions that are used as methods/functions of a class work correctly.

The mp_convert_member_lookup() function is pretty much completely changed
by this commit, but for the most part it's just reorganised and the
indenting changed.  The functional changes are:

- staticmethod and classmethod checks moved to later in the if-logic,
  because they are less common and so should be checked after the more
  common cases.

- The explicit mp_obj_is_type(member, &mp_type_type) check is removed
  because it's now subsumed by other, more general tests in this function.

- MP_TYPE_FLAG_BINDS_SELF and MP_TYPE_FLAG_BUILTIN_FUN type flags added to
  make the checks in this function much simpler (now they just test this
  bit in type->flags).

- An extra check is made for mp_obj_is_instance_type(type) to fix lookup of
  built-in functions.

Fixes #1326 and #6198.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-30 23:55:32 +10:00
Damien George d06ae1d2b1 py/obj.h: Make existing MP_TYPE_FLAG_xxx macros sequential.
There's no reason to have them non-sequential, this was likely a typo from
commit 9ec1caf42e.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-30 23:55:32 +10:00
Damien George 9f911d822e py/objcomplex: Add mp_obj_get_complex_maybe for use in complex bin-op.
This allows complex binary operations to fail gracefully with unsupported
operation rather than raising an exception, so that special methods work
correctly.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 01:03:10 +10:00
Damien George 41fa8b5482 py/emitnative: Implement binary operations for viper uint operands.
uint types in viper mode can now be used for all binary operators except
floor-divide and modulo.

Fixes issue #1847 and issue #6177.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 00:24:04 +10:00
Damien George b3b8706d27 py/asm: Add condition codes for signed comparisons.
Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 00:24:04 +10:00
Damien George aa26fe62d8 py/asm: Add funcs/macros to emit machine code for logical-shift-right.
Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 00:24:04 +10:00
Damien George 457fdf61c3 py/objtype: Support passing in an OrderedDict to type() as the locals.
An OrderedDict can now be used for the locals when creating a type
explicitly via type(name, bases, locals).

Signed-off-by: Damien George <damien@micropython.org>
2020-06-24 12:05:40 +10:00
Damien George 456a3abe8d py/obj.h: Add public mp_obj_is_dict_or_ordereddict() helper macro.
And use it in py/objdict.c instead of mp_obj_is_dict_type.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-24 12:05:00 +10:00
Damien George 6164c7e666 py/misc.h: Add missing semi-colon in mp_float_union_t for big-endian.
Fixes issue #6161.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-22 13:42:24 +10:00
Damien George f2e267da68 py/compile: Implement PEP 526, syntax for variable annotations.
This addition to the grammar was introduced in Python 3.6.  It allows
annotating the type of a varilable, like:

    x: int = 123
    s: str

The implementation in this commit is quite simple and just ignores the
annotation (the int and str bits above).  The reason to implement this is
to allow Python 3.6+ code that uses this feature to compile under
MicroPython without change, and for users to use type checkers.

In the future viper could use this syntax as a way to give types to
variables, which is currently done in a bit of an ad-hoc way, eg
x = int(123).  And this syntax could potentially be used in the inline
assembler to define labels in an way that's easier to read.
2020-06-16 23:18:01 +10:00
Damien George 131b0de70a py/grammar.h: Consolidate duplicate sub-rules for :test and =test. 2020-06-16 23:18:01 +10:00
Damien George 1783950311 py/compile: Implement PEP 572, assignment expressions with := operator.
The syntax matches CPython and the semantics are equivalent except that,
unlike CPython, MicroPython allows using := to assign to comprehension
iteration variables, because disallowing this would take a lot of code to
check for it.

The new compile-time option MICROPY_PY_ASSIGN_EXPR selects this feature and
is enabled by default, following MICROPY_PY_ASYNC_AWAIT.
2020-06-16 22:02:24 +10:00
Damien George 0fd91e39b1 py/compile: Convert scope test to SCOPE_IS_COMP_LIKE macro.
This macro can be used elsewhere.
2020-06-16 21:42:37 +10:00
David Lechner ecd7826316 tools/codeformat.py: Remove sizeof fixup.
Formatting for `* sizeof` was fixed in uncrustify v0.71, so we no longer
need the fixups for it.  Also, there was one file where the updated
uncrustify caught a problem that the regex didn't pick up, which is updated
in this commit.

Signed-off-by: David Lechner <david@pybricks.com>
2020-06-14 20:24:18 +10:00
Damien George f3062b5cbd py/obj.h: Clarify comments about mp_map_t is_fixed and is_ordered.
Long ago, prior to 0ef01d0a75, fixed and
ordered maps were the same setting with the "table_is_fixed_array" member
of mp_map_t.  But these settings are actually independent, and it is
possible to have is_fixed=1, is_ordered=0 (although this can currently
only be done by tools/cc1).  So update the comments to reflect this.
2020-06-10 22:27:10 +10:00
Andrew Leech 95cbe6b65e py/objtype: Use mp_obj_dict_copy() for creating obj.__dict__ attribute.
The resulting dict is now marked as read-only (is_fixed=1) to enforce the
fact that changes to this dict will not be reflected in the class instance.

This commit reduces code size by about 20 bytes, and should be more
efficient because it creates a direct copy of the dict rather than
reinserting all elements.
2020-06-10 22:03:39 +10:00
Andrew Leech 28370c0450 py/objtype: Add __dict__ attribute for class objects.
The behavior mirrors the instance object dict attribute where a copy of the
local attributes are provided (unless the dict is read-only, then that dict
itself is returned, as an optimisation).  MicroPython does not support
modifying this dict because the changes will not be reflected in the class.

The feature is only enabled if MICROPY_CPYTHON_COMPAT is set, the same as
the instance version.
2020-06-10 21:58:13 +10:00
Damien George bd06c698f0 py/dynruntime.h: Make mp_obj_str_get_str raise if arg not a str/bytes. 2020-06-08 09:05:03 +02:00
Jim Mussared e6881f0829 extmod/modbluetooth: Make modbluetooth event not a bitfield.
There doesn't appear to be any use for only triggering on specific events,
so it's just easier to number them sequentially.  This makes them smaller
values so they take up only 1 byte in the ringbuf, only 1 byte for the
opcode in the bytecode, and makes room for more events.

Also add a couple of new event types that need to be implemented (to avoid
re-numbering later).

And rename _COMPLETE and _STATUS to _DONE for consistency.

In the future the "trigger" keyword argument can be reinstated by requiring
the user to compute the bitmask, eg:

    ble.irq(handler, 1 << _IRQ_SCAN_RESULT | 1 << _IRQ_SCAN_DONE)
2020-06-05 14:04:20 +10:00
Damien George 203b10703e py/modbuiltins: Fix getattr to work with class raising AttributeError.
Fixes issue #6089.
2020-06-02 15:42:20 +10:00
David Lechner 093fd80760 py/modsys: Use consistent naming pattern for module-level const objects.
This renames a few identifiers to follow the usual naming convention of
mp_<module>_<name>.  This makes them easier to find, e.g. when grep'ing.
2020-05-28 10:02:14 +10:00
stijn 97ccde0c43 py/ringbuf: Fix compilation with msvc.
Older versions do not have "inline" so fetch the definition from
mpconfigport.h.
2020-05-28 09:56:18 +10:00
stijn 81db22f693 py/modmath: Work around msvc float bugs in atan2, fmod and modf.
Older implementations deal with infinity/negative zero incorrectly.  This
commit adds generic fixes that can be enabled by any port that needs them,
along with new tests cases.
2020-05-28 09:54:54 +10:00
Damien George a902b69dd5 py/py.mk: Use additional CFLAGS to compile string0.c.
Otherwise functions like memset might get optimised to call themselves (eg
with gcc 10).  And provide CFLAGS_BUILTIN so these options can be changed
by a port if needed.

Fixes issue #6053.
2020-05-27 23:10:23 +10:00
Damien George 801f7dca78 py/nativeglue.h: Rename "setjmp" entry to "setjmp_" to avoid any clash.
Because some compilers may define setjmp to something.

Fixes issue #6032.
2020-05-14 21:48:05 +10:00
Damien George 172fc040aa py/parse: Make mp_parse_node_extract_list return size_t instead of int.
Because this function can only return non-negative values, and having the
correct return type gives more information to the caller.
2020-05-09 00:55:44 +10:00
Damien George f792e6c283 py/scheduler: Convert mp_sched_full and mp_sched_num_pending to macros.
So they are guaranteed to be inlined within functions like
mp_sched_schedule which may be located in a special memory region.
2020-05-08 23:20:45 +10:00
Damien George 4ede703687 py/parse: Support constant folding of power operator for integers.
Constant expression like "2 ** 3" will now be folded, and the special form
"X = const(2 ** 3)" will now compile because the argument to the const is
now a constant.

Fixes issue #5865.
2020-05-03 16:23:19 +10:00
Damien George 544c308c18 py/scheduler: Add option to wrap mp_sched_schedule in arbitrary attr.
So ports can put it in a special memory section if needed.
2020-04-30 23:47:11 +10:00
Damien George bd6ca15444 py/modio: Allow uio.IOBase streams to return errno for read/write error.
This allows user code that inherits from uio.IOBase to return an errno
error code from the user readinto/write function, by returning a negative
value.  Eg returning -123 means an errno of 123.  This is already how the
custom ioctl works.
2020-04-27 23:58:46 +10:00
Damien George e08ca78f40 py/stream: Remove mp_stream_errno and use system errno instead.
This change is made for two reasons:

1. A 3rd-party library (eg berkeley-db-1.xx, axtls) may use the system
   provided errno for certain errors, and yet MicroPython stream objects
   that it calls will be using the internal mp_stream_errno.  So if the
   library returns an error it is not known whether the corresponding errno
   code is stored in the system errno or mp_stream_errno.  Using the system
   errno in all cases (eg in the mp_stream_posix_XXX wrappers) fixes this
   ambiguity.

2. For systems that have threading the system-provided errno should always
   be used because the errno value is thread-local.

For systems that do not have an errno, the new lib/embed/__errno.c file is
provided.
2020-04-27 23:58:46 +10:00
Jim Mussared 57fce3bdb2 py/objdict: Fix popitem for ordered dicts.
The popitem method wasn't implemented for ordered dicts and would result in
an invalid state.

Fixes issue #5956.
2020-04-27 23:53:17 +10:00
stijn 84fa3312cf all: Format code to add space after C++-style comment start.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
2020-04-23 11:24:25 +10:00
Damien George 388d419ba3 py/makecompresseddata.py: Make compression deterministic.
Error string compression is not deterministic in certain cases: it depends
on the Python version (whether dicts are ordered by default or not) and
probably also the order files are passed to this script, leading to a
difference in which words are included in the top 128 most common.

The changes in this commit use OrderedDict to keep parsed lines in a known
order, and, when computing how many bytes are saved by a given word, it
uses the word itself to break ties (which would otherwise be "random").
2020-04-20 10:32:49 +10:00
stijn f31f9a8b70 py/objint: Do not use fpclassify.
For combinations of certain versions of glibc and gcc the definition of
fpclassify always takes float as argument instead of adapting itself to
float/double/long double as required by the C99 standard.  At the time of
writing this happens for instance for glibc 2.27 with gcc 7.5.0 when
compiled with -Os and glibc 3.0.7 with gcc 9.3.0.  When calling fpclassify
with double as argument, as in objint.c, this results in an implicit
narrowing conversion which is not really correct plus results in a warning
when compiled with -Wfloat-conversion.  So fix this by spelling out the
logic manually.
2020-04-18 22:42:24 +10:00
stijn 70affd9ba2 all: Fix implicit floating point to integer conversions.
These are found when building with -Wfloat-conversion.
2020-04-18 22:42:24 +10:00
stijn bcf01d1686 all: Fix implicit conversion from double to float.
These are found when building with -Wfloat-conversion.
2020-04-18 22:42:24 +10:00
stijn dc4d119d3d py/objarray: Fix sign mismatch in comparison.
Found when compiling with clang and -Wsign-compare.
2020-04-18 22:42:19 +10:00
stijn 0ba68f8a1d all: Fix implicit floating point promotion.
Initially some of these were found building the unix coverage variant on
MacOS because that build uses clang and has -Wdouble-promotion enabled, and
clang performs more vigorous promotion checks than gcc.  Additionally the
codebase has been compiled with clang and msvc (the latter with warning
level 3), and with MICROPY_FLOAT_IMPL_FLOAT to find the rest of the
conversions.

Fixes are implemented either as explicit casts, or by using the correct
type, or by using one of the utility functions to handle floating point
casting; these have been moved from nativeglue.c to the public API.
2020-04-18 22:36:14 +10:00
stijn b909e8b2dd Revert "all: Fix implicit casts of float/double, and signed comparison."
This reverts commit a2110bd3fc.  There's
nothing inherently wrong with it, but upcoming commits will apply similar
fixes in a slightly different way.
2020-04-18 22:36:06 +10:00
Damien George 2725a79192 py: Always give noop defines when MICROPY_ROM_TEXT_COMPRESSION disabled.
This commit provides a typedef for mp_rom_error_text_t, and a macro define
for MP_COMPRESSED_ROM_TEXT, when MICROPY_ROM_TEXT_COMPRESSION is disabled.
This simplifies the configuration (it no longer has a special case for
MICROPY_ENABLE_DYNRUNTIME) and makes it work for other cases that don't use
compression (eg examples/embedding).  This commit also ensures
MICROPY_ROM_TEXT_COMPRESSION is defined during qstr processing.
2020-04-14 22:13:11 +10:00
Romain Goyet bd63c26dd5 py/scope: Add assert to check that low numbered qstrs do fit in uint8_t. 2020-04-13 22:27:27 +10:00
Damien George 7654907e1e py/makecompresseddata.py: Don't prefix str with mark if not compressed. 2020-04-13 22:21:57 +10:00
Damien George 8e048d2548 all: Clean up error strings to use lowercase and change cannot to can't.
Now that error string compression is supported it's more important to have
consistent error string formatting (eg all lowercase English words,
consistent contractions).  This commit cleans up some of the strings to
make them more consistent.
2020-04-13 22:19:37 +10:00
Jim Mussared 8470cd0be9 py/scheduler: Add assert that scheduler is locked when unlocking.
And add a test that shows how this can happen when multiple threads are
accessing the scheduler, which fails if atomic sections are not used.
2020-04-13 21:55:47 +10:00
Jim Mussared 243805d776 py/scheduler: Fix race in checking scheduler pending state.
Because the atomic section starts after checking whether the scheduler
state is pending, it's possible it can become a different state by the time
the atomic section starts.

This is especially likely on ports where MICROPY_BEGIN_ATOMIC_SECTION is
implemented with a mutex (i.e. it might block), but the race exists
regardless, i.e. if a context switch occurs between those two lines.
2020-04-13 21:55:47 +10:00
Damien George e292296d52 py/objexcept: Remove optional TimeoutError exception.
TimeoutError was added back in 077812b2ab for
the cc3200 port. In f522849a4d the cc3200
port enabled use of it in the socket module aliased to socket.timeout.  So
it was never added to the builtins.  Then it was replaced by
OSError(ETIMEDOUT) in 047af9b10b.

The esp32 port enables this exception, since the very beginning of that
port, but it could never be accessed because it's not in builtins.

It's being removed: 1) to not encourage its use; 2) because there are a lot
of other OSError subclasses which are not defined at all, and having
TimeoutError is a bit inconsistent.

Note that ports can add anything to the builtins via MICROPY_PORT_BUILTINS.
And they can also define their own exceptions using the
MP_DEFINE_EXCEPTION() macro.
2020-04-09 16:09:38 +10:00
Damien George 4914731e58 py/parse: Remove unnecessary check in const folding for ** operator.
In this part of the code there is no way to get the ** operator, so no need
to check for it.

This commit also adds tests for this, and other related, invalid const
operations.
2020-04-09 16:02:39 +10:00
Jim Mussared def76fe4d9 all: Use MP_ERROR_TEXT for all error messages. 2020-04-05 15:02:06 +10:00
Jim Mussared 85858e72df py/objexcept: Allow compression of exception message text.
The decompression of error-strings is only done if the string is accessed
via printing or via er.args.  Tests are added for this feature to ensure
the decompression works.
2020-04-05 15:02:06 +10:00
Jim Mussared 154b4eb354 py: Implement "common word" compression scheme for error messages.
The idea here is that there's a moderate amount of ROM used up by exception
text.  Obviously we try to keep the messages short, and the code can enable
terse errors, but it still adds up.  Listed below is the total string data
size for various ports:

    bare-arm 2860
    minimal 2876
    stm32 8926  (PYBV11)
    cc3200 3751
    esp32 5721

This commit implements compression of these strings.  It takes advantage of
the fact that these strings are all 7-bit ascii and extracts the top 128
frequently used words from the messages and stores them packed (dropping
their null-terminator), then uses (0x80 | index) inside strings to refer to
these common words.  Spaces are automatically added around words, saving
more bytes.  This happens transparently in the build process, mirroring the
steps that are used to generate the QSTR data.  The MP_COMPRESSED_ROM_TEXT
macro wraps any literal string that should compressed, and it's
automatically decompressed in mp_decompress_rom_string.

There are many schemes that could be used for the compression, and some are
included in py/makecompresseddata.py for reference (space, Huffman, ngram,
common word).  Results showed that the common-word compression gets better
results.  This is before counting the increased cost of the Huffman
decoder.  This might be slightly counter-intuitive, but this data is
extremely repetitive at a word-level, and the byte-level entropy coder
can't quite exploit that as efficiently.  Ideally one would combine both
approaches, but for now the common-word approach is the one that is used.

For additional comparison, the size of the raw data compressed with gzip
and zlib is calculated, as a sort of proxy for a lower entropy bound.  With
this scheme we come within 15% on stm32, and 30% on bare-arm (i.e. we use
x% more bytes than the data compressed with gzip -- not counting the code
overhead of a decoder, and how this would be hypothetically implemented).

The feature is disabled by default and can be enabled by setting
MICROPY_ROM_TEXT_COMPRESSION at the Makefile-level.
2020-04-05 14:20:57 +10:00
Jim Mussared c34e7b9d4c py/dynruntime.mk: Set MICROPY_ENABLE_DYNRUNTIME instead of per module.
So this setting could be used by other source files if needed.
2020-04-05 14:13:02 +10:00
Jim Mussared a9a745e4b4 py: Use preprocessor to detect error reporting level (terse/detailed).
Instead of compiler-level if-logic.  This is necessary to know what error
strings are included in the build at the preprocessor stage, so that string
compression can be implemented.
2020-04-05 14:11:51 +10:00
David Lechner a2110bd3fc all: Fix implicit casts of float/double, and signed comparison.
These were found by buiding the unix coverage variant on macOS (so clang
compiler).  Mostly, these are fixing implicit cast of float/double to
mp_float_t which is one of those two and one mp_int_t to size_t fix for
good measure.
2020-03-30 12:04:21 +11:00
Damien George 1a3e386c67 all: Remove spaces inside and around parenthesis.
Using new options enabled in the uncrustify configuration.
2020-03-28 23:36:44 +11:00
Damien George bc009fdd62 extmod/uasyncio: Add optional implementation of core uasyncio in C.
Implements Task and TaskQueue classes in C, using a pairing-heap data
structure.  Using this reduces RAM use of each Task, and improves overall
performance of the uasyncio scheduler.
2020-03-26 01:25:45 +11:00
Damien George 6c7e78de72 py/pairheap: Add helper function to initialise a new node. 2020-03-26 01:21:04 +11:00
Damien George c47a3ddf4a py/pairheap: Properly unlink node on pop and delete.
This fixes a bug in the pairing-heap implementation when nodes are deleted
with mp_pairheap_delete and then reinserted later on.
2020-03-26 01:21:04 +11:00
Damien George f8fc78691d py/mpconfig.h: Enable MICROPY_MODULE_GETATTR by default.
To enable lazy loading of submodules (among other things), which is very
useful for MicroPython libraries that want to have optional subcomponents.

Disabled explicitly on minimal ports.
2020-03-26 01:21:04 +11:00
David Lechner 2f7d2bb3e2 py/stream.h: Include sys/types.h to get size_t and off_t for POSIX API. 2020-03-25 01:00:52 +11:00
Damien George feb2577585 all: Remove spaces between nested paren and inside function arg paren.
Using new options enabled in the uncrustify configuration.
2020-03-25 00:39:46 +11:00
Damien George ad9a0ec8ab all: Convert exceptions to use mp_raise_XXX helpers in remaining places. 2020-03-18 17:26:19 +11:00
Andrew Leech 86bfabec11 py/modmicropython: Add heap_locked function to test state of heap.
This commit adds micropython.heap_locked() which returns the current
lock-depth of the heap, and can be used by Python code to check if the heap
is locked or not.  This new function is configured via
MICROPY_PY_MICROPYTHON_HEAP_LOCKED and is disabled by default.

This commit also changes the return value of micropython.heap_unlock() so
it returns the current lock-depth as well.
2020-03-11 16:54:16 +11:00
Andrew Leech ed93778e00 py/objstringio: Expose tell() on StringIO and BytesIO objects.
To match file objects.

Fixes issue #5581.
2020-03-11 14:43:03 +11:00
David Lechner 8a4ce6b79a tools/codeformat.py: Eliminate need for sizeof fixup.
This eliminates the need for the sizeof regex fixup by rearranging things a
bit.  All other bitfields already use the parentheses around expressions
with sizeof, so one case is fixed by following this convention.

VM_MAX_STATE_ON_STACK is the only remaining problem and it can be worked
around by changing the order of the operands.
2020-03-11 14:34:40 +11:00