Commit Graph

95 Commits

Author SHA1 Message Date
Jim Mussared 12ca918eb2 tools/mpremote: Add `mpremote mip install` to install packages.
This supports the same package sources as the new `mip` tool.
 - micropython-lib (by name)
 - http(s) & github packages with json description
 - directly downloading a .py/.mpy file

The version is specified with an optional `@version` on the end of the
package name. The target dir, index, and mpy/no-mpy can be set through
command line args.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-01 22:44:24 +10:00
Jim Mussared 924a3e03ec top: Replace upip with mip everywhere.
Updates all README.md and docs, and manifests to `require("mip")`.

Also extend and improve the documentation on freezing and packaging.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-30 17:34:03 +10:00
Jim Mussared d94141e147 py/persistentcode: Introduce .mpy sub-version.
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.

With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version.  A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION.  This allows 3
additional updates to the native ABI per bytecode revision.

The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-09-19 23:19:55 +10:00
Damien George 655c29351a drivers/display: Don't include tests by default.
The tests can be copied to the board if needed.

Also update the docs to reflect this change.

Signed-off-by: Damien George <damien@micropython.org>
2022-09-06 12:09:20 +10:00
Damien George 2e386bcf76 tools/mpremote: Print nicer errors for unsupported 'cp -r' arguments.
Also document support for 'cp :a :b'.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Jim Mussared 59e3348c10 tools/mpremote: Add "edit" command.
This allows a remote file to be edited locally by copying it over, running
the local editor, then copying it back.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:29:13 +10:00
Jim Mussared 263737ecfe tools/pyboard.py: Add "touch" filesystem command.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:24:25 +10:00
Jim Mussared a311e9e3d4 tools/mpremote: Allow + terminator for fs commands.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:22:36 +10:00
Damien George cf7d962cf3 docs/reference/mpyfiles: Update .mpy description to match latest format.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Andrew Leech ea9a904b72 docs/reference/manifest: Add link to details about opt-level for freeze. 2022-05-27 14:45:46 +10:00
Damien George 079f3e5e5b py/parse: Allow all constant objects to be used in "X = const(o)".
Now that constant tuples are supported in the parser, eg (1, True, "str"),
it's a small step to allow anything that is a constant to be used with the
pattern:

    from micropython import const

    X = const(obj)

This commit makes the required changes to allow the following types of
constants:

    from micropython import const

    _INT = const(123)
    _FLOAT = const(1.2)
    _COMPLEX = const(3.4j)
    _STR = const("str")
    _BYTES = const(b"bytes")
    _TUPLE = const((_INT, _STR, _BYTES))
    _TUPLE2 = const((None, False, True, ..., (), _TUPLE))

Prior to this, only integers could be used in const(...).

Signed-off-by: Damien George <damien@micropython.org>
2022-05-18 16:18:35 +10:00
Damien George 2a366e183e docs/reference/mpyfiles: Remove docs about mpy flags and qstr win size.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Damien George 59c5d41611 py/modsys: Rename sys.implementation.mpy to sys.implementation._mpy.
Per CPython docs, non-standard attributes must begin with an underscore.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 17:23:03 +10:00
Damien George 9ab66b50cb docs/reference: Update constrained docs now that tuples can be const.
Signed-off-by: Damien George <damien@micropython.org>
2022-04-15 00:17:08 +10:00
Rob Knegjens d3d862769a tools/mpremote: Add option to mount cmd to allow "unsafe" symlinks.
Specifying the option `--unsafe-links` (or `-l`) to `mpremote mount` will
allow symlinks to be followed in the local directory that point outside of
the base directory path.

For the unsafe case the `path_check()` method of `PyboardCommand` still
checks for a common path but without expanding symlinks.  While this check
is currently redundant, it makes the purpose of the method clearer for
possible future uses or extensions.
2022-04-11 15:15:04 +10:00
Andrew Leech 3b99b4d675 docs/reference/mpremote: Add details about mount usage over soft reboot. 2022-04-04 16:50:07 +10:00
Damien George d6564a3159 tools/mpremote: Add "umount" command.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-25 13:10:45 +11:00
Damien George ad1f523e7e tools/mpremote: Add "resume" and "soft-reset" commands.
This makes the auto soft-reset behaviour of mpremote more logical, and now
configurable via these new commands.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-25 13:09:28 +11:00
Peter Hinch 80b81acea5 docs/reference/isr_rules.rst: Describe uasyncio-IRQ interface. 2022-02-04 12:27:16 +11:00
Luiz Brandao ddda959e57 docs/reference/isr_rules.rst: Fix inconsistent variable name in example.
Fixed to be conistent with the code example above it.
2022-02-04 11:20:15 +11:00
Damien George d9d67adef1 docs: Remove trailing spaces and convert tabs to spaces.
Signed-off-by: Damien George <damien@micropython.org>
2021-12-15 11:49:22 +11:00
Damien George 01f1c3aac2 docs/reference/filesystem.rst: Add detail on how to use littlefs fuse.
Without the --block_count option the fuse will fail.

Signed-off-by: Damien George <damien@micropython.org>
2021-11-19 15:13:05 +11:00
NitiKaur e87b2e8bfa docs/reference/manifest.rst: Add docs for manifest.py files. 2021-10-14 14:03:03 +11:00
NitiKaur 135339ce3a docs/reference/mpremote.rst: Add docs for mpremote. 2021-10-14 13:09:51 +11:00
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
Peter Hinch 3720a570f2 docs/reference: Mention that slicing a memoryview causes allocation. 2021-08-30 23:55:29 +10:00
Jim Mussared d7fbc755dc docs/library/index.rst: Clarify module naming and purpose.
Adds section about extending built-in modules from Python.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-13 23:14:08 +10:00
Jim Mussared c737cde947 docs: Replace ufoo with foo in all docs.
Anywhere a module is mentioned, use its "non-u" name for consistency.

The "import module" vs "import umodule" is something of a FAQ, and this
commit intends to help clear that up.  As a first approximation MicroPython
is Python, and so imports should work the same as Python and use the same
name, to a first approximation.  The u-version of a module is a detail that
can be learned later on, when the user wants to understand more and have
finer control over importing.

Existing Python code should just work, as much as it is possible to do that
within the constraints of embedded systems, and the MicroPython
documentation should match the idiomatic way to write Python code.

With universal weak links for modules (via MICROPY_MODULE_WEAK_LINKS) users
can consistently use "import foo" across all ports (with the exception of
the minimal ports).  And the ability to override/extend via "foo.py"
continues to work well.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-13 22:53:29 +10:00
Brianna Laugher d128999938 tools: Add filesystem action examples to pyboard.py help.
Signed-off-by: Brianna Laugher <brianna.laugher@gmail.com>
2021-02-13 14:37:28 +11:00
Damien George a14ca31e85 docs/reference/repl.rst: Add information about new raw-paste mode.
Signed-off-by: Damien George <damien@micropython.org>
2020-12-01 22:35:13 +11:00
Kevin Thomas 6eebdbc495 docs/reference/glossary.rst: Fix minor grammar error, An -> A. 2020-10-22 15:40:11 +11:00
David Lechner ee7568ca8d docs/reference/packages.rst: Fix typo, remove duplicate "port".
Fixes #6485.
2020-09-30 23:32:24 +10:00
David Lechner 1e6d18c915 docs: Fix Sphinx 3.x warnings, and enable warnings-as-errors on build.
This enables warnings as errors and fixes all current errors, namely:

- reference to terms in the glossary must now be explicit (:term:)
- method overloads must not be declared as a separate method or must
  use :noindex:
- 2 cases where `` should have been used instead of `
2020-06-05 21:42:17 +10:00
Andrew Leech e54626f4c1 docs/reference: Add note about multiple exceptions when heap is locked. 2020-06-02 15:56:34 +10:00
Damien George e0905e85a7 esp8266: Change from FAT to littlefs v2 as default filesystem.
This commit changes the esp8266 boards to use littlefs v2 as the
filesystem, rather than FAT.  Since the esp8266 doesn't expose the
filesystem to the PC over USB there's no strong reason to keep it as FAT.
Littlefs is smaller in code size, is more efficient in use of flash to
store data, is resilient over power failure, and using it saves about 4k of
heap RAM, which can now be used for other things.

This is a backwards incompatible change because all existing esp8266 boards
will need to update their filesystem after installing new firmware (eg
backup old files, install firmware, restore files to new filesystem).

As part of this commit the memory layout of the default board (GENERIC) has
changed.  It now allocates all 1M of memory-mapped flash to the firmware,
so the filesystem area starts at the 2M point.  This is done to allow more
frozen bytecode to be stored in the 1M of memory-mapped flash.  This
requires an esp8266 module with 2M or more of flash to work, so a new board
called GENERIC_1M is added which has the old memory-mapping (but still
changed to use littlefs for the filesystem).

In summary there are now 3 esp8266 board definitions:
- GENERIC_512K: for 512k modules, doesn't have a filesystem.
- GENERIC_1M: for 1M modules, 572k for firmware+frozen code, 396k for
  filesystem (littlefs).
- GENERIC: for 2M (or greater) modules, 968k for firmware+frozen code,
  1M+ for filesystem (littlefs), FAT driver also included in firmware for
  use on, eg, external SD cards.
2020-04-04 16:30:36 +11:00
Lars Kellogg-Stedman 3a0f64fc7a tools/pyboard.py: Add -d as an alias for --device. 2020-03-30 11:37:32 +11:00
Lars Kellogg-Stedman 1cf994c48b tools/pyboard.py: Support setting device/baudrate from shell env vars.
Allow defaults for --device and --baudrate to be set in the environment
using PYBOARD_DEVICE and PYBOARD_BAUDRATE.
2020-03-30 11:37:32 +11:00
Jason Neal aec88ddf03 docs: More consistent capitalization and use of articles in headings.
See issue #3188.
2020-01-06 22:17:29 +11:00
Damien George e58c7ce3d6 docs/reference: Add documentation describing use of .mpy files.
Including information about .mpy versioning and how to debug failed imports
of .mpy files.
2019-12-19 17:06:27 +11:00
Damien George b310930dba docs/library/uos: Add notes and links about littlefs failures. 2019-12-10 16:58:53 +11:00
Damien George 381be9a745 docs/reference/filesystem: Add note and example about using filesystem. 2019-12-09 14:21:22 +11:00
Daniel Mizyrycki 50dc5f10a6 docs/reference/filesystem: Fix typo in block device code example. 2019-12-09 14:13:09 +11:00
Jim Mussared 9a849cc7ca docs: Add littlefs docs and a filesystem tutorial. 2019-12-04 23:18:23 +11:00
Jim Mussared d7dbd267e7 docs/reference: Add docs describing use of pyboard.py. 2019-12-04 23:18:03 +11:00
Damien George 90c524c114 docs: Remove spaces on lines that are empty. 2019-12-04 15:02:54 +11:00
Mike Causer a2c4cb484d docs: Fix spelling in various parts of the docs. 2019-10-16 14:08:28 +11:00
Jim Mussared a93495b66d docs/reference/glossary.rst: Add new terms and reduce complexity of old. 2019-10-15 16:36:02 +11:00
Paul m. p. P afc8596c15 docs/reference/speed_python: Add missing self to var caching example. 2019-08-28 12:55:41 +10:00
Peter Hinch 3a679eaf00 docs/reference/speed_python: Update that read-only buffers are accepted.
As allowed by recent cd35dd9d9a
2019-08-19 16:39:30 +10:00
Andrew Leech 89ff506513 py: Update and rework build system for including external C modules.
How to use this feature is documented in docs/develop/cmodules.rst.
2019-03-08 22:58:42 +11:00