djsundog/circuitpython
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

top: Replace upip with mip everywhere.

Browse Source 
Updates all README.md and docs, and manifests to `require("mip")`.

Also extend and improve the documentation on freezing and packaging.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit is contained in:
Jim Mussared 2022-09-29 17:49:58 +10:00
parent ba3652f15d
commit 924a3e03ec
16 changed files with 404 additions and 878 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/develop/gettingstarted.rst
View File

@ -322,7 +322,8 @@ tests
tools
Contains helper tools including the ``upip`` and the ``pyboard.py`` module.
Contains scripts used by the build and CI process, as well as user tools such
as ``pyboard.py`` and ``mpremote``.
examples

2
docs/develop/optimizations.rst
View File

@ -25,6 +25,8 @@ into the firmware image as part of the main firmware compilation process, which
the bytecode will be executed from ROM. This can lead to a significant memory saving, and
reduce heap fragmentation.
See :ref:`manifest` for more information.
Variables
---------

2
docs/esp8266/tutorial/intro.rst
View File

@ -23,7 +23,7 @@ convertor to make the UART available to your PC.
The minimum requirement for flash size is 1Mbyte. There is also a special
build for boards with 512KB, but it is highly limited comparing to the
normal build: there is no support for filesystem, and thus features which
depend on it won't work (WebREPL, upip, etc.). As such, 512KB build will
depend on it won't work (WebREPL, mip, etc.). As such, 512KB build will
be more interesting for users who build from source and fine-tune parameters
for their particular application.

2
docs/pyboard/tutorial/lcd160cr_skin.rst
View File

@ -42,7 +42,7 @@ There is a test program which you can use to test the features of the display,
and which also serves as a basis to start creating your own code that uses the
LCD. This test program is available on GitHub
`here <https://github.com/micropython/micropython/blob/master/drivers/display/lcd160cr_test.py>`__.
Copy it to the board over USB mass storage, or by using `mpremote`.
Copy it to the board over USB mass storage, or by using :ref:`mpremote`.
To run the test from the MicroPython prompt do::

30
docs/reference/glossary.rst
View File

@ -52,7 +52,7 @@ Glossary
cross-compiler
Also known as ``mpy-cross``. This tool runs on your PC and converts a
:term:`.py file` containing MicroPython code into a :term:`.mpy file`
containing MicroPython bytecode. This means it loads faster (the board
containing MicroPython :term:`bytecode`. This means it loads faster (the board
doesn't have to compile the code), and uses less space on flash (the
bytecode is more space efficient).
@ -128,7 +128,7 @@ Glossary
Unlike the :term:`CPython` stdlib, micropython-lib modules are
intended to be installed individually - either using manual copying or
using :term:`upip`.
using :term:`mip`.
MicroPython port
MicroPython supports different :term:`boards <board>`, RTOSes, and
@ -151,16 +151,26 @@ Glossary
machine-independent features. It can also function in a similar way to
:term:`CPython`'s ``python`` executable.
mip
A package installer for MicroPython (mip - "mip installs packages"). It
installs MicroPython packages either from :term:`micropython-lib`,
GitHub, or arbitrary URLs. mip can be used on-device on
network-capable boards, and internally by tools such
as :term:`mpremote`.
mpremote
A tool for interacting with a MicroPython device. See :ref:`mpremote`.
.mpy file
The output of the :term:`cross-compiler`. A compiled form of a
:term:`.py file` that contains MicroPython bytecode instead of Python
source code.
:term:`.py file` that contains MicroPython :term:`bytecode` instead of
Python source code.
native
Usually refers to "native code", i.e. machine code for the target
microcontroller (such as ARM Thumb, Xtensa, x86/x64). The ``@native``
decorator can be applied to a MicroPython function to generate native
code instead of bytecode for that function, which will likely be
code instead of :term:`bytecode` for that function, which will likely be
faster but use more RAM.
port
@ -193,8 +203,10 @@ Glossary
as a serial port over USB.
upip
(Literally, "micro pip"). A package manager for MicroPython, inspired
A now-obsolete package manager for MicroPython, inspired
by :term:`CPython`'s pip, but much smaller and with reduced
functionality.
upip runs both on the :term:`Unix port <MicroPython Unix port>` and on
:term:`baremetal` ports which offer filesystem and networking support.
functionality. See its replacement, :term:`mip`.
webrepl
A way of connecting to the REPL (and transferring files) on a device
over the internet from a browser. See https://micropython.org/webrepl

271
docs/reference/manifest.rst
View File

@ -1,35 +1,177 @@
.. _manifest:
MicroPython manifest files
==========================
When building firmware for a device the following components are included in
the compilation process:
Summary
-------
- the core MicroPython virtual machine and runtime
- port-specific system code and drivers to interface with the
microcontroller/device that the firmware is targeting
- standard built-in modules, like ``sys``
- extended built-in modules, like ``json`` and ``machine``
- extra modules written in C/C++
- extra modules written in Python
MicroPython has a feature that allows Python code to be "frozen" into the
firmware, as an alternative to loading code from the filesystem.
All the modules included in the firmware are available via ``import`` from
Python code. The extra modules written in Python that are included in a build
(the last point above) are called *frozen modules*, and are specified by a
``manifest.py`` file. Changing this manifest requires rebuilding the firmware.
This has the following benefits:
It's also possible to add additional modules to the filesystem of the device
once it is up and running. Adding and removing modules to/from the filesystem
does not require rebuilding the firmware so is a simpler process than rebuilding
firmware. The benefit of using a manifest is that frozen modules are more
efficient: they are faster to import and take up less RAM once imported.
- the code is pre-compiled to bytecode, avoiding the need for the Python
source to be compiled at load-time.
- the bytecode can be executed directly from ROM (i.e. flash memory) rather than
being copied into RAM. Similarly any constant objects (strings, tuples, etc)
are loaded from ROM also. This can lead to significantly more memory being
available for your application.
- on devices that do not have a filesystem, this is the only way to
load Python code.
MicroPython manifest files are Python files and can contain arbitrary Python
code. There are also a set of commands (predefined functions) which are used
to specify the Python source files to include. These commands are described
below.
During development, freezing is generally not recommended as it will
significantly slow down your development cycle, as each update will require
re-flashing the entire firmware. However, it can still be useful to
selectively freeze some rarely-changing dependencies (such as third-party
libraries).
Freezing source code
--------------------
The way to list the Python files to be be frozen into the firmware is via
a "manifest", which is a Python file that will be interpreted by the build
process. Typically you would write a manifest file as part of a board
definition, but you can also write a stand-alone manifest file and use it with
an existing board definition.
Manifest files can define dependencies on libraries from :term:`micropython-lib`
as well as Python files on the filesystem, and also on other manifest files.
Writing manifest files
----------------------
A manifest file is a Python file containing a series of function calls. See the
available functions defined below.
Any paths used in manifest files can include the following variables. These all
resolve to absolute paths.
- ``$(MPY_DIR)`` -- path to the micropython repo.
- ``$(MPY_LIB_DIR)`` -- path to the micropython-lib submodule. Prefer to use
``require()``.
- ``$(PORT_DIR)`` -- path to the current port (e.g. ``ports/stm32``)
- ``$(BOARD_DIR)`` -- path to the current board
(e.g. ``ports/stm32/boards/PYBV11``)
Custom manifest files should not live in the main MicroPython repository. You
should keep them in version control with the rest of your project.
Typically a manifest used for compiling firmware will need to include the port
manifest, which might include frozen modules that are required for the board to
function. If you just want to add additional modules to an existing board, then
include the board manifest (which will in turn include the port manifest).
Building with a custom manifest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Your manifest can be specified on the ``make`` command line with:
.. code-block:: bash
$ make BOARD=MYBOARD FROZEN_MANIFEST=/path/to/my/project/manifest.py
This applies to all ports, including CMake-based ones (e.g. esp32, rp2), as the
Makefile wrapper that will pass this into the CMake build.
Adding a manifest to a board definition
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you have a custom board definition, you can make it include your custom
manifest automatically. On make-based ports (most ports), in your
``mpconfigboard.mk`` set the ``FROZEN_MANIFEST`` variable.
.. code-block:: makefile
FROZEN_MANIFEST ?= $(BOARD_DIR)/manifest.py
On CMake-based ports (e.g. esp32, rp2), instead use ``mpconfigboard.cmake``
.. code-block:: cmake
set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py)
High-level functions
~~~~~~~~~~~~~~~~~~~~
Note: The ``opt`` keyword argument can be set on the various functions, this controls
the optimisation level used by the cross-compiler.
See :func:`micropython.opt_level`.
.. function:: package(package_path, files=None, base_path=".", opt=None)
This is equivalent to copying the "package_path" directory to the device
(except as frozen code).
In the simplest case, to freeze a package "foo" in the current directory:
.. code-block:: python3
package("foo")
will recursively include all .py files in foo, and will be frozen as
``foo/**/*.py``.
If the package isn't in the same directory as the manifest file, use ``base_path``:
.. code-block:: python3
package("foo", base_path="path/to/libraries")
You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``.
To restrict to certain files in the package use ``files`` (note: paths
should be relative to the package): ``package("foo", files=["bar/baz.py"])``.
.. function:: module(module_path, base_path=".", opt=None)
Include a single Python file as a module.
If the file is in the current directory:
.. code-block:: python3
module("foo.py")
Otherwise use base_path to locate the file:
.. code-block:: python3
module("foo.py", base_path="src/drivers")
You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``.
.. function:: require(name, unix_ffi=False)
Require a package by name (and its dependencies) from :term:`micropython-lib`.
Optionally specify unix_ffi=True to use a module from the unix-ffi directory.
.. function:: include(manifest_path)
Include another manifest.
Typically a manifest used for compiling firmware will need to include the
port manifest, which might include frozen modules that are required for
the board to function.
The *manifest* argument can be a string (filename) or an iterable of
strings.
Relative paths are resolved with respect to the current manifest file.
If the path is to a directory, then it implicitly includes the
manifest.py file inside that directory.
You can use the variables above, such as ``$(PORT_DIR)`` in ``manifest_path``.
.. function:: metadata(description=None, version=None, license=None, author=None)
Define metadata for this manifest file. This is useful for manifests for
micropython-lib packages.
Low-level functions
~~~~~~~~~~~~~~~~~~~
These functions are documented for completeness, but with the exception of
``freeze_as_str`` all functionality can be accessed via the high-level functions.
.. function:: freeze(path, script=None, opt=0)
@ -42,9 +184,7 @@ Freezing source code
module will start after *path*, i.e. *path* is excluded from the module
name.
If *path* is relative, it is resolved to the current ``manifest.py``. Use
``$(MPY_DIR)``, ``$(MPY_LIB_DIR)``, ``$(PORT_DIR)``, ``$(BOARD_DIR)`` if you
need to access specific paths.
If *path* is relative, it is resolved to the current ``manifest.py``.
If *script* is None, all files in *path* will be frozen.
@ -75,71 +215,48 @@ Freezing source code
Freeze the input, which must be ``.mpy`` files that are frozen directly.
See ``freeze()`` for further details on the arguments.
Including other manifest files
------------------------------
.. function:: include(manifest, **kwargs)
Include another manifest.
The *manifest* argument can be a string (filename) or an iterable of
strings.
Relative paths are resolved with respect to the current manifest file.
Optional *kwargs* can be provided which will be available to the included
script via the *options* variable.
For example:
.. code-block:: python3
include("path.py", extra_features=True)
then in path.py:
.. code-block:: python3
options.defaults(standard_features=True)
# freeze minimal modules.
if options.standard_features:
# freeze standard modules.
if options.extra_features:
# freeze extra modules.
Examples
--------
To freeze a single file which is available as ``import mydriver``, use:
To freeze a single file from the current directory which will be available as
``import mydriver``, use:
.. code-block:: python3
freeze(".", "mydriver.py")
module("mydriver.py")
To freeze a set of files which are available as ``import test1`` and
``import test2``, and which are compiled with optimisation level 3, use:
To freeze a directory of files in a subdirectory "mydriver" of the current
directory which will be available as ``import mydriver``, use:
.. code-block:: python3
freeze("/path/to/tests", ("test1.py", "test2.py"), opt=3)
package("mydriver")
To freeze a module which can be imported as ``import mymodule``, use:
To freeze the "hmac" library from :term:`micropython-lib`, use:
.. code-block:: python3
freeze(
"../relative/path",
(
"mymodule/__init__.py",
"mymodule/core.py",
"mymodule/extra.py",
),
)
require("hmac")
To include a manifest from the MicroPython repository, use:
A more complete example of a custom ``manifest.py`` file for the ``PYBD_SF2``
board is:
.. code-block:: python3
include("$(MPY_DIR)/extmod/uasyncio/manifest.py")
# Include the board's default manifest.
include("$(BOARD_DIR)/manifest.py")
# Add a custom driver
module("mydriver.py")
# Add aiorepl from micropython-lib
require("aiorepl")
Then the board can be compiled with
.. code-block:: bash
$ cd ports/stm32
$ make BOARD=PYBD_SF2 FROZEN_MANIFEST=~/src/myproject/manifest.py
Note that most boards do not have their own ``manifest.py``, rather they use the
port one directly, in which case your manifest should just
``include("$(PORT_DIR)/boards/manifest.py")`` instead.

463
docs/reference/packages.rst
View File

@ -1,314 +1,153 @@
.. _packages:
Distribution packages, package management, and deploying applications
=====================================================================
Just as the "big" Python, MicroPython supports creation of "third party"
packages, distributing them, and easily installing them in each user's
environment. This chapter discusses how these actions are achieved.
Some familiarity with Python packaging is recommended.
Overview
--------
Steps below represent a high-level workflow when creating and consuming
packages:
1. Python modules and packages are turned into distribution package
archives, and published at the Python Package Index (PyPI).
2. :term:`upip` package manager can be used to install a distribution package
on a :term:`MicroPython port` with networking capabilities (for example,
on the Unix port).
3. For ports without networking capabilities, an "installation image"
can be prepared on the Unix port, and transferred to a device by
suitable means.
4. For low-memory ports, the installation image can be frozen as the
bytecode into MicroPython executable, thus minimizing the memory
storage overheads.
The sections below describe this process in details.
Distribution packages
---------------------
Python modules and packages can be packaged into archives suitable for
transfer between systems, storing at the well-known location (PyPI),
and downloading on demand for deployment. These archives are known as
*distribution packages* (to differentiate them from Python packages
(means to organize Python source code)).
The MicroPython distribution package format is a well-known tar.gz
format, with some adaptations however. The Gzip compressor, used as
an external wrapper for TAR archives, by default uses 32KB dictionary
size, which means that to uncompress a compressed stream, 32KB of
contiguous memory needs to be allocated. This requirement may be not
satisfiable on low-memory devices, which may have total memory available
less than that amount, and even if not, a contiguous block like that
may be hard to allocate due to memory fragmentation. To accommodate
these constraints, MicroPython distribution packages use Gzip compression
with the dictionary size of 4K, which should be a suitable compromise
with still achieving some compression while being able to uncompressed
even by the smallest devices.
Besides the small compression dictionary size, MicroPython distribution
packages also have other optimizations, like removing any files from
the archive which aren't used by the installation process. In particular,
:term:`upip` package manager doesn't execute ``setup.py`` during installation
(see below), and thus that file is not included in the archive.
At the same time, these optimizations make MicroPython distribution
packages not compatible with :term:`CPython`'s package manager, ``pip``.
This isn't considered a big problem, because:
1. Packages can be installed with :term:`upip`, and then can be used with
CPython (if they are compatible with it).
2. In the other direction, majority of CPython packages would be
incompatible with MicroPython by various reasons, first of all,
the reliance on features not implemented by MicroPython.
Summing up, the MicroPython distribution package archives are highly
optimized for MicroPython's target environments, which are highly
resource constrained devices.
``upip`` package manager
------------------------
MicroPython distribution packages are intended to be installed using
the :term:`upip` package manager. :term:`upip` is a Python application which is
usually distributed (as frozen bytecode) with network-enabled
:term:`MicroPython ports <MicroPython port>`. At the very least,
:term:`upip` is available in the :term:`MicroPython Unix port`.
On any :term:`MicroPython port` providing :term:`upip`, it can be accessed as
following::
import upip
upip.help()
upip.install(package_or_package_list, [path])
Where *package_or_package_list* is the name of a distribution
package to install, or a list of such names to install multiple
packages. Optional *path* parameter specifies filesystem
location to install under and defaults to the standard library
location (see below).
An example of installing a specific package and then using it::
>>> import upip
>>> upip.install("micropython-pystone_lowmem")
[...]
>>> import pystone_lowmem
>>> pystone_lowmem.main()
Note that the name of Python package and the name of distribution
package for it in general don't have to match, and oftentimes they
don't. This is because PyPI provides a central package repository
for all different Python implementations and versions, and thus
distribution package names may need to be namespaced for a particular
implementation. For example, all packages from `micropython-lib`
follow this naming convention: for a Python module or package named
``foo``, the distribution package name is ``micropython-foo``.
For the ports which run MicroPython executable from the OS command
prompts (like the Unix port), `upip` can be (and indeed, usually is)
run from the command line instead of MicroPython's own REPL. The
commands which corresponds to the example above are::
micropython -m upip -h
micropython -m upip install [-p <path>] <packages>...
micropython -m upip install micropython-pystone_lowmem
[TODO: Describe installation path.]
Cross-installing packages
-------------------------
For :term:`MicroPython ports <MicroPython port>` without native networking
capabilities, the recommend process is "cross-installing" them into a
"directory image" using the :term:`MicroPython Unix port`, and then
transferring this image to a device by suitable means.
Installing to a directory image involves using ``-p`` switch to :term:`upip`::
micropython -m upip install -p install_dir micropython-pystone_lowmem
After this command, the package content (and contents of every dependency
packages) will be available in the ``install_dir/`` subdirectory. You
would need to transfer contents of this directory (without the
``install_dir/`` prefix) to the device, at the suitable location, where
it can be found by the Python ``import`` statement (see discussion of
the :term:`upip` installation path above).
Cross-installing packages with freezing
---------------------------------------
For the low-memory :term:`MicroPython ports <MicroPython port>`, the process
described in the previous section does not provide the most efficient
resource usage,because the packages are installed in the source form,
so need to be compiled to the bytecome on each import. This compilation
requires RAM, and the resulting bytecode is also stored in RAM, reducing
its amount available for storing application data. Moreover, the process
above requires presence of the filesystem on a device, and the most
resource-constrained devices may not even have it.
The bytecode freezing is a process which resolves all the issues
mentioned above:
* The source code is pre-compiled into bytecode and store as such.
* The bytecode is stored in ROM, not RAM.
* Filesystem is not required for frozen packages.
Using frozen bytecode requires building the executable (firmware)
for a given :term:`MicroPython port` from the C source code. Consequently,
the process is:
1. Follow the instructions for a particular port on setting up a
toolchain and building the port. For example, for ESP8266 port,
study instructions in ``ports/esp8266/README.md`` and follow them.
Make sure you can build the port and deploy the resulting
executable/firmware successfully before proceeding to the next steps.
2. Build :term:`MicroPython Unix port` and make sure it is in your PATH and
you can execute ``micropython``.
3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266).
4. Run ``make clean-frozen``. This step cleans up any previous
modules which were installed for freezing (consequently, you need
to skip this step to add additional modules, instead of starting
from scratch).
5. Run ``micropython -m upip install -p modules <packages>...`` to
install packages you want to freeze.
6. Run ``make clean``.
7. Run ``make``.
After this, you should have the executable/firmware with modules as
the bytecode inside, which you can deploy the usual way.
Few notes:
1. Step 5 in the sequence above assumes that the distribution package
is available from PyPI. If that is not the case, you would need
to copy Python source files manually to ``modules/`` subdirectory
of the port directory. (Note that upip does not support
installing from e.g. version control repositories).
2. The firmware for baremetal devices usually has size restrictions,
so adding too many frozen modules may overflow it. Usually, you
would get a linking error if this happens. However, in some cases,
an image may be produced, which is not runnable on a device. Such
cases are in general bugs, and should be reported and further
investigated. If you face such a situation, as an initial step,
you may want to decrease the amount of frozen modules included.
Creating distribution packages
------------------------------
Distribution packages for MicroPython are created in the same manner
as for CPython or any other Python implementation, see references at
the end of chapter. Setuptools (instead of distutils) should be used,
because distutils do not support dependencies and other features. "Source
distribution" (``sdist``) format is used for packaging. The post-processing
discussed above, (and pre-processing discussed in the following section)
is achieved by using custom ``sdist`` command for setuptools. Thus, packaging
steps remain the same as for the standard setuptools, the user just
needs to override ``sdist`` command implementation by passing the
appropriate argument to ``setup()`` call::
from setuptools import setup
import sdist_upip
setup(
...,
cmdclass={'sdist': sdist_upip.sdist}
)
The sdist_upip.py module as referenced above can be found in
`micropython-lib`:
https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py
Application resources
---------------------
A complete application, besides the source code, oftentimes also consists
of data files, e.g. web page templates, game images, etc. It's clear how
to deal with those when application is installed manually - you just put
those data files in the filesystem at some location and use the normal
file access functions.
The situation is different when deploying applications from packages - this
is more advanced, streamlined and flexible way, but also requires more
advanced approach to accessing data files. This approach is treating
the data files as "resources", and abstracting away access to them.
Python supports resource access using its "setuptools" library, using
``pkg_resources`` module. MicroPython, following its usual approach,
implements subset of the functionality of that module, specifically
``pkg_resources.resource_stream(package, resource)`` function.
The idea is that an application calls this function, passing a
resource identifier, which is a relative path to data file within
the specified package (usually top-level application package). It
returns a stream object which can be used to access resource contents.
Thus, the ``resource_stream()`` emulates interface of the standard
`open()` function.
Implementation-wise, ``resource_stream()`` uses file operations
underlyingly, if distribution package is install in the filesystem.
However, it also supports functioning without the underlying filesystem,
e.g. if the package is frozen as the bytecode. This however requires
an extra intermediate step when packaging application - creation of
"Python resource module".
The idea of this module is to convert binary data to a Python bytes
object, and put it into the dictionary, indexed by the resource name.
This conversion is done automatically using overridden ``sdist`` command
described in the previous section.
Let's trace the complete process using the following example. Suppose
your application has the following structure::
my_app/
__main__.py
utils.py
data/
page.html
image.png
``__main__.py`` and ``utils.py`` should access resources using the
following calls::
import pkg_resources
pkg_resources.resource_stream(__name__, "data/page.html")
pkg_resources.resource_stream(__name__, "data/image.png")
You can develop and debug using the :term:`MicroPython Unix port` as usual.
When time comes to make a distribution package out of it, just use
overridden "sdist" command from sdist_upip.py module as described in
the previous section.
This will create a Python resource module named ``R.py``, based on the
files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py``
file will be considered a resource and added to ``R.py``) - before
proceeding with the normal packaging steps.
Prepared like this, your application will work both when deployed to
filesystem and as frozen bytecode.
If you would like to debug ``R.py`` creation, you can run::
python3 setup.py sdist --manifest-only
Alternatively, you can use tools/mpy_bin2res.py script from the
MicroPython distribution, in which can you will need to pass paths
to all resource files::
mpy_bin2res.py data/page.html data/image.png
References
----------
* Python Packaging User Guide: https://packaging.python.org/
* Setuptools documentation: https://setuptools.readthedocs.io/
* Distutils documentation: https://docs.python.org/3/library/distutils.html
Package management
==================
Installing packages with ``mip``
--------------------------------
Network-capable boards include the ``mip`` module, which can install packages
from :term:`micropython-lib` and from third-party sites (including GitHub).
``mip`` ("mip installs packages") is similar in concept to Python's ``pip`` tool,
however it does not use the PyPI index, rather it uses :term:`micropython-lib`
as its index by default. ``mip`` will automatically fetch compiled
:term:`.mpy file` when downloading from micropython-lib.
The most common way to use ``mip`` is from the REPL::
>>> import mip
>>> mip.install("pkgname") # Installs the latest version of "pkgname" (and dependencies)
>>> mip.install("pkgname", version="x.y") # Installs version x.y of "pkgname"
>>> mip.install("pkgname", mpy=False) # Installs the source version (i.e. .py rather than .mpy files)
``mip`` will detect an appropriate location on the filesystem by searching
``sys.path`` for the first entry ending in ``/lib``. You can override the
destination using ``target``, but note that this path must be in ``sys.path`` to be
able to subsequently import it.::
>>> mip.install("pkgname", target="third-party")
>>> sys.path.append("third-party")
As well as downloading packages from the micropython-lib index, ``mip`` can also
install third-party libraries. The simplest way is to download a file directly::
>>> mip.install("http://example.com/x/y/foo.py")
>>> mip.install("http://example.com/x/y/foo.mpy")
When installing a file directly, the ``target`` argument is still supported to set
the destination path, but ``mpy`` and ``version`` are ignored.
The URL can also start with ``github:`` as a simple way of pointing to content
hosted on GitHub::
>>> mip.install("github:org/repo/path/foo.py") # Uses default branch
>>> mip.install("github:org/repo/path/foo.py", version="branch-or-tag") # Optionally specify the branch or tag
More sophisticated packages (i.e. with more than one file, or with dependencies)
can be downloaded by specifying the path to their ``package.json``.
>>> mip.install("http://example.com/x/package.json")
>>> mip.install("github:org/user/path/package.json")
If no json file is specified, then "package.json" is implicitly added::
>>> mip.install("http://example.com/x/")
>>> mip.install("github:org/repo")
>>> mip.install("github:org/repo", version="branch-or-tag")
Using ``mip`` on the Unix port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On the Unix port, ``mip`` can be used at the REPL as above, and also by using ``-m``::
$ ./micropython -m mip install pkgname-or-url
$ ./micropython -m mip install pkgname-or-url@version
The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
$ ./micropython -m mip install --target=third-party pkgname
$ ./micropython -m mip install --no-mpy pkgname
$ ./micropython -m mip install --index https://host/pi pkgname
Installing packages with ``mpremote``
-------------------------------------
The :term:`mpremote` tool also includes the same functionality as ``mip`` and
can be used from a host PC to install packages to a locally connected device
(e.g. via USB or UART)::
$ mpremote install pkgname
$ mpremote install pkgname@x.y
$ mpremote install http://example.com/x/y/foo.py
$ mpremote install github:org/repo
$ mpremote install github:org/repo@branch-or-tag
The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set::
$ mpremote install --target=/flash/third-party pkgname
$ mpremote install --no-mpy pkgname
$ mpremote install --index https://host/pi pkgname
Installing packages manually
----------------------------
Packages can also be installed (in either .py or .mpy form) by manually copying
the files to the device. Depending on the board this might be via USB Mass Storage,
the :term:`mpremote` tool (e.g. ``mpremote fs cp path/to/package.py :package.py``),
:term:`webrepl`, etc.
Writing & publishing packages
-----------------------------
Publishing to :term:`micropython-lib` is the easiest way to make your package
broadly accessible to MicroPython users, and automatically available via
``mip`` and ``mpremote`` and compiled to bytecode. See
https://github.com/micropython/micropython-lib for more information.
To write a "self-hosted" package that can be downloaded by ``mip`` or
``mpremote``, you need a static webserver (or GitHub) to host either a
single .py file, or a package.json file alongside your .py files.
A typical package.json for an example ``mlx90640`` library looks like::
{
"urls": [
["mlx90640/__init__.py", "github:org/micropython-mlx90640/mlx90640/__init__.py"],
["mlx90640/utils.py", "github:org/micropython-mlx90640/mlx90640/utils.py"]
],
"deps": [
["collections-defaultdict", "latest"],
["os-path", "latest"]
],
"version": "0.2"
}
This includes two files, hosted at a GitHub repo named
``org/micropython-mlx90640``, which install into the ``mlx90640`` directory on
the device. It depends on ``collections-defaultdict`` and ``os-path`` which will
be installed automatically.
Freezing packages
-----------------
When a Python module or package is imported from the device filesystem, it is
compiled into :term:`bytecode` in RAM, ready to be executed by the VM. For
a :term:`.mpy file`, this conversion has been done already, but the bytecode
still ends up in RAM.
For low-memory devices, or for large applications, it can be advantageous to
instead run the bytecode from ROM (i.e. flash memory). This can be done
by "freezing" the bytecode into the MicroPython firmware, which is then flashed
to the device. The runtime performance is the same (although importing is
faster), but it can free up significant amounts of RAM for your program to
use.
The downside of this approach is that it's much slower to develop, because you
have to flash the firmware each time, but it can be still useful to freeze
dependencies that don't change often.
Freezing is done by writing a manifest file and using it in the build, often as
part of a custom board definition. See the :ref:`manifest` guide for more
information.

2
examples/hwapi/README.md
View File

@ -116,7 +116,7 @@ For example, one may invent a "configuration manager" helper module which will
try to detect current board (among well-known ones), and load appropriate
`hwconfig_*.py` - this assumes that a user would lazily deploy them all
(or that application will be automatically installed, e.g. using MicroPython's
`upip` package manager). The key point in this case remains the same as
`mip` package manager). The key point in this case remains the same as
elaborated above - always assume there can, and will be a custom configuration,
and it should be well supported. So, any automatic detection should be
overridable by a user, and instructions how to do so are among the most

3
ports/esp32/boards/manifest.py
View File

@ -1,11 +1,10 @@
freeze("$(PORT_DIR)/modules")
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
include("$(MPY_DIR)/extmod/uasyncio")
# Require some micropython-lib modules.
require("dht")
require("ds18x20")
require("mip")
require("neopixel")
require("ntptime")
require("onewire")

18
ports/esp8266/README.md
View File

@ -200,20 +200,22 @@ Python prompt over WiFi, connecting through a browser.
- GitHub repository https://github.com/micropython/webrepl.
Please follow the instructions there.
__upip__
__mip__
The ESP8266 port comes with builtin `upip` package manager, which can
be used to install additional modules (see the main README for more
information):
The ESP8266 port comes with the built-in `mip` package manager, which can
be used to install additional modules:
```
>>> import upip
>>> upip.install("micropython-pystone_lowmem")
>>> import mip
>>> mip.install("hmac")
[...]
>>> import pystone_lowmem
>>> pystone_lowmem.main()
>>> import hmac
>>> hmac.new(b"1234567890", msg="hello world").hexdigest()
```
See [Package management](https://docs.micropython.org/en/latest/reference/packages.html) for more
information about `mip`.
Downloading and installing packages may requite a lot of free memory,
if you get an error, retry immediately after the hard reset.

7
ports/esp8266/boards/manifest.py
View File

@ -1,9 +1,8 @@
freeze("$(PORT_DIR)/modules")
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
require("ntptime")
require("dht")
require("onewire")
require("ds18x20")
require("mip")
require("neopixel")
require("ntptime")
require("onewire")
require("webrepl")

4
ports/rp2/boards/PICO_W/manifest.py
View File

@ -1,7 +1,5 @@
include("../manifest.py")
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
require("mip")
require("ntptime")
require("urequests")

26
ports/unix/README.md
View File

@ -24,21 +24,26 @@ Use `CTRL-D` (i.e. EOF) to exit the shell.
Learn about command-line options (in particular, how to increase heap size
which may be needed for larger applications):
$ ./micropython -h
$ ./build-standard/micropython -h
To run the complete testsuite, use:
$ make test
The Unix port comes with a builtin package manager called upip, e.g.:
The Unix port comes with a built-in package manager called `mip`, e.g.:
$ ./micropython -m upip install micropython-pystone
$ ./micropython -m pystone
$ ./build-standard/micropython -m mip install hmac
Browse available modules on
[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython).
Standard library modules come from the
[micropython-lib](https://github.com/micropython/micropython-lib) project.
or
$ ./build-standard/micropython
>>> import mip
>>> mip.install("hmac")
Browse available modules at [micropython-lib]
(https://github.com/micropython/micropython-lib). See
[Package management](https://docs.micropython.org/en/latest/reference/packages.html)
for more information about `mip`.
External dependencies
---------------------
@ -65,6 +70,5 @@ or not). If you intend to build MicroPython with additional options
(like cross-compiling), the same set of options should be passed to `make
deplibs`. To actually enable/disable use of dependencies, edit the
`ports/unix/mpconfigport.mk` file, which has inline descriptions of the
options. For example, to build SSL module (required for the `upip` tool
described above, and so enabled by default), `MICROPY_PY_USSL` should be set
to 1.
options. For example, to build the SSL module, `MICROPY_PY_USSL` should be
set to 1.

3
ports/unix/variants/manifest.py
View File

@ -1,2 +1 @@
module("upip.py", base_path="$(MPY_DIR)/tools", opt=3)
module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3)
require("mip")

351
tools/upip.py
View File

@ -1,351 +0,0 @@
#
# upip - Package manager for MicroPython
#
# Copyright (c) 2015-2018 Paul Sokolovsky
#
# Licensed under the MIT license.
#
import sys
import gc
import uos as os
import uerrno as errno
import ujson as json
import uzlib
import upip_utarfile as tarfile
gc.collect()
debug = False
index_urls = ["https://micropython.org/pi", "https://pypi.org/pypi"]
install_path = None
cleanup_files = []
gzdict_sz = 16 + 15
file_buf = bytearray(512)
class NotFoundError(Exception):
pass
def op_split(path):
if path == "":
return ("", "")
r = path.rsplit("/", 1)
if len(r) == 1:
return ("", path)
head = r[0]
if not head:
head = "/"
return (head, r[1])
# Expects *file* name
def _makedirs(name, mode=0o777):
ret = False
s = ""
comps = name.rstrip("/").split("/")[:-1]
if comps[0] == "":
s = "/"
for c in comps:
if s and s[-1] != "/":
s += "/"
s += c
try:
os.mkdir(s)
ret = True
except OSError as e:
if e.errno != errno.EEXIST and e.errno != errno.EISDIR:
raise e
ret = False
return ret
def save_file(fname, subf):
global file_buf
with open(fname, "wb") as outf:
while True:
sz = subf.readinto(file_buf)
if not sz:
break
outf.write(file_buf, sz)
def install_tar(f, prefix):
meta = {}
for info in f:
# print(info)
fname = info.name
try:
fname = fname[fname.index("/") + 1 :]
except ValueError:
fname = ""
save = True
for p in ("setup.", "PKG-INFO", "README"):
# print(fname, p)
if fname.startswith(p) or ".egg-info" in fname:
if fname.endswith("/requires.txt"):
meta["deps"] = f.extractfile(info).read()
save = False
if debug:
print("Skipping", fname)
break
if save:
outfname = prefix + fname
if info.type != tarfile.DIRTYPE:
if debug:
print("Extracting " + outfname)
_makedirs(outfname)
subf = f.extractfile(info)
save_file(outfname, subf)
return meta
def expandhome(s):
if "~/" in s:
h = os.getenv("HOME")
s = s.replace("~/", h + "/")
return s
import ussl
import usocket
warn_ussl = True
def url_open(url):
global warn_ussl
if debug:
print(url)
proto, _, host, urlpath = url.split("/", 3)
try:
port = 443
if ":" in host:
host, port = host.split(":")
port = int(port)
ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM)
except OSError as e:
fatal("Unable to resolve %s (no Internet?)" % host, e)
# print("Address infos:", ai)
ai = ai[0]
s = usocket.socket(ai[0], ai[1], ai[2])
try:
# print("Connect address:", addr)
s.connect(ai[-1])
if proto == "https:":
s = ussl.wrap_socket(s, server_hostname=host)
if warn_ussl:
print("Warning: %s SSL certificate is not validated" % host)
warn_ussl = False
# MicroPython rawsocket module supports file interface directly
s.write("GET /%s HTTP/1.0\r\nHost: %s:%s\r\n\r\n" % (urlpath, host, port))
l = s.readline()
protover, status, msg = l.split(None, 2)
if status != b"200":
if status == b"404" or status == b"301":
raise NotFoundError("Package not found")
raise ValueError(status)
while 1:
l = s.readline()
if not l:
raise ValueError("Unexpected EOF in HTTP headers")
if l == b"\r\n":
break
except Exception as e:
s.close()
raise e
return s
def get_pkg_metadata(name):
for url in index_urls:
try:
f = url_open("%s/%s/json" % (url, name))
except NotFoundError:
continue
try:
return json.load(f)
finally:
f.close()
raise NotFoundError("Package not found")
def fatal(msg, exc=None):
print("Error:", msg)
if exc and debug:
raise exc
sys.exit(1)
def install_pkg(pkg_spec, install_path):
package = pkg_spec.split("==")
data = get_pkg_metadata(package[0])
if len(package) == 1:
latest_ver = data["info"]["version"]
else:
latest_ver = package[1]
packages = data["releases"][latest_ver]
del data
gc.collect()
assert len(packages) == 1
package_url = packages[0]["url"]
print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url))
f1 = url_open(package_url)
try:
f2 = uzlib.DecompIO(f1, gzdict_sz)
f3 = tarfile.TarFile(fileobj=f2)
meta = install_tar(f3, install_path)
finally:
f1.close()
del f3
del f2
gc.collect()
return meta
def install(to_install, install_path=None):
# Calculate gzip dictionary size to use
global gzdict_sz
sz = gc.mem_free() + gc.mem_alloc()
if sz <= 65536:
gzdict_sz = 16 + 12
if install_path is None:
install_path = get_install_path()
if install_path[-1] != "/":
install_path += "/"
if not isinstance(to_install, list):
to_install = [to_install]
print("Installing to: " + install_path)
# sets would be perfect here, but don't depend on them
installed = []
try:
while to_install:
if debug:
print("Queue:", to_install)
pkg_spec = to_install.pop(0)
if pkg_spec in installed:
continue
meta = install_pkg(pkg_spec, install_path)
installed.append(pkg_spec)
if debug:
print(meta)
deps = meta.get("deps", "").rstrip()
if deps:
deps = deps.decode("utf-8").split("\n")
to_install.extend(deps)
except Exception as e:
print(
"Error installing '{}': {}, packages may be partially installed".format(pkg_spec, e),
file=sys.stderr,
)
def get_install_path():
global install_path
if install_path is None:
# sys.path[0] is current module's path
install_path = sys.path[1]
if install_path == ".frozen":
install_path = sys.path[2]
install_path = expandhome(install_path)
return install_path
def cleanup():
for fname in cleanup_files:
try:
os.unlink(fname)
except OSError:
print("Warning: Cannot delete " + fname)
def help():
print(
"""\
upip - Simple PyPI package manager for MicroPython
Usage: micropython -m upip install [-p <path>] <package>... | -r <requirements.txt>
import upip; upip.install(package_or_list, [<path>])
If <path> isn't given, packages will be installed to sys.path[1], or
sys.path[2] if the former is .frozen (path can be set from MICROPYPATH
environment variable if supported)."""
)
print("Default install path:", get_install_path())
print(
"""\
Note: only MicroPython packages (usually, named micropython-*) are supported
for installation, upip does not support arbitrary code in setup.py.
"""
)
def main():
global debug
global index_urls
global install_path
install_path = None
if len(sys.argv) < 2 or sys.argv[1] == "-h" or sys.argv[1] == "--help":
help()
return
if sys.argv[1] != "install":
fatal("Only 'install' command supported")
to_install = []
i = 2
while i < len(sys.argv) and sys.argv[i][0] == "-":
opt = sys.argv[i]
i += 1
if opt == "-h" or opt == "--help":
help()
return
elif opt == "-p":
install_path = sys.argv[i]
i += 1
elif opt == "-r":
list_file = sys.argv[i]
i += 1
with open(list_file) as f:
while True:
l = f.readline()
if not l:
break
if l[0] == "#":
continue
to_install.append(l.rstrip())
elif opt == "-i":
index_urls = [sys.argv[i]]
i += 1
elif opt == "--debug":
debug = True
else:
fatal("Unknown/unsupported option: " + opt)
to_install.extend(sys.argv[i:])
if not to_install:
help()
return
install(to_install)
if not debug:
cleanup()
if __name__ == "__main__":
main()

95
tools/upip_utarfile.py
View File

@ -1,95 +0,0 @@
import uctypes
# http://www.gnu.org/software/tar/manual/html_node/Standard.html
TAR_HEADER = {
"name": (uctypes.ARRAY | 0, uctypes.UINT8 | 100),
"size": (uctypes.ARRAY | 124, uctypes.UINT8 | 11),
}
DIRTYPE = "dir"
REGTYPE = "file"
def roundup(val, align):
return (val + align - 1) & ~(align - 1)
class FileSection:
def __init__(self, f, content_len, aligned_len):
self.f = f
self.content_len = content_len
self.align = aligned_len - content_len
def read(self, sz=65536):
if self.content_len == 0:
return b""
if sz > self.content_len:
sz = self.content_len
data = self.f.read(sz)
sz = len(data)
self.content_len -= sz
return data
def readinto(self, buf):
if self.content_len == 0:
return 0
if len(buf) > self.content_len:
buf = memoryview(buf)[: self.content_len]
sz = self.f.readinto(buf)
self.content_len -= sz
return sz
def skip(self):
sz = self.content_len + self.align
if sz:
buf = bytearray(16)
while sz:
s = min(sz, 16)
self.f.readinto(buf, s)
sz -= s
class TarInfo:
def __str__(self):
return "TarInfo(%r, %s, %d)" % (self.name, self.type, self.size)
class TarFile:
def __init__(self, name=None, fileobj=None):
if fileobj:
self.f = fileobj
else:
self.f = open(name, "rb")
self.subf = None
def next(self):
if self.subf:
self.subf.skip()
buf = self.f.read(512)
if not buf:
return None
h = uctypes.struct(uctypes.addressof(buf), TAR_HEADER, uctypes.LITTLE_ENDIAN)
# Empty block means end of archive
if h.name[0] == 0:
return None
d = TarInfo()
d.name = str(h.name, "utf-8").rstrip("\0")
d.size = int(bytes(h.size), 8)
d.type = [REGTYPE, DIRTYPE][d.name[-1] == "/"]
self.subf = d.subf = FileSection(self.f, d.size, roundup(d.size, 512))
return d
def __iter__(self):
return self
def __next__(self):
v = self.next()
if v is None:
raise StopIteration
return v
def extractfile(self, tarinfo):
return tarinfo.subf