circuitpython/docs/develop/natmod.rst

.. _natmod:

Native machine code in .mpy files
=================================

This section describes how to build and work with .mpy files that contain native
machine code from a language other than Python.  This allows you to
write code in a language like C, compile and link it into a .mpy file, and then
import this file like a normal Python module.  This can be used for implementing
functionality which is performance critical, or for including an existing
library written in another language.

One of the main advantages of using native .mpy files is that native machine code
can be imported by a script dynamically, without the need to rebuild the main
MicroPython firmware.  This is in contrast to :ref:`cmodules` which also allows
defining custom modules in C but they must be compiled into the main firmware image.

The focus here is on using C to build native modules, but in principle any
language which can be compiled to stand-alone machine code can be put into a
.mpy file.

A native .mpy module is built using the ``mpy_ld.py`` tool, which is found in the
``tools/`` directory of the project.  This tool takes a set of object files
(.o files) and links them together to create a native .mpy files.  It requires
CPython 3 and the library pyelftools v0.25 or greater.

Supported features and limitations
----------------------------------

A .mpy file can contain MicroPython bytecode and/or native machine code.  If it
contains native machine code then the .mpy file has a specific architecture
associated with it.  Current supported architectures are (these are the valid
options for the ``ARCH`` variable, see below):

* ``x86`` (32 bit)
* ``x64`` (64 bit x86)
* ``armv7m`` (ARM Thumb 2, eg Cortex-M3)
* ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7)
* ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7)
* ``xtensa`` (non-windowed, eg ESP8266)
* ``xtensawin`` (windowed with window size 8, eg ESP32)

When compiling and linking the native .mpy file the architecture must be chosen
and the corresponding file can only be imported on that architecture.  For more
details about .mpy files see :ref:`mpy_files`.

Native code must be compiled as position independent code (PIC) and use a global
offset table (GOT), although the details of this varies from architecture to
architecture.  When importing .mpy files with native code the import machinery
is able to do some basic relocation of the native code.  This includes
relocating text, rodata and BSS sections.

Supported features of the linker and dynamic loader are:

* executable code (text)
* read-only data (rodata), including strings and constant data (arrays, structs, etc)
* zeroed data (BSS)
* pointers in text to text, rodata and BSS
* pointers in rodata to text, rodata and BSS

The known limitations are:

* data sections are not supported; workaround: use BSS data and initialise the
  data values explicitly

* static BSS variables are not supported; workaround: use global BSS variables

So, if your C code has writable data, make sure the data is defined globally,
without an initialiser, and only written to within functions.

Linker limitation: the native module is not linked against the symbol table of the
full MicroPython firmware.  Rather, it is linked against an explicit table of exported
symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware
build time.  It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system
function, for example.

New symbols can be added to the end of the table and the firmware rebuilt.
The symbols also need to be added to ``tools/mpy_ld.py``'s ``fun_table`` dict in the
same location.  This allows ``mpy_ld.py`` to be able to pick the new symbols up and
provide relocations for them when the mpy is imported.  Finally, if the symbol is a
function, a macro or stub should be added to ``py/dynruntime.h`` to make it easy to
call the function.

Defining a native module
------------------------

A native .mpy module is defined by a set of files that are used to build the .mpy.
The filesystem layout consists of two main parts, the source files and the Makefile:

* In the simplest case only a single C source file is required, which contains all
  the code that will be compiled into the .mpy module.  This C source code must
  include the ``py/dynruntime.h`` file to access the MicroPython dynamic API, and
  must at least define a function called ``mpy_init``.  This function will be the
  entry point of the module, called when the module is imported.

  The module can be split into multiple C source files if desired.  Parts of the
  module can also be implemented in Python.  All source files should be listed in
  the Makefile, by adding them to the ``SRC`` variable (see below).  This includes
  both C source files as well as any Python files which will be included in the
  resulting .mpy file.

* The ``Makefile`` contains the build configuration for the module and list the
  source files used to build the .mpy module.  It should define ``MPY_DIR`` as the
  location of the MicroPython repository (to find header files, the relevant Makefile
  fragment, and the ``mpy_ld.py`` tool), ``MOD`` as the name of the module, ``SRC``
  as the list of source files, optionally specify the machine architecture via ``ARCH``,
  and then include ``py/dynruntime.mk``.

Minimal example
---------------

This section provides a fully working example of a simple module named ``factorial``.
This module provides a single function ``factorial.factorial(x)`` which computes the
factorial of the input and returns the result.

Directory layout::

    factorial/
    ├── factorial.c
    └── Makefile

The file ``factorial.c`` contains:

.. code-block:: c

    // Include the header file to get access to the MicroPython API
    #include "py/dynruntime.h"

    // Helper function to compute factorial
    STATIC mp_int_t factorial_helper(mp_int_t x) {
        if (x == 0) {
            return 1;
        }
        return x * factorial_helper(x - 1);
    }

    // This is the function which will be called from Python, as factorial(x)
    STATIC mp_obj_t factorial(mp_obj_t x_obj) {
        // Extract the integer from the MicroPython input object
        mp_int_t x = mp_obj_get_int(x_obj);
        // Calculate the factorial
        mp_int_t result = factorial_helper(x);
        // Convert the result to a MicroPython integer object and return it
        return mp_obj_new_int(result);
    }
    // Define a Python reference to the function above
    STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial);

    // This is the entry point and is called when the module is imported
    mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
        // This must be first, it sets up the globals dict and other things
        MP_DYNRUNTIME_INIT_ENTRY

        // Make the function available in the module's namespace
        mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj));

        // This must be last, it restores the globals dict
        MP_DYNRUNTIME_INIT_EXIT
    }

The file ``Makefile`` contains:

.. code-block:: make

    # Location of top-level MicroPython directory
    MPY_DIR = ../../..

    # Name of module
    MOD = factorial

    # Source files (.c or .py)
    SRC = factorial.c

    # Architecture to build for (x86, x64, armv7m, xtensa, xtensawin)
    ARCH = x64

    # Include to get the rules for compiling and linking the module
    include $(MPY_DIR)/py/dynruntime.mk

Compiling the module
--------------------

The prerequisite tools needed to build a native .mpy file are:

* The MicroPython repository (at least the ``py/`` and ``tools/`` directories).
* CPython 3, and the library pyelftools (eg ``pip install 'pyelftools>=0.25'``).
* GNU make.
* A C compiler for the target architecture (if C source is used).
* Optionally ``mpy-cross``, built from the MicroPython repository (if .py source is used).

Be sure to select the correct ``ARCH`` for the target you are going to run on.
Then build with::

    $ make

Without modifying the Makefile you can specify the target architecture via::

    $ make ARCH=armv7m

Module usage in MicroPython
---------------------------

Once the module is built there should be a file called ``factorial.mpy``.  Copy
this so it is accessible on the filesystem of your MicroPython system and can be
found in the import path.  The module can now be accessed in Python just like any
other module, for example::

    import factorial
    print(factorial.factorial(10))
    # should display 3628800

Further examples
----------------

See ``examples/natmod/`` for further examples which show many of the available
features of native .mpy modules.  Such features include:

* using multiple C source files
* including Python code alongside C code
* rodata and BSS data
* memory allocation
* use of floating point
* exception handling
* including external C libraries