318 lines
12 KiB
ReStructuredText
318 lines
12 KiB
ReStructuredText
|
.. _compiler:
|
||
|
|
|
||
|
|
The Compiler
|
||
|
|
============
|
||
|
|
|
||
|
|
The compilation process in MicroPython involves the following steps:
|
||
|
|
|
||
|
|
* The lexer converts the stream of text that makes up a MicroPython program into tokens.
|
||
|
|
* The parser then converts the tokens into an abstract syntax (parse tree).
|
||
|
|
* Then bytecode or native code is emitted based on the parse tree.
|
||
|
|
|
||
|
|
For purposes of this discussion we are going to add a simple language feature ``add1``
|
||
|
|
that can be use in Python as:
|
||
|
|
|
||
|
|
.. code-block:: bash
|
||
|
|
|
||
|
|
>>> add1 3
|
||
|
|
4
|
||
|
|
>>>
|
||
|
|
|
||
|
|
The ``add1`` statement takes an integer as argument and adds ``1`` to it.
|
||
|
|
|
||
|
|
Adding a grammar rule
|
||
|
|
----------------------
|
||
|
|
|
||
|
|
MicroPython's grammar is based on the `CPython grammar <https://docs.python.org/3.5/reference/grammar.html>`_
|
||
|
|
and is defined in `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.
|
||
|
|
This grammar is what is used to parse MicroPython source files.
|
||
|
|
|
||
|
|
There are two macros you need to know to define a grammar rule: ``DEF_RULE`` and ``DEF_RULE_NC``.
|
||
|
|
``DEF_RULE`` allows you to define a rule with an associated compile function,
|
||
|
|
while ``DEF_RULE_NC`` has no compile (NC) function for it.
|
||
|
|
|
||
|
|
A simple grammar definition with a compile function for our new ``add1`` statement
|
||
|
|
looks like the following:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
DEF_RULE(add1_stmt, c(add1_stmt), and(2), tok(KW_ADD1), rule(testlist))
|
||
|
|
|
||
|
|
The second argument ``c(add1_stmt)`` is the corresponding compile function that should be implemented
|
||
|
|
in ``py/compile.c`` to turn this rule into executable code.
|
||
|
|
|
||
|
|
The third required argument can be ``or`` or ``and``. This specifies the number of nodes associated
|
||
|
|
with a statement. For example, in this case, our ``add1`` statement is similar to ADD1 in assembly
|
||
|
|
language. It takes one numeric argument. Therefore, the ``add1_stmt`` has two nodes associated with it.
|
||
|
|
One node is for the statement itself, i.e the literal ``add1`` corresponding to ``KW_ADD1``,
|
||
|
|
and the other for its argument, a ``testlist`` rule which is the top-level expression rule.
|
||
|
|
|
||
|
|
.. note::
|
||
|
|
The ``add1`` rule here is just an example and not part of the standard
|
||
|
|
MicroPython grammar.
|
||
|
|
|
||
|
|
The fourth argument in this example is the token associated with the rule, ``KW_ADD1``. This token should be
|
||
|
|
defined in the lexer by editing ``py/lexer.h``.
|
||
|
|
|
||
|
|
Defining the same rule without a compile function is achieved by using the ``DEF_RULE_NC`` macro
|
||
|
|
and omitting the compile function argument:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
DEF_RULE_NC(add1_stmt, and(2), tok(KW_ADD1), rule(testlist))
|
||
|
|
|
||
|
|
The remaining arguments take on the same meaning. A rule without a compile function must
|
||
|
|
be handled explicitly by all rules that may have this rule as a node. Such NC-rules are usually
|
||
|
|
used to express sub-parts of a complicated grammar structure that cannot be expressed in a
|
||
|
|
single rule.
|
||
|
|
|
||
|
|
.. note::
|
||
|
|
The macros ``DEF_RULE`` and ``DEF_RULE_NC`` take other arguments. For an in-depth understanding of
|
||
|
|
supported parameters, see `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.
|
||
|
|
|
||
|
|
Adding a lexical token
|
||
|
|
----------------------
|
||
|
|
|
||
|
|
Every rule defined in the grammar should have a token associated with it that is defined in ``py/lexer.h``.
|
||
|
|
Add this token by editing the ``_mp_token_kind_t`` enum:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
:emphasize-lines: 12
|
||
|
|
|
||
|
|
typedef enum _mp_token_kind_t {
|
||
|
|
...
|
||
|
|
MP_TOKEN_KW_OR,
|
||
|
|
MP_TOKEN_KW_PASS,
|
||
|
|
MP_TOKEN_KW_RAISE,
|
||
|
|
MP_TOKEN_KW_RETURN,
|
||
|
|
MP_TOKEN_KW_TRY,
|
||
|
|
MP_TOKEN_KW_WHILE,
|
||
|
|
MP_TOKEN_KW_WITH,
|
||
|
|
MP_TOKEN_KW_YIELD,
|
||
|
|
MP_TOKEN_KW_ADD1,
|
||
|
|
...
|
||
|
|
} mp_token_kind_t;
|
||
|
|
|
||
|
|
Then also edit ``py/lexer.c`` to add the new keyword literal text:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
:emphasize-lines: 12
|
||
|
|
|
||
|
|
STATIC const char *const tok_kw[] = {
|
||
|
|
...
|
||
|
|
"or",
|
||
|
|
"pass",
|
||
|
|
"raise",
|
||
|
|
"return",
|
||
|
|
"try",
|
||
|
|
"while",
|
||
|
|
"with",
|
||
|
|
"yield",
|
||
|
|
"add1",
|
||
|
|
...
|
||
|
|
};
|
||
|
|
|
||
|
|
Notice the keyword is named depending on what you want it to be. For consistency, maintain the
|
||
|
|
naming standard accordingly.
|
||
|
|
|
||
|
|
.. note::
|
||
|
|
The order of these keywords in ``py/lexer.c`` must match the order of tokens in the enum
|
||
|
|
defined in ``py/lexer.h``.
|
||
|
|
|
||
|
|
Parsing
|
||
|
|
-------
|
||
|
|
|
||
|
|
In the parsing stage the parser takes the tokens produced by the lexer and converts them to an abstract syntax tree (AST) or
|
||
|
|
*parse tree*. The implementation for the parser is defined in `py/parse.c <https://github.com/micropython/micropython/blob/master/py/parse.c>`_.
|
||
|
|
|
||
|
|
The parser also maintains a table of constants for use in different aspects of parsing, similar to what a
|
||
|
|
`symbol table <https://steemit.com/programming/@drifter1/writing-a-simple-compiler-on-my-own-symbol-table-basic-structure>`_
|
||
|
|
does.
|
||
|
|
|
||
|
|
Several optimizations like `constant folding <http://compileroptimizations.com/category/constant_folding.htm>`_
|
||
|
|
on integers for most operations e.g. logical, binary, unary, etc, and optimizing enhancements on parenthesis
|
||
|
|
around expressions are performed during this phase, along with some optimizations on strings.
|
||
|
|
|
||
|
|
It's worth noting that *docstrings* are discarded and not accessible to the compiler.
|
||
|
|
Even optimizations like `string interning <https://en.wikipedia.org/wiki/String_interning>`_ are
|
||
|
|
not applied to *docstrings*.
|
||
|
|
|
||
|
|
Compiler passes
|
||
|
|
---------------
|
||
|
|
|
||
|
|
Like many compilers, MicroPython compiles all code to MicroPython bytecode or native code. The functionality
|
||
|
|
that achieves this is implemented in `py/compile.c <https://github.com/micropython/micropython/blob/master/py/compile.c>`_.
|
||
|
|
The most relevant method you should know about is this:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, bool is_repl) {
|
||
|
|
// Compile the input parse_tree to a raw-code structure.
|
||
|
|
mp_raw_code_t *rc = mp_compile_to_raw_code(parse_tree, source_file, is_repl);
|
||
|
|
// Create and return a function object that executes the outer module.
|
||
|
|
return mp_make_function_from_raw_code(rc, MP_OBJ_NULL, MP_OBJ_NULL);
|
||
|
|
}
|
||
|
|
|
||
|
|
The compiler compiles the code in four passes: scope, stack size, code size and emit.
|
||
|
|
Each pass runs the same C code over the same AST data structure, with different things
|
||
|
|
being computed each time based on the results of the previous pass.
|
||
|
|
|
||
|
|
First pass
|
||
|
|
~~~~~~~~~~
|
||
|
|
|
||
|
|
In the first pass, the compiler learns about the known identifiers (variables) and
|
||
|
|
their scope, being global, local, closed over, etc. In the same pass the emitter
|
||
|
|
(bytecode or native code) also computes the number of labels needed for the emitted
|
||
|
|
code.
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
// Compile pass 1.
|
||
|
|
comp->emit = emit_bc;
|
||
|
|
comp->emit_method_table = &emit_bc_method_table;
|
||
|
|
|
||
|
|
uint max_num_labels = 0;
|
||
|
|
for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
|
||
|
|
if (s->emit_options == MP_EMIT_OPT_ASM) {
|
||
|
|
compile_scope_inline_asm(comp, s, MP_PASS_SCOPE);
|
||
|
|
} else {
|
||
|
|
compile_scope(comp, s, MP_PASS_SCOPE);
|
||
|
|
|
||
|
|
// Check if any implicitly declared variables should be closed over.
|
||
|
|
for (size_t i = 0; i < s->id_info_len; ++i) {
|
||
|
|
id_info_t *id = &s->id_info[i];
|
||
|
|
if (id->kind == ID_INFO_KIND_GLOBAL_IMPLICIT) {
|
||
|
|
scope_check_to_close_over(s, id);
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
...
|
||
|
|
}
|
||
|
|
|
||
|
|
Second and third passes
|
||
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
|
||
|
|
The second and third passes involve computing the Python stack size and code size
|
||
|
|
for the bytecode or native code. After the third pass the code size cannot change,
|
||
|
|
otherwise jump labels will be incorrect.
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
|
||
|
|
...
|
||
|
|
|
||
|
|
// Pass 2: Compute the Python stack size.
|
||
|
|
compile_scope(comp, s, MP_PASS_STACK_SIZE);
|
||
|
|
|
||
|
|
// Pass 3: Compute the code size.
|
||
|
|
if (comp->compile_error == MP_OBJ_NULL) {
|
||
|
|
compile_scope(comp, s, MP_PASS_CODE_SIZE);
|
||
|
|
}
|
||
|
|
|
||
|
|
...
|
||
|
|
}
|
||
|
|
|
||
|
|
Just before pass two there is a selection for the type of code to be emitted, which can
|
||
|
|
either be native or bytecode.
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
// Choose the emitter type.
|
||
|
|
switch (s->emit_options) {
|
||
|
|
case MP_EMIT_OPT_NATIVE_PYTHON:
|
||
|
|
case MP_EMIT_OPT_VIPER:
|
||
|
|
if (emit_native == NULL) {
|
||
|
|
emit_native = NATIVE_EMITTER(new)(&comp->compile_error, &comp->next_label, max_num_labels);
|
||
|
|
}
|
||
|
|
comp->emit_method_table = NATIVE_EMITTER_TABLE;
|
||
|
|
comp->emit = emit_native;
|
||
|
|
break;
|
||
|
|
|
||
|
|
default:
|
||
|
|
comp->emit = emit_bc;
|
||
|
|
comp->emit_method_table = &emit_bc_method_table;
|
||
|
|
break;
|
||
|
|
}
|
||
|
|
|
||
|
|
The bytecode option is the default but something unique to note for the native
|
||
|
|
code option is that there is another option via ``VIPER``. See the
|
||
|
|
:ref:`Emitting native code <emitting_native_code>` section for more details on
|
||
|
|
viper annotations.
|
||
|
|
|
||
|
|
There is also support for *inline assembly code*, where assembly instructions are
|
||
|
|
written as Python function calls but are emitted directly as the corresponding
|
||
|
|
machine code. This assembler has only three passes (scope, code size, emit)
|
||
|
|
and uses a different implementation, not the ``compile_scope`` function.
|
||
|
|
See the `inline assembler tutorial <https://docs.micropython.org/en/latest/pyboard/tutorial/assembler.html#pyboard-tutorial-assembler>`_
|
||
|
|
for more details.
|
||
|
|
|
||
|
|
Fourth pass
|
||
|
|
~~~~~~~~~~~
|
||
|
|
|
||
|
|
The fourth pass emits the final code that can be executed, either bytecode in
|
||
|
|
the virtual machine, or native code directly by the CPU.
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
|
||
|
|
...
|
||
|
|
|
||
|
|
// Pass 4: Emit the compiled bytecode or native code.
|
||
|
|
if (comp->compile_error == MP_OBJ_NULL) {
|
||
|
|
compile_scope(comp, s, MP_PASS_EMIT);
|
||
|
|
}
|
||
|
|
}
|
||
|
|
|
||
|
|
Emitting bytecode
|
||
|
|
-----------------
|
||
|
|
|
||
|
|
Statements in Python code usually correspond to emitted bytecode, for example ``a + b``
|
||
|
|
generates "push a" then "push b" then "binary op add". Some statements do not emit
|
||
|
|
anything but instead affect other things like the scope of variables, for example
|
||
|
|
``global a``.
|
||
|
|
|
||
|
|
The implementation of a function that emits bytecode looks similar to this:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
void mp_emit_bc_unary_op(emit_t *emit, mp_unary_op_t op) {
|
||
|
|
emit_write_bytecode_byte(emit, 0, MP_BC_UNARY_OP_MULTI + op);
|
||
|
|
}
|
||
|
|
|
||
|
|
We use the unary operator expressions for an example here but the implementation
|
||
|
|
details are similar for other statements/expressions. The method ``emit_write_bytecode_byte()``
|
||
|
|
is a wrapper around the main function ``emit_get_cur_to_write_bytecode()`` that all
|
||
|
|
functions must call to emit bytecode.
|
||
|
|
|
||
|
|
.. _emitting_native_code:
|
||
|
|
|
||
|
|
Emitting native code
|
||
|
|
---------------------
|
||
|
|
|
||
|
|
Similar to how bytecode is generated, there should be a corresponding function in ``py/emitnative.c`` for each
|
||
|
|
code statement:
|
||
|
|
|
||
|
|
.. code-block:: c
|
||
|
|
|
||
|
|
STATIC void emit_native_unary_op(emit_t *emit, mp_unary_op_t op) {
|
||
|
|
vtype_kind_t vtype;
|
||
|
|
emit_pre_pop_reg(emit, &vtype, REG_ARG_2);
|
||
|
|
if (vtype == VTYPE_PYOBJ) {
|
||
|
|
emit_call_with_imm_arg(emit, MP_F_UNARY_OP, op, REG_ARG_1);
|
||
|
|
emit_post_push_reg(emit, VTYPE_PYOBJ, REG_RET);
|
||
|
|
} else {
|
||
|
|
adjust_stack(emit, 1);
|
||
|
|
EMIT_NATIVE_VIPER_TYPE_ERROR(emit,
|
||
|
|
MP_ERROR_TEXT("unary op %q not implemented"), mp_unary_op_method_name[op]);
|
||
|
|
}
|
||
|
|
}
|
||
|
|
|
||
|
|
The difference here is that we have to handle *viper typing*. Viper annotations allow
|
||
|
|
us to handle more than one type of variable. By default all variables are Python objects,
|
||
|
|
but with viper a variable can also be declared as a machine-typed variable like a native
|
||
|
|
integer or pointer. Viper can be thought of as a superset of Python, where normal Python
|
||
|
|
objects are handled as usual, while native machine variables are handled in an optimised
|
||
|
|
way by using direct machine instructions for the operations. Viper typing may break
|
||
|
|
Python equivalence because, for example, integers become native integers and can overflow
|
||
|
|
(unlike Python integers which extend automatically to arbitrary precision).
|