After discussing with danh, I noticed that `a/**/b` would not match `a/b`.
After correcting this and re-running "pre-commit run --all", additional
files were reindented, including the codeformat script itself.
In #4683, tannewt noticed that uncrustify was not running on some
file in common-hal.
I investigated and found that it was not being run on a bunch of paths.
Rather than make incremental changes, I rewrote list_files to work
bsaed on regular expressions; these regular expressions are created from
the same git-style glob patterns.
I spot-checked some specific filenames after this change, and all looks good:
```
$ python3 tools/codeformat.py -v --dry-run tests/basics/int_small.py ports/raspberrypi/common-hal/pulseio/PulseIn.c extmod/virtpin.c tests/thread/thread_exit1.py ports/raspberrypi/background.h extmod/re1.5/recursiveloop.c
tools/codeformat.py -v --dry-run tests/basics/int_small.py ports/raspberrypi/common-hal/pulseio/PulseIn.c extmod/virtpin.c tests/thread/thread_exit1.py ports/raspberrypi/background.h extmod/re1.5/recursiveloop.c
uncrustify -c /home/jepler/src/circuitpython/tools/uncrustify.cfg -lC --no-backup extmod/virtpin.c ports/raspberrypi/background.h ports/raspberrypi/common-hal/pulseio/PulseIn.c
black --fast --line-length=99 -v tests/thread/thread_exit1.py
```
recursiveloop and int_small are excluded, while PulseIn, virtpin,
and background are included.
Testing running from a subdirectory (not _specifically_ supported though):
```
(cd ports && python3 ../tools/codeformat.py -v --dry-run raspberrypi/common-hal/pulseio/PulseIn.c ../extmod/virtpin.c)
../tools/codeformat.py -v --dry-run raspberrypi/common-hal/pulseio/PulseIn.c ../extmod/virtpin.c
uncrustify -c /home/jepler/src/circuitpython/tools/uncrustify.cfg -lC --no-backup ../extmod/virtpin.c raspberrypi/common-hal/pulseio/PulseIn.
```
As a side-effect, a bunch more files are re-formatted now. :-P
Since version 21.4b0, Black now processes one-line docstrings by stripping
leading and trailing spaces, and adding a padding space when needed to
break up """"; see https://github.com/psf/black/pull/1740
This commit makes the Python code in this repository conform to this rule.
This is now the default, but can be overridden with CLI `--no-exclusive`,
or constructing `Pyboard(..., exclusive=False)`.
Signed-off-by: Damien George <damien@micropython.org>
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update"
will all point to the same memory.
Shaves ~1KB off the image size, and potentially allows
bigger savings if qstr attrs are initialized in qstr_init(),
and not stored in the image.
It's a bit of a pitfall with user C modules that including them in the
build does not automatically enable them. This commit changes the docs and
examples for user C modules to encourage writers of user C modules to
enable them unconditionally. This makes things simpler and covers most use
cases.
See discussion in issue #6960, and also #7086.
Signed-off-by: Damien George <damien@micropython.org>
When running from pre-commit, we believe the different invocations of
sed were racing with each other, sometimes leaving zero-byte files in
the filesystem (ow)
Instead of raising a ZeroDivisionError, this tool now just skips any
elements in the DFU file that have zero size.
Signed-off-by: Damien George <damien@micropython.org>
This saves about 60 bytes (Feather M4 went from 45040 -> 45100 bytes free)
66 bytes of data eliminated, but 6 bytes paid back to initialize the length
field.
Updates the zephyr docker image to the latest, v0.11.13. This updates CI
to use zephyr SDK v0.12.2 and GCC v10.2.0 for the zephyr port.
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
Refactors the zephyr build infrastructure to build MicroPython as a
cmake target, using the recently introduced core cmake rules.
This change makes it possible to build the zephyr port like most other
zephyr applications using west or cmake directly. It simplifies building
with extra cmake arguments, such as specifying an alternate conf file or
adding an Arduino shield. It also enables building the zephyr port
anywhere in the host file system, which will allow regressing across
multiple boards with the zephyr twister script.
Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
This allows customising which features can be enabled in a frozen library.
e.g. `include("path.py", extra_features=True)`
in path.py:
options.defaults(standard_features=True)
if options.standard_features:
# freeze standard modules.
if options.extra_features:
# freeze extra modules.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
So that all MicroPython ports that use tinyusb use the same version. Also
requires fewer submodule checkouts when building rp2 along with other ports
that use tinyusb.
Signed-off-by: Damien George <damien@micropython.org>
The upip module is frozen into ports supporting it, and it is included in
the source tree, so there is no need to get it from PyPi. Moreover the
PyPi package referred to is an out-of-date version of upip which is
basically unrelated to our upip.py because the source is taken from a fork
of micropython-lib instead of this repository.
The main rules enforced are:
- At most 72 characters in the subject line, with a ": " in it.
- At most 75 characters per line in the body.
- No "noreply" email addresses.
The RP2040 is new microcontroller from Raspberry Pi that features
two Cortex M0s and eight PIO state machines that are good for
crunching lots of data. It has 264k RAM and a built in UF2
bootloader too.
Datasheet: https://pico.raspberrypi.org/files/rp2040_datasheet.pdf
Because the version included in xtensa-lx106-elf-standalone.tar.gz needs
Python 2 (and pyserial for Python 2).
Signed-off-by: Damien George <damien@micropython.org>
The -Og optimisation level produces a more realistic build, gives a better
debugging experience, and generates smaller code than -O0, allowing debug
builds to fit in flash.
This commit also assigns variables in can.c to prevent warnings when -Og is
used, and builds a board in CI with DEBUG=1 enabled.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support to pyboard.py for the new raw REPL paste mode.
Note that this new pyboard.py is fully backwards compatible with old
devices (it detects if the device supports the new raw REPL paste mode).
Signed-off-by: Damien George <damien@micropython.org>
The msvc compiler doesn't accept zero-sized arrays so let the freezing
process generate compatible C code in case no modules are found and the
involved arrays are all empty. This doesn't affect the functionality in
any way because those arrays only get accessed when mp_frozen_mpy_names
contains names, i.e. when modules are actually found.
MP_BC_CALL_FUNCTION will leave the result on the Python stack, so that
result must be discarded by MP_BC_POP_TOP.
Signed-off-by: Damien George <damien@micropython.org>
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:
- If there is a trailing comma in a list (eg [], () or function call) then
that list is now written out with one line per element. So remove such
trailing commas where the list should stay on one line.
- Spaces at the start of """ doc strings are removed.
Signed-off-by: Damien George <damien@micropython.org>
The existing implementation of mkdir() in this file is not sophisticated
enough to work correctly on all operating systems (eg Mac can raise
EISDIR). Using the standard os.makedirs() function handles all cases
correctly.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, pyboard.py used eval() to "parse" file data received
from the board. Using eval() on received data from a device is dangerous,
because a malicious device may inject arbitrary code execution on the PC
that is doing the operation.
Consider the following scenario:
Eve may write a malicious script to Bob's board in his absence. On return
Bob notices that something is wrong with the board, because it doesn't work
as expected anymore. He wants to read out boot.py (or any other file) to
see what is wrong. What he gets is a remote code execution on his PC.
Proof of concept:
Eve:
$ cat boot.py
_print = print
print = lambda *x, **y: _print("os.system('ls /; echo Pwned!')", end="\r\n\x04")
$ ./pyboard.py -f cp boot.py :
cp boot.py :boot.py
Bob:
$ ./pyboard.py -f cp :boot.py /tmp/foo
cp :boot.py /tmp/foo
bin chroot dev home lib32 media opt root sbin sys usr
boot config etc lib lib64 mnt proc run srv tmp var
Pwned!
There's also the possibility that the device is malfunctioning and sends
random and possibly dangerous data back to the PC, to be eval'd.
Fix this problem by using ast.literal_eval() to parse the received bytes,
instead of eval().
Signed-off-by: Michael Buesch <m@bues.ch>
.. however, the number of endpoints is only set for SAMD (8).
Other ports need to set the value. Otherwise, the build will show
the message
```
Unable to check whether maximum number of endpoints is respected
```
Since we made the decision to allow translations which do not have coverage in
the terminal font, these routinely occur and are expected. The message is
unhelpful and due to their voume make it harder to find relevant information
particularly in github actions results.
If mpy-cross exits with an error be sure to print that error in a way that
is readable, instead of a long bytes object.
Signed-off-by: Damien George <damien@micropython.org>
The file `mbedtls_errors/mp_mbedtls_errors.c` can be used instead of
`mbedtls/library/error.c` to give shorter error strings, reducing the build
size of the error strings from about 12-16kB down to about 2-5kB.
Also:
- Remove download count update because the files are no longer
on GitHub.
- Add "extensions" and "languages" to each board dictionary so we
can stop using "files" entirely.
This reverts commit 4d6f60d428.
This implementation used the timeout as a maximum amount of time needed for
the operation, when actually the spec and other tools suggest that it's the
minumum delay needed between subsequent USB transfers.
This adds support for freezing an entire directory while keeping the
directory as part of the import path. For example
freeze("path/to/library", "module")
will recursively freeze all scripts in "path/to/library/module" and have
them importable as "from module import ...".
Signed-off-by: Damien George <damien@micropython.org>
With only `sp_func_proto_paren = remove` set there are some cases where
uncrustify misses removing a space between the function name and the
opening '('. This sets all of the related options to `force` as well.
This is the result of running...
uncrustify -c tools/uncrustify.cfg --update-config-with-doc -o tools/uncrustify.cfg
...with some manual fixups to correct places where it changed things it
should not have.
Essentially it just adds new config parameters introduced in v0.71.0
with their default values.
Signed-off-by: David Lechner <david@pybricks.com>
Formatting for `* sizeof` was fixed in uncrustify v0.71, so we no longer
need the fixups for it. Also, there was one file where the updated
uncrustify caught a problem that the regex didn't pick up, which is updated
in this commit.
Signed-off-by: David Lechner <david@pybricks.com>
This adds a new command line option `-v` to `tools/codeformat.py` to enable
verbose printing of all files that are scanned.
Normally `uncrustify` and `black` are called with the `-q` option so
setting verbose suppresses the `-q` option and on `black` also enables the
`-v` option which makes it print out all file names matching the filter
similar to how `uncrustify` does by default.
This suppresses the Parsing: <file> as language C lines. This makes
parsing run a bit faster and on CI it makes for less scrolling through logs
(and black already uses the -q option).
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
The HID descriptor reported by circuitpython erroneously limited the
maximum keycode to 101, which prevented circuitpython from sending a
number of otherwise valid keycodes.
Closes#274
The decompression of error-strings is only done if the string is accessed
via printing or via er.args. Tests are added for this feature to ensure
the decompression works.
This adds the Python files in the tests/ directory to be formatted with
./tools/codeformat.py. The basics/ subdirectory is excluded for now so we
aren't changing too much at once.
In a few places `# fmt: off`/`# fmt: on` was used where the code had
special formatting for readability or where the test was actually testing
the specific formatting.
This eliminates the need for the sizeof regex fixup by rearranging things a
bit. All other bitfields already use the parentheses around expressions
with sizeof, so one case is fixed by following this convention.
VM_MAX_STATE_ON_STACK is the only remaining problem and it can be worked
around by changing the order of the operands.
When using a manifest on Windows the reference to mpy-cross compiler was
missing the .exe file extension, so add it when appropriate.
Also allow the default path to mpy-cross to be overridden by the (optional)
MICROPY_MPYCROSS environment variable, to allow full flexibility on any OS.
This commit adds a tool, codeformat.py, which will reformat C and Python
code to fit a certain style. By default the tool will reformat (almost)
all the original (ie not 3rd-party) .c, .h and .py files in this
repository. Passing filenames on the command-line to codeformat.py will
reformat only those. Reformatting is done in-place.
uncrustify is used for C reformatting, which is available for many
platforms and can be easily built from source, see
https://github.com/uncrustify/uncrustify. The configuration for uncrustify
is also added in this commit and values are chosen to best match the
existing code style. A small post-processing stage on .c and .h files is
done by codeformat.py (after running uncrustify) to fix up some minor
items:
- space inserted after * when used as multiplication with sizeof
- #if/ifdef/ifndef/elif/else/endif are dedented by one level when they are
configuring if-blocks and case-blocks.
For Python code, the formatter used is black, which can be pip-installed;
see https://github.com/psf/black. The defaults are used, except for line-
length which is set at 99 characters to match the "about 100" line-length
limit used in C code.
The formatting tools used and their configuration were chosen to strike a
balance between keeping existing style and not changing too many lines of
code, and enforcing a relatively strict style (especially for Python code).
This should help to keep the code consistent across everything, and reduce
cognitive load when writing new code to match the style.
This option makes pyboard.py exit as soon as the script/command is
successfully sent to the device, ie it does not wait for any output. This
can help to avoid hangs if the board is being rebooted with --comman (for
example).
Example usage:
$ python3 ./tools/pyboard.py --device /dev/ttyUSB0 --no-follow \
--command 'import machine; machine.reset()'
Some parts of code have been aligned to increase readability. In general
'' instead of "" were used wherever possible to keep the same convention
for entire file. Import inspect line has been moved to the top according
to hints reported by pep8 tools. A few extra spaces were removed, a few
missing spaces were added. Comments have been updated, mostly in
"read_dfu_file" function. Some other comments have been capitalized and/or
slightly updated. A few docstrings were fixed as well. No real code
changes intended.
Introduces a way to place CircuitPython code and data into
tightly coupled memory (TCM) which is accessible by the CPU in a
single cycle. It also frees up room in the corresponding cache for
intermittent data. Loading from external flash is slow!
The data cache is also now enabled.
Adds support for the iMX RT 1021 chip. Adds three new boards:
* iMX RT 1020 EVK
* iMX RT 1060 EVK
* Teensy 4.0
Related to #2492, #2472 and #2477. Fixes#2475.
It was intended that the `f.load_glyphs` line was fast and did most of
the work. However, it actually didn't, because it's necessary to pass
in a code point by number, not by string.
Additionally, a little light layer violation is needed to make the check
for missing characters fast. This used to be less important, as no
fonts had missing characters. However, it would take an appreciable
length of time on the Korean translation when failing to find hundreds
of different code points.
Testing performed: built
build-circuitplayground_express_displayio/autogen_display_resources.c with ko
translation before and after change. verified the file content was identical.
Time went from about 7s on my machine to way under 1 second.
This is an alternative to f4ed2df that adds a newline so that the output of
the test starts on a new line and the result of the test is prefixed with
"result: " to distinguish it from the test output.
Suggested-by: @dpgeorge
This patch allows executing .mpy files (including native ones) directly on
a target, eg a board over a serial connection. So there's no need to copy
the file to its filesystem to test it.
For example:
$ mpy-cross foo.py
$ pyboard.py foo.mpy
This makes the loading of viper-code-with-relocations a bit neater and
easier to understand, by treating the rodata/bss like a special object to
be loaded into the constant table (which is how it behaves).
We don't want to add a feature flag to .mpy files that indicate float
support because it will get complex and difficult to use. Instead the .mpy
is built using whatever precision it chooses (float or double) and the
native glue API will convert between this choice and what the host runtime
actually uses.
This commit adds a new tool called mpy_ld.py which is essentially a linker
that builds .mpy files directly from .o files. A new header file
(dynruntime.h) and makefile fragment (dynruntime.mk) are also included
which allow building .mpy files from C source code. Such .mpy files can
then be dynamically imported as though they were a normal Python module,
even though they are implemented in C.
Converting .o files directly (rather than pre-linked .elf files) allows the
resulting .mpy to be more efficient because it has more control over the
relocations; for example it can skip PLT indirection. Doing it this way
also allows supporting more architectures, such as Xtensa which has
specific needs for position-independent code and the GOT.
The tool supports targets of x86, x86-64, ARM Thumb and Xtensa (windowed
and non-windowed). BSS, text and rodata sections are supported, with
relocations to all internal sections and symbols, as well as relocations to
some external symbols (defined by dynruntime.h), and linking of qstrs.
Usage:
mpy-tool.py -o merged.mpy --merge mod1.mpy mod2.mpy
The constituent .mpy files are executed sequentially when the merged file
is imported, and they all use the same global namespace.
In cases where more than one board is connected to a single computer it can become pretty hard to figure out which board you're actually talking to. For example, if you have several MIDI-compatible boards they all show up as "CircuitPython MIDI". This change allows boards to replace the "CircuitPython" part of their USB descriptors with more specific text, for example, "CircuitPython Feather" or just "Feather". This will let folks more easily tell boards apart.
The new option is named `USB_INTERFACE_NAME` and is available in `mkconfigboard.mk`. For example:
```
USB_INTERFACE_NAME = "Feather"
```
While the new manifest.py style got introduced for freezing python code
into the resulting binary, the old way - where files and modules within
ports/*/modules where baked into the resulting binary - was still
supported via `freeze('$(PORT_DIR)/modules')` within manifest.py.
However behaviour changed for symlinked directories (=modules), as those
links weren't followed anymore.
This commit restores the original behaviour by explicitly following
symlinks within a modules/ directory
When loading a manifest file, e.g. by include(), it will chdir first to the
directory of that manifest. This means that all file operations within a
manifest are relative to that manifest's location.
As a consequence of this, additional environment variables are needed to
find absolute paths, so the following are added: $(MPY_LIB_DIR),
$(PORT_DIR), $(BOARD_DIR). And rename $(MPY) to $(MPY_DIR) to be
consistent.
Existing manifests are updated to match.
This introduces a new build variable FROZEN_MANIFEST which can be set to a
manifest listing (written in Python) that describes the set of files to be
frozen in to the firmware.
Instead of encoding 4 zero bytes as placeholders for the simple_name and
source_file qstrs, and storing the qstrs after the bytecode, store the
qstrs at the location of these 4 bytes. This saves 4 bytes per bytecode
function stored in a .mpy file (for example lcd160cr.mpy drops by 232
bytes, 4x 58 functions). And resulting code size is slightly reduced on
ports that use this feature.
This patch compresses the second part of the bytecode prelude which
contains the source file name, function name, source-line-number mapping
and cell closure information. This part of the prelude now begins with a
single varible length unsigned integer which encodes 2 numbers, being the
byte-size of the following 2 sections in the header: the "source info
section" and the "closure section". After decoding this variable unsigned
integer it's possible to skip over one or both of these sections very
easily.
This scheme saves about 2 bytes for most functions compared to the original
format: one in the case that there are no closure cells, and one because
padding was eliminated.
The start of the bytecode prelude contains 6 numbers telling the amount of
stack needed for the Python values and exceptions, and the signature of the
function. Prior to this patch these numbers were all encoded one after the
other (2x variable unsigned integers, then 4x bytes), but using so many
bytes is unnecessary.
An entropy analysis of around 150,000 bytecode functions from the CPython
standard library showed that the optimal Shannon coding would need about
7.1 bits on average to encode these 6 numbers, compared to the existing 48
bits.
This patch attempts to get close to this optimal value by packing the 6
numbers into a single, varible-length unsigned integer via bit-wise
interleaving. The interleaving scheme is chosen to minimise the average
number of bytes needed, and at the same time keep the scheme simple enough
so it can be implemented without too much overhead in code size or speed.
The scheme requires about 10.5 bits on average to store the 6 numbers.
As a result most functions which originally took 6 bytes to encode these 6
numbers now need only 1 byte (in 80% of cases).
