Commit Graph

293 Commits

Author SHA1 Message Date
Damien George a59282b9bf tools/pyboard.py: Add fast raw-paste mode.
This commit adds support to pyboard.py for the new raw REPL paste mode.

Note that this new pyboard.py is fully backwards compatible with old
devices (it detects if the device supports the new raw REPL paste mode).

Signed-off-by: Damien George <damien@micropython.org>
2020-12-01 22:35:13 +11:00
Damien George 547e8a9fe7 tools/ci.sh: Add helper script to run CI tasks.
The aim is for this script to be used on any CI platform, as well as run
locally.

Signed-off-by: Damien George <damien@micropython.org>
2020-11-29 22:21:28 +11:00
awachtler 56e0932485 tools/upip.py: Support explicit port number in host.
Adding a port number other then 443 to a PyPI URL may be needed if a local
server like devpi is used.
2020-10-20 12:44:30 +11:00
stijn 4b35aa5730 tools: Write msvc-compatible frozen content.
The msvc compiler doesn't accept zero-sized arrays so let the freezing
process generate compatible C code in case no modules are found and the
involved arrays are all empty.  This doesn't affect the functionality in
any way because those arrays only get accessed when mp_frozen_mpy_names
contains names, i.e.  when modules are actually found.
2020-09-11 10:51:55 +10:00
Damien George 4f2fe34623 tools/mpy-tool.py: Fix merge of multiple mpy files to POP_TOP correctly.
MP_BC_CALL_FUNCTION will leave the result on the Python stack, so that
result must be discarded by MP_BC_POP_TOP.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-09 00:11:51 +10:00
Damien George 06659077a8 all: Update Python code to conform to latest black formatting.
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:

- If there is a trailing comma in a list (eg [], () or function call) then
  that list is now written out with one line per element.  So remove such
  trailing commas where the list should stay on one line.

- Spaces at the start of """ doc strings are removed.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 15:18:01 +10:00
Damien George 448319a745 tools/makemanifest.py: Use os.makedirs to make path for generated files.
The existing implementation of mkdir() in this file is not sophisticated
enough to work correctly on all operating systems (eg Mac can raise
EISDIR).  Using the standard os.makedirs() function handles all cases
correctly.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 11:18:48 +10:00
Martin Milata 492cf34fd8 tools/mpy-tool.py: Fix offset of line number info.
Signed-off-by: Martin Milata <martin@martinmilata.cz>
2020-08-21 16:17:07 +10:00
Michael Buesch 60cf2c0959 tools/pyboard.py: Replace eval() of received data with alternative.
Prior to this commit, pyboard.py used eval() to "parse" file data received
from the board.  Using eval() on received data from a device is dangerous,
because a malicious device may inject arbitrary code execution on the PC
that is doing the operation.

Consider the following scenario:

Eve may write a malicious script to Bob's board in his absence.  On return
Bob notices that something is wrong with the board, because it doesn't work
as expected anymore.  He wants to read out boot.py (or any other file) to
see what is wrong.  What he gets is a remote code execution on his PC.

Proof of concept:

Eve:

  $ cat boot.py
  _print = print
  print = lambda *x, **y: _print("os.system('ls /; echo Pwned!')", end="\r\n\x04")
  $ ./pyboard.py -f cp boot.py :
  cp boot.py :boot.py

Bob:

  $ ./pyboard.py -f cp :boot.py /tmp/foo
  cp :boot.py /tmp/foo
  bin   chroot  dev  home  lib32  media  opt   root  sbin  sys  usr
  boot  config  etc  lib   lib64  mnt    proc  run   srv   tmp  var
  Pwned!

There's also the possibility that the device is malfunctioning and sends
random and possibly dangerous data back to the PC, to be eval'd.

Fix this problem by using ast.literal_eval() to parse the received bytes,
instead of eval().

Signed-off-by: Michael Buesch <m@bues.ch>
2020-08-21 16:08:03 +10:00
Damien George b731bd0ce6 tools/makemanifest.py: Print nicely formatted errors from mpy-cross.
If mpy-cross exits with an error be sure to print that error in a way that
is readable, instead of a long bytes object.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-08 14:46:05 +10:00
Damien George 952de5cb77 tools/makemanifest.py: Use errno.EEXIST instead of number 17.
To make this code more portable, across different platforms.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-26 10:56:24 +10:00
Thorsten von Eicken 3e758ef235 lib/mbedtls_errors: Add code to patch mbedtls for shortened error strs.
The file `mbedtls_errors/mp_mbedtls_errors.c` can be used instead of
`mbedtls/library/error.c` to give shorter error strings, reducing the build
size of the error strings from about 12-16kB down to about 2-5kB.
2020-07-20 23:53:27 +10:00
Andrew Leech 07f181a216 Revert "tools/pydfu.py: Respect longer timeouts requested by DFU dev..."
This reverts commit 4d6f60d428.

This implementation used the timeout as a maximum amount of time needed for
the operation, when actually the spec and other tools suggest that it's the
minumum delay needed between subsequent USB transfers.
2020-07-01 16:54:03 +10:00
Andrew Leech 4d6f60d428 tools/pydfu.py: Respect longer timeouts requested by DFU device/mboot. 2020-06-30 21:22:00 +10:00
Damien George 76faeed098 tools/makemanifest.py: Support freezing a subdirectory recursively.
This adds support for freezing an entire directory while keeping the
directory as part of the import path.  For example

    freeze("path/to/library", "module")

will recursively freeze all scripts in "path/to/library/module" and have
them importable as "from module import ...".

Signed-off-by: Damien George <damien@micropython.org>
2020-06-24 12:21:30 +10:00
David Lechner 77ed6f69ac tools/uncrustify: Enable more opts to remove space between func and '('.
With only `sp_func_proto_paren = remove` set there are some cases where
uncrustify misses removing a space between the function name and the
opening '('.  This sets all of the related options to `force` as well.
2020-06-19 22:07:32 +10:00
Damien George 026fda605e tools/codeformat.py: Include extmod/{btstack,nimble} in code formatting.
Signed-off-by: Damien George <damien@micropython.org>
2020-06-18 22:20:20 +10:00
David Lechner b4d0d7bf03 tools/uncrustify: Update config for v0.71.0.
This is the result of running...

    uncrustify -c tools/uncrustify.cfg --update-config-with-doc -o tools/uncrustify.cfg

...with some manual fixups to correct places where it changed things it
should not have.

Essentially it just adds new config parameters introduced in v0.71.0
with their default values.

Signed-off-by: David Lechner <david@pybricks.com>
2020-06-14 11:08:05 -05:00
David Lechner ecd7826316 tools/codeformat.py: Remove sizeof fixup.
Formatting for `* sizeof` was fixed in uncrustify v0.71, so we no longer
need the fixups for it.  Also, there was one file where the updated
uncrustify caught a problem that the regex didn't pick up, which is updated
in this commit.

Signed-off-by: David Lechner <david@pybricks.com>
2020-06-14 20:24:18 +10:00
David Lechner bd7c92e17d tools/uncrustify.cfg: Remove deprecated sp_word_brace option.
This option was removed in uncrustify v0.71.

Signed-off-by: David Lechner <david@pybricks.com>
2020-06-14 20:24:03 +10:00
David Lechner 8f642677f7 tools/codeformat.py: Add verbose option to pass to uncrustify and black.
This adds a new command line option `-v` to `tools/codeformat.py` to enable
verbose printing of all files that are scanned.

Normally `uncrustify` and `black` are called with the `-q` option so
setting verbose suppresses the `-q` option and on `black` also enables the
`-v` option which makes it print out all file names matching the filter
similar to how `uncrustify` does by default.
2020-05-29 22:59:56 +10:00
David Lechner 2d1fef7096 tools/codeformat.py: Use -q option on uncrustify to make output quiet.
This suppresses the Parsing: <file> as language C lines.  This makes
parsing run a bit faster and on CI it makes for less scrolling through logs
(and black already uses the -q option).
2020-05-28 10:08:38 +10:00
stijn 84fa3312cf all: Format code to add space after C++-style comment start.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
2020-04-23 11:24:25 +10:00
Damien George 6e2871df59 tools/check_code_size.sh: Remove unused script.
It's now replaced by tools/metrics.py.
2020-04-22 14:07:56 +10:00
Damien George 17dc86369f tools/metrics.py: Use OrderedDict when reading build log.
So that the output (eg of the diff command) always has the lines in the
same order.
2020-04-22 14:03:26 +10:00
Damien George 1cc24cd39a tools/metrics.py: Don't build mpy-cross if not needed by any ports.
To save build time.
2020-04-22 14:03:07 +10:00
Damien George a4423570e2 tools/metrics.py: Add option to diff to error if delta above threshold.
Useful for things like CI where the size check is automated.
2020-04-22 14:02:14 +10:00
stijn bcf01d1686 all: Fix implicit conversion from double to float.
These are found when building with -Wfloat-conversion.
2020-04-18 22:42:24 +10:00
Jim Mussared 85858e72df py/objexcept: Allow compression of exception message text.
The decompression of error-strings is only done if the string is accessed
via printing or via er.args.  Tests are added for this feature to ensure
the decompression works.
2020-04-05 15:02:06 +10:00
David Lechner 3dc324d3f1 tests: Format all Python code with black, except tests in basics subdir.
This adds the Python files in the tests/ directory to be formatted with
./tools/codeformat.py.  The basics/ subdirectory is excluded for now so we
aren't changing too much at once.

In a few places `# fmt: off`/`# fmt: on` was used where the code had
special formatting for readability or where the test was actually testing
the specific formatting.
2020-03-30 13:21:58 +11:00
Lars Kellogg-Stedman 3a0f64fc7a tools/pyboard.py: Add -d as an alias for --device. 2020-03-30 11:37:32 +11:00
Lars Kellogg-Stedman 1cf994c48b tools/pyboard.py: Support setting device/baudrate from shell env vars.
Allow defaults for --device and --baudrate to be set in the environment
using PYBOARD_DEVICE and PYBOARD_BAUDRATE.
2020-03-30 11:37:32 +11:00
Damien George 1a3e386c67 all: Remove spaces inside and around parenthesis.
Using new options enabled in the uncrustify configuration.
2020-03-28 23:36:44 +11:00
Damien George feb2577585 all: Remove spaces between nested paren and inside function arg paren.
Using new options enabled in the uncrustify configuration.
2020-03-25 00:39:46 +11:00
stijn 1b3e0e10b9 tools/codeformat.py: Include all msvc C code in auto-format. 2020-03-25 00:36:42 +11:00
Andrew Leech 38ccb4c643 tools/pydfu.py: Display any error strings from device/mboot. 2020-03-22 15:24:57 +11:00
Andrew Leech 2966d83a65 tools/pydfu.py: Add args for VID/PID & exit with cleaner error handling. 2020-03-22 13:33:04 +11:00
David Lechner 8a4ce6b79a tools/codeformat.py: Eliminate need for sizeof fixup.
This eliminates the need for the sizeof regex fixup by rearranging things a
bit.  All other bitfields already use the parentheses around expressions
with sizeof, so one case is fixed by following this convention.

VM_MAX_STATE_ON_STACK is the only remaining problem and it can be worked
around by changing the order of the operands.
2020-03-11 14:34:40 +11:00
Damien George 554c01fc25 tools/metrics.py: Use check_call instead of run to error out on error. 2020-03-11 14:23:39 +11:00
MikeTeachman 8db5d2d1f1 tools/makemanifest.py: Fix build on Windows by adding .exe to mpy-cross.
When using a manifest on Windows the reference to mpy-cross compiler was
missing the .exe file extension, so add it when appropriate.

Also allow the default path to mpy-cross to be overridden by the (optional)
MICROPY_MPYCROSS environment variable, to allow full flexibility on any OS.
2020-03-05 10:26:31 +11:00
Damien George 69661f3343 all: Reformat C and Python source code with tools/codeformat.py.
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-28 10:33:03 +11:00
Damien George 4b23e98fb0 tools/codeformat.py: Add formatter using uncrustify for C, black for Py.
This commit adds a tool, codeformat.py, which will reformat C and Python
code to fit a certain style.  By default the tool will reformat (almost)
all the original (ie not 3rd-party) .c, .h and .py files in this
repository.  Passing filenames on the command-line to codeformat.py will
reformat only those.  Reformatting is done in-place.

uncrustify is used for C reformatting, which is available for many
platforms and can be easily built from source, see
https://github.com/uncrustify/uncrustify.  The configuration for uncrustify
is also added in this commit and values are chosen to best match the
existing code style.  A small post-processing stage on .c and .h files is
done by codeformat.py (after running uncrustify) to fix up some minor
items:
- space inserted after * when used as multiplication with sizeof
- #if/ifdef/ifndef/elif/else/endif are dedented by one level when they are
  configuring if-blocks and case-blocks.

For Python code, the formatter used is black, which can be pip-installed;
see https://github.com/psf/black.  The defaults are used, except for line-
length which is set at 99 characters to match the "about 100" line-length
limit used in C code.

The formatting tools used and their configuration were chosen to strike a
balance between keeping existing style and not changing too many lines of
code, and enforcing a relatively strict style (especially for Python code).
This should help to keep the code consistent across everything, and reduce
cognitive load when writing new code to match the style.
2020-02-28 10:14:28 +11:00
Damien George a636837987 tools/makemanifest.py: Support freezing with empty list of mpy files.
Fixes issue #5655.
2020-02-20 10:41:50 +11:00
Michael Buesch 1604606238 tools/pyboard.py: Change shebang to use python3.
This script still works with Python 2 but Python 3 is recommended.
2020-02-01 00:06:26 +11:00
Michael Buesch 1cadb12d1c tools/pyboard.py: Use slice del instead of list.clear() for Py2 compat.
Python 2 does not have list.clear().
2020-02-01 00:05:29 +11:00
Michael Buesch 83afd48ad9 tools/pyboard.py: Add option --no-follow to detach after sending script.
This option makes pyboard.py exit as soon as the script/command is
successfully sent to the device, ie it does not wait for any output.  This
can help to avoid hangs if the board is being rebooted with --comman (for
example).

Example usage:

    $ python3 ./tools/pyboard.py --device /dev/ttyUSB0 --no-follow \
        --command 'import machine; machine.reset()'
2020-02-01 00:03:37 +11:00
c0rejump 6db5cede06 tools/pydfu.py: Clean up syntax, update comments and docstrings.
Some parts of code have been aligned to increase readability.  In general
'' instead of "" were used wherever possible to keep the same convention
for entire file.  Import inspect line has been moved to the top according
to hints reported by pep8 tools.  A few extra spaces were removed, a few
missing spaces were added.  Comments have been updated, mostly in
"read_dfu_file" function.  Some other comments have been capitalized and/or
slightly updated.  A few docstrings were fixed as well.  No real code
changes intended.
2020-01-23 13:23:12 +11:00
Damien George a11e306227 tools: Add metrics.py script to build and compute port sizes/metrics. 2020-01-23 12:48:52 +11:00
Jason Neal de78a9e317 tools/gen-cpydiff.py: Adjust subsections to sentence case. 2020-01-06 22:16:18 +11:00
David Lechner fd0ba7be07 tools/tinytest-codegen.py: Add extra newline and result message.
This is an alternative to f4ed2df that adds a newline so that the output of
the test starts on a new line and the result of the test is prefixed with
"result: " to distinguish it from the test output.

Suggested-by: @dpgeorge
2019-12-19 17:55:50 +11:00