Commit Graph

407 Commits

Author SHA1 Message Date
Damien George bfe9eba484 tools/autobuild: Build esp8266 OTA image with GENERIC_1M board.
Because the GENERIC board won't fit in the flash defined by esp8266_ota.ld.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-30 15:47:49 +11:00
Damien George a29c70c9b4 esp8266: Allow building a board to any dest directory.
Signed-off-by: Damien George <damien@micropython.org>
2021-12-30 15:47:49 +11:00
Damien George 959e6f7da9 tools/upip.py: Skip '.frozen' entry in sys.path for install path.
Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:55:37 +11:00
Damien George 028776d97b tools/mpremote: Add link to mpremote docs URL in help message.
Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:35:24 +11:00
Sergei Silnov 74e995dfd2 tools/mpremote: Add help command.
Fixes issue #7480
2021-12-29 09:46:33 +01:00
Michael Bentley 7566d107d5 tools/mpremote: Add mkdir and rmdir to RemoteFS.
This allows the remote MicroPython instance to create and delete
directories from the mounted host filesystem in addition to the already
existing functionality of reading, creating, and modifying files.

Signed-off-by: Michael Bentley <mikebentley15@gmail.com>
2021-12-22 12:13:39 +11:00
Jim Mussared e0bf4611c3 py: Only search frozen modules when '.frozen' is found in sys.path.
This changes makemanifest.py & mpy-tool.py to merge string and mpy names
into the same list (now mp_frozen_names).

The various paths for loading a frozen module (mp_find_frozen_module) and
checking existence of a frozen module (mp_frozen_stat) use a common
function that searches this list.

In addition, the frozen lookup will now only take place if the path starts
with ".frozen", which needs to be added to sys.path.

This fixes issues #1804, #2322, #3509, #6419.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-18 00:01:59 +11:00
Jim Mussared f853e3e106 tools/makemanifest.py: Merge make-frozen.py.
Takes the functionality from tools/make-frozen.py, adds support for
multiple frozen directories, and moves it to tools/makemanifest.py.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-17 23:54:05 +11:00
Damien George 598618e8cf tools/makemanifest.py: Make str conversion compatible with Python 2.
Signed-off-by: Damien George <damien@micropython.org>
2021-12-15 16:51:08 +11:00
Christian Decker e9f880482c tools/upip.py: Support == to specify exact package version. 2021-12-15 12:49:14 +11:00
Damien George 3f589e2f39 tools/autobuild: Automatically build all esp32 boards.
Any board with a board.json file will be built.  ESP32-based boards will be
built using the IDF at $IDF_PATH_V42, all other MCU variants (S2, S3, C3)
will be built using the IDF at $IDF_PATH_V44.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-10 21:29:27 +11:00
Damien George 67f66795c0 tools/mpremote: Implement seek and flush in ioctl method.
Fixes issue #8058.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-09 13:35:59 +11:00
Jim Mussared 3770fab334 all: Update Python formatting to latest Black version 21.12b0.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-09 12:09:40 +11:00
Damien George bb7aae557b tools/autobuild: Automatically build all stm32 boards.
Any board with a board.json file will be built.  Additional variants for
certain pyboards will also be built by the explicit build-stm32-extra.sh
script.  Both .dfu and .hex files will be made available.

Also build boards in a sorted order, and don't stop building if a single
board fails.

Signed-off-by: Damien George <damien@micropython.org>
2021-11-22 17:12:16 +11:00
Damien George 90554d03c0 stm32/boards: Build NUCLEO_WB55 and STM32F769DISC without mboot enabled.
This is to make the builds for all nucleo/discovery boards uniform, so they
can be treated the same by the auto build scripts.

The CI script is updated to explicitly enable mboot and packing, to test
these features.

Signed-off-by: Damien George <damien@micropython.org>
2021-11-22 17:12:16 +11:00
iabdalkader 43079aaf86 drivers/ninaw10: Add ublox Nina-W10 WiFi/BT module driver.
- Add WiFi/BT drivers for ublox Nina-W10 (esp32 based) module.
- Add ublox Nina-W10 Python module in extmod.
2021-11-13 23:01:03 +11:00
Damien George 1bd47db688 tools/autobuild: Automatically build all mimxrt, rp2 and samd boards.
Any board with a board.json file will be automatically built.

Signed-off-by: Damien George <damien@micropython.org>
2021-11-05 13:50:08 +11:00
Dave Hylands cb99ca9862 tools/dfu.py: Make tool work with python3 when parsing DFU files. 2021-11-01 15:46:59 +11:00
Jim Mussared ab754d5924 tools/autobuild: Add script to generate website board metadata.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-10-27 14:04:53 +11:00
Damien George 64e4bae129 tools/ci.sh: Use a specific ESP IDF v4.4 commit.
There is no release of IDF v4.4 yet but master is now on v5.0-dev so a
specific commit must be chosen to stick to v4.4.

Signed-off-by: Damien George <damien@micropython.org>
2021-10-25 23:58:44 +11:00
robert-hh dc8be7ccad tools/autobuild: Add the MIMXRT1010_EVK board to autobuild.
Having a board now available for testing, this binary can be provided with
good confidence.
2021-10-25 23:52:23 +11:00
Damien George da4593f937 tools/ci.sh: Use IDF v4.4 as part of esp32 CI and build GENERIC_S3.
IDF v4.4 does not have an official release so for now use the latest
master.  Also remove building GENERIC with no options (all the other boards
are no-option builds), to keep CI time reasonable.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-16 22:59:05 +10:00
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
Philipp Ebensberger 0d7366c912 mimxrt: Rework flash configuration.
- Moves definition of BOARD_FLASH_SIZE and other header files related to
	flash configuration into the Makefile.
- Adds board specific clock_config.h.
- Adds board.h, pin_mux.h, and peripherals.h as they are
	required by NXP MCU SDK in order to use our own clock_config.h.
- Renames board specific FlexSPI configuration files.
- Updates flash frequency of MIMXRT1020_EVK
- Creates separated flash_config files for QSPI NOR and
	QSPI Hyper flash.
- Unifies VFS start address to be @ 1M for 1010 and 1020 boards.
- Unifies 1050EVK boards
- Adds support to both NOR and HyperFlash on boards with
	both capabilities.
- Adds automatic FlexRAM initialization to start-up code based on
	linker script and NXP HAL.
- Applies code formatting to all files in mimxrt port.

With this change the flash configuration is restructured and
organized. This simplifies the configuration process and
provides a better overview of each board's settings. With the integration
of clock_config.h, board.h, pin_mux.h, and peripherals.h we gain better
control of the settings and clock configurations. Furthermore the
implementation of an explicit FlexRAM setup improves the system
performance and allows for performance tuning.

Signed-off-by: Philipp Ebensberger
2021-09-14 13:52:52 +02:00
Patrick Van Oosterwijck 0ec5052f62 tools/autobuild: Add auto build for GENERIC_C3_USB. 2021-09-10 15:41:52 +10:00
Seon Rozenblum 9a7f77bfbc esp32/boards: Add new FeatherS2-Neo board definition. 2021-09-10 15:40:32 +10:00
Damien George 89145c6aad tools/mpremote: Bump version to 0.0.6.
Signed-off-by: Damien George <damien@micropython.org>
2021-09-01 00:49:16 +10:00
David Lechner 86371781e9 tools/uncrustify: Force 1 newline at end of file.
To keep things neat and tidy, we ensure that each file has 1 and only 1
newline at the end of each file.

Signed-off-by: David Lechner <david@pybricks.com>
2021-08-31 13:14:45 +10:00
Damien George 8c4ba575fd tests/basics: Split f-string debug printing to separate file with .exp.
This feature {x=} was introduced in Python 3.8 so needs a separate .exp
file to run on earlier Python versions.

See https://bugs.python.org/issue36817

Signed-off-by: Damien George <damien@micropython.org>
2021-08-26 23:56:02 +10:00
Jim Mussared 145fedef8d tools/pyboard.py: Make --no-soft-reset consistent with other args.
This makes it work like --no-follow and --no-exclusive using a mutex group
and dest.  Although the current implementation with BooleanOptionAction is
neater it requires Python 3.9, so don't use this feature.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-25 15:47:01 +10:00
Jim Mussared 064a145097 tools/pyboard.py: Add --exclusive to match --no-exclusive.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-25 15:46:38 +10:00
Jim Mussared be43164d82 tools/pyboard.py: Make --no-follow use same variable as --follow.
You can set one or the other (or neither) but not both.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-25 15:46:00 +10:00
Jim Mussared 2a290bbfe1 tools/pyboard.py: Move --no-exclusive/--soft-reset out of mutex group.
The --no-exclusive flag was accidentally added to the mutex group in
178198a01d.

The --soft-reset flag was accidentally added to the mutex group in
41adf17830.

These flags can be specified independently to --[no-]follow so should not
be in that mutex group.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-25 15:36:00 +10:00
Damien George ed42002c39 tools/autobuild: Don't use "-B" for make, it's already a fresh build.
And using "-B" means mpy-cross is forcefully rebuilt, sometimes with
invalid CFLAGS_EXTRA options which makes the auto-build fail.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-19 23:30:17 +10:00
Damien George 226c0341ca tools/mpremote: Remove support for pyb.USB_VCP in/out specialisation.
The sys.stdin.buffer and sys.stdout.buffer streams work just as well (and
are just as fast) as pyb.USB_VCP on stm32 devices, so there's no need to
have the USB_VCP specialisation code, which just adds complexity.

Also, on stm32 devices with both USB and UART (or other serial interface),
if something other than the USB_VCP port is used for the serial connection
then mpremote mount will not work because it will default to reading and
writing on USB_VCP instead of the other connected serial stream.

As part of this simplification, support for a second port as input is
removed (this feature was never exposed to the user).

Signed-off-by: Damien George <damien@micropython.org>
2021-08-19 18:56:11 +10:00
Jim Mussared 692d36d779 py: Implement partial PEP-498 (f-string) support.
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 16:58:40 +10:00
Damien George 162bf3c5d8 tools/mpremote: Add "devs" shortcut for "connect list".
See issue #7480.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-14 00:13:35 +10:00
Damien George 1f48934312 tools/mpremote: Fix connect-list in case VID/PID are None.
Which can be the case on Windows and macOS for certain serial devices.

Fixes issue #7636.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-14 00:13:35 +10:00
Patrick Van Oosterwijck e49f609186 tools/autobuild: Add auto build for Silicognition wESP32. 2021-08-13 23:25:24 +10:00
Jim Mussared ee549d725a tools/gen-cpydiff.py: Don't rename foo to ufoo in diff output.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-13 23:14:08 +10:00
Glenn Ruben Bakke 85f0ce214e tools/codeformat.py: Include ports/nrf/modules/nrf in code formatting. 2021-08-08 23:17:55 +10:00
Glenn Ruben Bakke 4f76f66185 tools/ci.sh: Add mpy-cross build to nrf port. 2021-08-08 23:17:55 +10:00
Damien George 96c6b8cae3 ports: Rename USBD_VID/PID config macros to MICROPY_HW_USB_VID/PID.
For consistency with other board-level config macros that begin with
MICROPY_HW_USB.

Also allow boards in the mimxrt, nrf and samd ports to configure these
values.

Signed-off-by: Damien George <damien@micropython.org>
2021-08-07 23:13:55 +10:00
robert-hh 4445c73b11 tools/autobuild: Add the MIMXRT1050_EVKB board to the daily builds. 2021-07-31 16:27:47 +10:00
Tobias Thyrrestrup 60e3e51753 stm32/Makefile: Update to only pull in used Bluetooth library. 2021-07-31 15:11:46 +10:00
Michel Bouwmans 92464f11b0 tools/mpremote: Raise OSError on unsupported RemoteFile.seek.
Signed-off-by: Michel Bouwmans <m.bouwmans@ep-games.eu>
2021-07-23 13:21:53 +10:00
Michel Bouwmans 7870ec0370 tools/mpremote: Add seek whence for mounted files.
Fixes issue #7534.

Signed-off-by: Michel Bouwmans <m.bouwmans@ep-games.eu>
2021-07-23 13:21:42 +10:00
Seon Rozenblum 25159129dd tools/autobuild: Add FeatherS2 and TinyS2 to esp32 auto builds. 2021-07-22 22:42:37 +10:00
Damien George 0256e1ab8b tools/autobuild: Use separate IDF version to build newer esp32 SoCs.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-22 22:42:37 +10:00
Damien George 9f71a11d3f tools/ci.sh: Build GENERIC_C3 board as part of esp32 CI.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-18 23:58:24 +10:00