Commit Graph

1305 Commits

Author SHA1 Message Date
Christian Clauss 4376c969f6 all: Fix Python comparison to None and True, and use "not in".
These are basic PEP8 recommendations.
2023-03-10 13:32:24 +11:00
MicroDev 03bb7ff652
refactor `boards / build` to `ports / board` 2023-03-07 03:22:31 +05:30
MicroDev b17364a447
conditionally build all languages 2023-03-04 15:08:54 +05:30
MicroDev 055c9bfb40
split boards dynamically 2023-03-04 12:10:36 +05:30
Dan Halbert 8c714ec864
Merge pull request #7497 from tannewt/banglejs2
Add Bangle.js 2, JDI memory displays and ACeP epd
2023-02-24 18:20:06 -05:00
MicroDev 0e17deff05
ci fix scheduler 2023-02-24 11:41:41 +05:30
Damien George 5327cd1021 tools/pyboard.py: Use '/' exclusively when dealing with paths.
Currently, certain mpremote filesystem operations can fail on Windows due
to a mixing of '/' and '\' for path separators.  Eg if filesystem_command()
is called with a destination that ends in / then dest.endswith(os.path.sep)
will return False, which gives the wrong behaviour (it does end in a path
separator).

For similar reasons to 7e9a15966a, it's best
to use '/' everywhere in pyboard.py and mpremote, because the target device
understands only '/'.  mpremote already does this, so the remaining place
to fix it is in pyboard.y, to convert all incoming paths to use '/' instead
of '\'.

This effectively reverts 57fd66b80f which
tried to fix the problem in a different way.

See also related 1f84440538.

Signed-off-by: Damien George <damien@micropython.org>
2023-02-24 13:08:34 +11:00
Jim Mussared ce9f7cd00a tools/mpremote: Use hatch to build mpremote package.
This allows the entire configuration to be defined in a single file,
including the logic for including pyboard.py and automatically versioning
based on the git tag.

Building the package works both via `python -m build` as well as
`hatch build`.  `python -m build ` has the advantage of automatically
fetching all dependencies, you don't need to manually install any hatch
packages.

In order to make the versioning work, and also keep things simpler for end
users, mpremote releases will now be the same as MicroPython releases and
use the same tag.  The version strings for mpremote will look like:
- X.Y.Z -- clean build at the tag
- X.Y.Z.postN+gHASH -- clean build, N revisions from the most recent tag
- X.Y.Z.postN+gHASH.dYYYYMMDD -- dirty build, N revisions from out

This commit extends on the idea from #8404.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-02-24 11:40:22 +11:00
Scott Shawcroft f832386cda
Merge remote-tracking branch 'adafruit/main' into banglejs2 2023-02-23 14:43:38 -08:00
MicroDev 60a9c7e5b2
move windows workflow to build ci and more 2023-02-23 14:39:37 +05:30
MicroDev 592fea6822
split `atmel-samd` instead of `raspberrypi` 2023-02-22 20:08:13 +05:30
Damien George e0d1127246 tools/ci.sh: Add unix, stm32 and rp2 ports to code size check.
Now that the code-size-check CI action gives a nice report (as a comment)
on the code size difference, it's possible to have a few more ports
reported there.  In this commit, unix, stm32 and rp2 are added.  Unix
represents non-MCU builds, and stm32 and rp2 represent ARM-based builds,
for ports that have lots of features enabled.

Signed-off-by: Damien George <damien@micropython.org>
2023-02-20 11:26:23 +11:00
MicroDev dc80133d7e
re-enable changes per commit 2023-02-18 16:29:59 +05:30
Damien George 4bd4b6a47e tools/ci.sh: Install pkg-config for macos builds.
Signed-off-by: Damien George <damien@micropython.org>
2023-02-17 13:13:57 +11:00
Scott Shawcroft 931c7c1c51
Add Bangle.js 2, JDI memory displays and ACeP epd
This 2-in-1 PR started with the goal of support the Bangle.js 2
smartwatch with *no USB*.
* Adds "secure" DFU build support with a committed private key.
* Adds 3-bit color support with one dummy bit for the JDI memory display
* Allows nrf boards to have a board_background_task() run in RUN_BACKGROUND_TASK.
  This is needed because the Bangle.js 2 uses the watchdog to reset.
* Renamed port_background_task() to port_background_tick() to indicate it
  runs on tick, not RUN_BACKGROUND_TASK.
* Marks serial connected when the display terminal is inited. This means
  that safe mode messages show up on the display.

ACep, 7-color epaper displays also pack 3 bits in 4. So, I added that
support as well.
* Adds 3-bit ACeP color support for 7-color e-paper displays. (Not
  watch related but similar due to color depth.)
* Allows a refresh sequence instead of a single int command. The 7" ACeP
  display requires a data byte for refresh.
* Adds optional delay after resetting the display. The ACeP displays
  need this. (Probably to load LUTs from flash.)
* Adds a cleaning phase for ACeP displays before the real refresh.

For both:
* Add dither support to Palette.
* Palette no longer converts colors when set. Instead, it caches
  converted colors at each index.
* ColorConverter now caches the last converted color. It should make
  conversions faster for repeated colors (not dithering.)
2023-02-15 15:03:40 -08:00
MicroDev 50c52fca88
split raspberrypi and refactor espressif 2023-02-14 20:38:14 +05:30
MicroDev 65e9d937ac
use python based msgfmt 2023-02-13 11:22:35 +05:30
MicroDev 17e03c4768
ci max parallelism 2023-02-10 21:32:23 +05:30
MicroDev baaa2362c2
use already built mpy-cross 2023-02-10 16:38:25 +05:30
Neradoc 17751ad283 Update build_board_info.py to sh module 2.0.0 2023-02-10 05:23:29 +01:00
MicroDev d9d94eacca
run updated pre-commit 2023-02-01 13:38:41 +05:30
Martin Milata 850f09b109 tools/mpy-tool.py: Initialize line_info_top.
Without it the line number mapping doesn't work.

Signed-off-by: Martin Milata <martin@martinmilata.cz>
2023-02-01 13:17:22 +11:00
MicroDev a6861adb1f
Merge branch 'main' of origin into ESP32-S3-DevKitC-1-N32R8 2023-01-26 11:21:43 +05:30
MicroDev 39b4f26932
update ci 2023-01-21 11:01:09 +05:30
MicroDev 434c2355a0
multiple ci enhancements
- selectively fetch submodule tags
- conditionally build mpy-cross
- remove frozen from submodule cache
2023-01-14 17:10:40 +05:30
MicroDev 6bc0274aec
cache common submodules and share cp version 2023-01-14 10:58:04 +05:30
Jos Verlinde d263438a6e tools/pyboard.py: Set DTR on Windows to avoid ESPxx hard reset.
Fixes issue #9659.

Signed-off-by: Jos Verlinde <Jos.Verlinde@Microsoft.com>
2023-01-13 16:51:31 +11:00
Jim Mussared aa64280666 tools/pyboard.py: Add fs_{listdir,readfile,writefile,stat}.
These are for working with the filesystem when using pyboard.py as a
library, rather than at the command line.

- fs_listdir returns a list of tuples, in the same format as os.ilistdir().
- fs_readfile returns the contents of a file as a bytes object.
- fs_writefile allows writing a bytes object to a file.
- fs_stat returns an os.statresult.

All raise FileNotFoundError (or OSError(ENOENT) on Python 2) if the file is
not found (or PyboardError on other errors).

Updated fs_cp and fs_get to use fs_stat to compute file size.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-01-13 16:38:34 +11:00
Jim Mussared 6013d27dd5 tools/pyboard.py: Add parse kwarg to eval.
This is useful when using pyboard.py as a library rather than at the
command line.

    pyb.eval("1+1") --> b"2"
    pyb.eval("{'a': '\x00'}") --> b"{'a': '\\x00'}"

Now you can also do

    pyb.eval("1+1", parse=True) --> 2
    pyb.eval("{'a': '\x00'}", parse=True) --> {'a': '\x00'}

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-01-13 16:38:34 +11:00
Jim Mussared 77002a92bf tools/pyboard.py: Fix Python 2 compatibility.
In Python 2, serial.read()[0] a string, not int. Use struct.unpack to do
this instead.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-01-13 16:38:34 +11:00
MicroDev 4a9d934008
escape `.` and `/` in regex 2023-01-11 20:56:53 +05:30
MicroDev 645499e819
check `//|` only after a file matches the doc pattern 2023-01-11 11:52:19 +05:30
MicroDev c94f83f2e6
fix pagination and some refactoring 2023-01-09 18:17:51 +05:30
MicroDev 565bbd5002
fix path in `set_docs_to_build` 2023-01-08 13:23:51 +05:30
MicroDev 70273316f8
run pre-commit formatting 2023-01-07 19:00:54 +05:30
MicroDev 8096c94e1b
build docs only when `//|` matches 2023-01-07 18:17:48 +05:30
MicroDev 0e46c77c14
fix for bug `fatal: error in object: unshallow` 2023-01-07 17:52:33 +05:30
MicroDev 03d4d63ab5
simplify fetching changed files 2023-01-07 17:48:39 +05:30
MicroDev da31232cdf
refactor variable naming 2023-01-07 11:50:31 +05:30
Bernhard Bablok 868a98323a added comment for IGNORE_DIRS 2022-12-21 15:18:33 +01:00
Bernhard Bablok f97925c66c ignore .devcontainer/* 2022-12-21 13:27:23 +01:00
robert-hh 57bb1e0474 tools/autobuild: Create .uf2 file for seeed_xiao_nrf52.
And for all other nrf boards that have or get a uf2 build tag.
2022-12-19 16:16:12 +11:00
Jeff Epler 41beca14af
only import a module if we use something from it
This prevents a bogus line such as `from types import ` from being
generated.

surprisingly this was not detected as a problem until a change
in isort! Formerly it would remove the bad line, but now it (correctly!)
errors
2022-12-13 16:04:07 -06:00
stijn 9c7ff87643 all: Keep msvc build output in build/ directories.
This follow the change made for Makefile-based projects in b2e82402.
2022-12-13 17:18:53 +11:00
Dan Halbert 845b68c351
Merge pull request #7204 from jepler/build-python3x
return to using python 3.x for builds
2022-11-30 15:18:29 -05:00
Scott Shawcroft ad2d190507
Merge pull request #7247 from tannewt/picow_web_workflow
Enable* web workflow for Pico W
2022-11-28 14:19:42 -08:00
Michael Mogenson 921f397acb tools/mpremote: Only auto connect to serial device with USB VID/PID.
On MacOS and Windows there are a few default serial devices that are
returned by `serial.tools.list_ports.comports()`. For example on MacOS:

```
{'description': 'n/a',
 'device': '/dev/cu.Bluetooth-Incoming-Port',
 'hwid': 'n/a',
 'interface': None,
 'location': None,
 'manufacturer': None,
 'name': 'cu.Bluetooth-Incoming-Port',
 'pid': None,
 'product': None,
 'serial_number': None,
 'vid': None}

{'description': 'n/a',
 'device': '/dev/cu.wlan-debug',
 'hwid': 'n/a',
 'interface': None,
 'location': None,
 'manufacturer': None,
 'name': 'cu.wlan-debug',
 'pid': None,
 'product': None,
 'serial_number': None,
 'vid': None}
```

Users of mpremote most likely do not want to connect to these ports. It
would be desirable if mpremote did not select this ports when using the
auto connect behavior. These serial ports do not have USB VID or PID
values and serial ports for Micropython boards with FTDI/serial-to-USB
adapter or native USB CDC/ACM support do.

Check for the presence of a USB VID / PID int value when selecting a
serial port to auto connect to. All serial ports will still be listed by
the `list` command and can still be selected by name when connecting.

Signed-off-by: Michael Mogenson <michael.mogenson@gmail.com>
2022-11-25 17:20:14 -05:00
Tobias Thyrrestrup 2fcd93cdd0 tools/mpremote: Allow EDITOR environment variable to work on Windows. 2022-11-25 15:02:52 +11:00
Damien George d75f49c0f0 tools/ci.sh: Don't print python2 version.
Because python2 may not be available.

Signed-off-by: Damien George <damien@micropython.org>
2022-11-23 10:32:55 +11:00
Scott Shawcroft 167a50658a
Try and fix ci change script 2022-11-22 09:42:59 -08:00
MicroDev 7ea563e661
fix json parse issue 2022-11-19 23:29:05 +05:30
MicroDev 4f2f489df4
ignore espressif_esp32s3_devkitc_1_n32r8 in vid:pid check 2022-11-18 09:54:40 +05:30
Scott Shawcroft c525322a29
Merge pull request #7132 from MicroDev1/ci
CI: Schedule PR jobs based on commit specific changes
2022-11-17 10:37:07 -08:00
Dominic Davis-Foster ca84d79809
Allow duplicate VID and PID in CI check. 2022-11-15 23:06:36 +00:00
Damien George 3927ce6035 tools/ci.sh: Remove one build from zephyr CI to speed it up.
The zephyr CI takes the most time out of all CI jobs, so remove the
standard qemu_x86 build to speed it up.  The remaining builds should still
cover enough cases to catch errors.

Signed-off-by: Damien George <damien@micropython.org>
2022-11-16 10:05:17 +11:00
iabdalkader 3bae10417e tools/ci.sh: Add ARDUINO_NANO_RP2040_CONNECT to rp2 CI build. 2022-11-16 09:47:06 +11:00
Jim Mussared 1504204606 tools/gendoc.py: Remove unused script.
This used to be used to generate .rst docs from inline comments in the C
code (specifically for APIs) but is now unused.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-11-15 23:17:26 +11:00
Andrew Scheller 02ad71468f tools/gen-cpydiff.py: Use os.path.join and os.path.isdir.
Makes path handling clearer and simpler.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-11-15 23:17:26 +11:00
Jeff Epler dec802a73b
build docs when .github/workflows changes
Closes: #7205
2022-11-13 13:26:46 -06:00
Angus Gratton 2e6e53057b shared/tinyusb: Further refactor static USB device implementation.
App the mp_ prefix to usbd_ symbols and files which are defined here and
not in TinyUSB.

rp2 only for now. This includes some groundwork for dynamic USB devices
(defined in Python).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-11-11 16:47:36 +11:00
Damien George b52fe52d31 tools/makemanifest.py: Fix printing of mpy_cross.CrossCompileError.
This change was missed by 5852fd7708

Signed-off-by: Damien George <damien@micropython.org>
2022-11-02 14:06:18 +11:00
Damien George 3ed017677b tools/mpremote: Make RemoteFile objects iterable.
So that filesystems mounted with "mpremote mount" can have their files
iterated over, making them consistent with other files.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-31 12:40:17 +11:00
Jim Mussared 3427e12e8f tools/mpremote: Handle FileNotFound exceptions in listdir.
The except handler for OSError didn't include the line that actually calls
os.listdir, so an invalid path wasn't handled correctly.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-31 12:38:25 +11:00
microDev 30f07fbf22
schedule pr jobs based on commit specific changes 2022-10-26 23:19:13 +05:30
Jim Mussared 1ba0e8ff96 py/persistentcode: Only emit sub-version if generated code has native.
In order for v1.19.1 to load a .mpy, the formerly-feature-flags which are
now used for the sub-version must be zero.

The sub-version is only used to indicate a native version change, so it
should be zero when emitting bytecode-only .mpy files.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-25 14:57:04 +11:00
Jeff Epler d08b43f704
We refer to port multiple times, make it convenient 2022-10-14 11:33:04 -05:00
Jeff Epler 641a398a54
only build subset of boards for bindings changes 2022-10-14 11:33:03 -05:00
Jeff Epler 3b600ac9b3
Potentially compute settings of fewer boards
.. when the file is within ports, just get the settings for boards
within the port
2022-10-14 11:33:03 -05:00
Jeff Epler ad130e87f0
Parallelize finding board settings
This reduces the _elapsed_ time running the script from ~90s to ~15s
on my AMD Ryzen 7 5700U. The CPU time is still around 2 minutes.
2022-10-14 11:33:03 -05:00
Jeff Epler cab4063057
Make it easier to locally test ci_set_matrix
Now you can e.g., `tools/ci_set_matrix.py ports/raspberrypi/mpconfigport.h`
and see what outputs would be set.
2022-10-14 11:33:02 -05:00
Jeff Epler 81154b4ee6
Revert "don't build other ports due to common-hal changes"
This reverts commit 91985cef7e.
2022-10-14 11:32:58 -05:00
Dan Halbert 91985cef7e don't build other ports due to common-hal changes 2022-10-14 11:26:37 -04:00
Jeff Epler a31ad57225
Use non-deprecated way to set build outputs
see https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-10-12 13:45:30 -05:00
Jim Mussared 87011f6353 extmod/extmod.mk: Make extmod.mk handle GIT_SUBMODULES.
This applies to nimble, btstack, axtls, mbedtls, lwip.

Rather than having the ports individually manage GIT_SUBMODULES for these
components, make extmod.mk append them when the relevant feature is
enabled.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:29:09 +11:00
Dan Halbert e0517c7379
Merge pull request #6999 from jepler/picow-ssl
pico_w: implement ssl with caveats
2022-10-06 10:51:36 -04:00
Jim Mussared b8982ec5f9 tools/verifygitlog.py: Add additional help for subject line issues.
This check used to just show the regular expression that failed to match,
but the rules are pretty subtle and hard to interpret from the regular
expression alone.

Add some basic checks for the main things that go wrong:
 - Missing capitalisation.
 - Missing full-stop.
 - Missing path.
 - Single-word subject.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-07 00:26:31 +11:00
Jim Mussared f6d06b3ce0 tools/verifygitlog.py: Ignore comment lines in commit messages.
The "signed-off" check assumes that the Signed-off-by: line is the last,
but there may me many lines of comments after this.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-07 00:19:40 +11:00
Jeff Epler ecd140213b
populate nina-fw submodule whe needed 2022-10-06 08:14:50 -05:00
Dan Halbert 3d91ea444a run black on tools/ci_check_duplicate_usb_vid_pid.py 2022-10-05 17:37:33 -04:00
Jeff Epler 905a5957b9
allow gen_crt_bundle to generate a .S file directly
.. this gets rid of one of the steps of converting it
2022-10-05 14:56:25 -05:00
Jeff Epler b47d1d777b
black and fix doc typo 2022-10-05 14:56:24 -05:00
Jeff Epler b1ce1d05d7
Copy from esp-idf@d51f7d8821 2022-10-05 14:56:24 -05:00
Jeff Epler dcb650c513 pico w: add ssl module
Note: at this time, the ssl module on pico_w never verifies the server
certificate. This means it does not actually provide a higher security
level than regular socket / http protocols.
2022-10-05 13:12:43 -04:00
Damien George 46d02c2469 tools/mpremote: Bump version to 0.4.0.
Signed-off-by: Damien George <damien@micropython.org>
2022-10-06 01:02:39 +11:00
Jim Mussared f13134e403 tools/mpremote: Fix argument handling for follow and add help strings.
Fixes in this commit are:
- Make --follow the default for "run" (accidentally changed in 68d094358).
- Add help strings for "repl": --capture --inject-file --inject-code
- Update help strings for "run".
- Fix encoding for --inject-code (accidentally broken in 68d094358).
- Remove ability to --no-follow for "eval". It was there previously because
  it shared the same code path with "exec" and "run", but makes no sense
  for "eval", so might as well remove.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-06 00:58:31 +11:00
Kyle Mohr b417eacc5e
Update ci_check_duplicate_usb_vid_pid.py 2022-10-04 00:03:38 -05:00
Kyle Mohr ba2368c59e
Utilize VID/PID from native USB port 2022-10-03 23:46:23 -05:00
Angus Gratton 0e35c4de9b tools: Add pre-commit support.
Tweak the existing codeformat.py and verifygitlog.py to allow them to be
easily called by pre-commit.

(This turned out to be easier than using any existing pre-commit hooks,
without making subtle changes in the formatting.)

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-10-04 14:52:48 +11:00
Kyle Mohr a948dc399b add esp32-s2-devkitc-1-n8r2 variant 2022-10-04 03:51:51 +00:00
Angus Gratton bdac8272d8 tools: Add note about uncrustify versions.
Uncrustify versions are not mutually compatible:

1. Version 0.73 or newer produce slightly different formatting. It may be
possible to tweak these by adding more config items, but this will cause
older versions to error out with 'Unknown option'.

2. Version 0.75 prints a range of deprecation warnings due to config file
changes, and returns a non-zero exit code. These are actually fixable
as most are the default value, and pp_indent has changed from 'true' to '1'
which is backwards compatible. However issue 1 remains, so probably better
to have it fail explicitly.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-10-04 14:34:54 +11:00
Jim Mussared 7705b9b9d5 tools/pyboard.py: Handle unsupported fs command.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-01 22:44:24 +10:00
Jim Mussared 12ca918eb2 tools/mpremote: Add `mpremote mip install` to install packages.
This supports the same package sources as the new `mip` tool.
 - micropython-lib (by name)
 - http(s) & github packages with json description
 - directly downloading a .py/.mpy file

The version is specified with an optional `@version` on the end of the
package name. The target dir, index, and mpy/no-mpy can be set through
command line args.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-01 22:44:24 +10:00
Jim Mussared 68d094358e tools/mpremote: Use argparse for command line parsing.
No functional change other than to allow slightly more flexibility in how
--foo arguments are specified.

This removes all custom handling for --foo args in all commands and
replaces it with per-command argparse configs.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-01 22:44:24 +10:00
Jim Mussared 413a69b94b tools/mpremote: Simplify dispatch of commands.
No functional change.

This makes each built-in command defined by just a handler method and
simplifies a lot of the logic around tracking the board state.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-01 22:44:24 +10:00
Jeff Epler 907c5d387f
Tweak black_bindings
Originally, black_bindings found each contiguous "//|" block and sent
it to black independently. This was slower than it needed to be.

Instead, swap the comment prefix: when running black, take off
"//|" prefixes and put "##|" prefixes on all un-prefixed lines.
Then, after black is run, do the opposite operation

This more than doubles the overall speed of "pre-commit run --all",
from 3m20s to 55s CPU time on my local machine (32.5s to under 10s
"elapsed" time)

It also causes a small amount of churn in the bindings, because
black now sees enough context to know whether one 'def' follows another
or ends the 'def's in a 'class'. In the latter case, it adds an extra
newline, which becomes a "//|" line.

I'm less sure why a trailing comma was omitted before down in
rp2pio/StateMachine.c but let's roll with it.
2022-09-30 11:18:13 -05:00
Jim Mussared 282401da5c tools/manifestfile.py: Replace recursive glob with os.walk.
Recursive glob isn't supported before Python 3.5.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-30 17:34:03 +10:00
Jim Mussared 924a3e03ec top: Replace upip with mip everywhere.
Updates all README.md and docs, and manifests to `require("mip")`.

Also extend and improve the documentation on freezing and packaging.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-30 17:34:03 +10:00
Jim Mussared 65ab0ec91c tools/manifestfile.py: Add `author` kwarg to metadata().
This allows future micropython-lib packages to specify an author.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-29 23:53:35 +10:00
Dan Halbert db065a299f
Merge pull request #6933 from jepler/🥧🐮
Implement a useful subset of `wifi` and `socketpool` modules on 🥧🐮
2022-09-28 18:09:24 -04:00
Jeff Epler 6c3cdceb45
Implement scan, connect, ping
My pings go out, and then they come back

```py
import os
import wifi
import ipaddress

wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
ipv4 = ipaddress.ip_address("8.8.4.4")
print("Ping google.com: %f ms" % (wifi.radio.ping(ipv4)*1000))
```
2022-09-28 10:06:33 -05:00
Jeff Epler 3a0d237581
Run black_bindings during pre-commit 2022-09-27 15:22:55 -05:00
Jeff Epler 6a20aef382
new script to blacken docstrings in bindings 2022-09-27 15:20:49 -05:00
Jeff Epler a769beed3f
Use new unique ID for this board 2022-09-22 16:23:38 -05:00
Jim Mussared 920da9c5e3 unix/variants/coverage: Add test for manifest freeze_mpy().
This uses the frozentest.mpy that is also used by ports/minimal.

Also fixes two bugs that these new tests picked up:
 - File extension matching in manifestfile.py.
 - Handling of freeze_mpy results in makemanifest.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:51:10 +10:00
Jim Mussared 6ecdf1a240 tests/frozen: Move frozentest.mpy from ports/ to tests/.
frozentest.mpy was previously duplicated in ports/minimal and
ports/powerpc.

This needs to be re-generated on every .mpy version increase, so might as
well just have a single copy of it.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:51:10 +10:00
Jim Mussared d94141e147 py/persistentcode: Introduce .mpy sub-version.
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.

With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version.  A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION.  This allows 3
additional updates to the native ABI per bytecode revision.

The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-09-19 23:19:55 +10:00
Jim Mussared c1530a0ce8 unix: Refactor mpconfigport.h and mpconfigvariant.h.
This is a no-op for coverage and minimal.

The standard and dev variants have been merged and enable the same feature
set as a typical bare-metal board.  And remove the CI for the dev build.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:39:03 +10:00
Wind-stormger 57fd66b80f tools/pyboard.py: Support Windows pathname separators.
Addresses issue #9132.
2022-09-13 13:28:39 +10:00
Andrew Leech 4e0964b59f extmod/vfs: Add finaliser to ilistdir to close directory handle.
When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally.  Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.

If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed.  In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.

Fixes issues #6568 and #8506.
2022-09-13 13:00:42 +10:00
Jim Mussared 4903e48e34 tools/makemanifest.py: Force the repo version of the mpy_cross package.
In case the version from pypi is installed or some other version is
available in sys.path, prepend `$(TOP)/mpy-cross` to sys.path instead.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-09 18:12:55 +10:00
Dan Halbert dd37c81136
Merge pull request #6847 from dhalbert/status-bar-control
Allow enabling and disabling of status bar
2022-09-08 10:48:36 -04:00
Jim Mussared 9070a24940 tools/autobuild: Use distinct directory for building stm32 variants.
Previous the build directory just used the board name, now make it use the
variant name too.

This shouldn't have any change because the existing directory should not
exist (all builds run by these scripts remove their build directory after
completion), but it makes debugging easier.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-08 14:54:11 +10:00
Jim Mussared 5852fd7708 tools/manifestfile.py: Allow manifests to set metadata.
The metadata can be version, description, and license.

After executing a manifest, the top-level metadata can be queried, and also
each file output from the manifest will have the metadata of the
containing manifest.

Use the version metadata to "tag" files before freezing such that they have
__version__ available.
2022-09-05 17:07:13 +10:00
Jim Mussared bc23f207ce tools/manifestfile.py: Allow require() to specify unix packages.
By default, don't include micropython-lib/unix-ffi in the search.

If unix_ffi=True is passed to require(), then include unix-ffi and make it
take precedence over the other locations (e.g. python-stdlib).

This does two things:
 - Prevents non-unix builds from using unix-only packages.
 - Allows the unix build to optionally use a more full-featured (e.g. ffi)
   based package, even with the same name as one from e.g. stdlib.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:06:52 +10:00
Jim Mussared e9a28ce312 tools/manifestfile.py: Allow include of directory path.
If an include path is a directory, then it implicitly grabs the manifest.py
file inside that directory. This simplifies most manifest.py files.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:06:52 +10:00
Jim Mussared 6bd0ec7a70 tools/makemanifest.py: Update to use mpy_cross module.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:06:43 +10:00
Jim Mussared f3cdb052db tools/manifestfile.py: Add library for working with manifests.
This splits the manifest file loading logic from makemanifest.py and
updates makemanifest.py to use it.

This will allow non-freezing uses of manifests, such as defining packages
and dependencies in micropython-lib.

Also adds additional methods to the manifest "API":
 - require() - to get a package from micropython-lib.
 - module() - to define a single-file module
 - package() - to define a multi-file package

module() and package() should replace most uses of freeze() and can also
be also used in non-freezing scenarios.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:00:43 +10:00
Dan Halbert 52080e24eb status bar control 2022-08-30 15:23:44 -04:00
Damien George 730e975091 esp32/boards: Merge manifest_release modules into standard manifest.
Having two separate manifests is confusing.  It's simpler to have the daily
builds use the same configuration as the stable, release builds.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-30 11:34:34 +10:00
Jeff Epler cb202ea344
Check uniqueness of creatorid:creationid
Turns out we already have one duplicate, so I 'endorsed' it, but probably
one should be reassigned.

Closes: #8628
2022-08-27 21:32:58 -05:00
Damien George 2e386bcf76 tools/mpremote: Print nicer errors for unsupported 'cp -r' arguments.
Also document support for 'cp :a :b'.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George f5fedf4676 tools/pyboard.py: Add fs_cp function for direct device-to-device copy.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George 24f1161fe2 tools/pyboard.py: Remove implicit fs_put if source starts with ./.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George 858707181d tools/mpremote: Print a nicer error when a filesystem command fails.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George 1855df6361 stm32: Remove support for CC3000 WiFi driver.
It has been about 8 years since support for this chip was added.  Reasons
to remove it are:
- It is no longer easy to obtain this part.
- There are now many other options for WiFi.
- It's not a good use of developer time to maintain it.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 12:51:37 +10:00
Jim Mussared 492ba5eaf2 stm32: Move board variant config to mpconfigboard.mk.
Rather than having the autobuild know about the particular variants, have
the mpconfigboard.mk describe them and make autobuild discover them
automatically.

Adds a "query-variants" target to stm32/Makefile to allow the set of
possible variants to be queried.

Removes pybv3 from the autobuild as this isn't use by the downloads page.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 09:38:38 +10:00
robert-hh f6ec01d1da tools/ci.sh: Split the stm32 builds for wiznet5k and cc3k.
- Add lib/wiznet5k into the 'make submodules' step.
- Split the stm32 builds for wiznet5k and cc3k.
- Run 'make .... clean' after making the wiznet5k build.
2022-08-23 15:00:00 +10:00
Nicholas H.Tollervey af54d2ce9f
javascript: Rename this port to 'webassembly'. 2022-08-22 12:03:39 +01:00
Jim Mussared 59e3348c10 tools/mpremote: Add "edit" command.
This allows a remote file to be edited locally by copying it over, running
the local editor, then copying it back.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:29:13 +10:00
Jim Mussared 263737ecfe tools/pyboard.py: Add "touch" filesystem command.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:24:25 +10:00
Jim Mussared a311e9e3d4 tools/mpremote: Allow + terminator for fs commands.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:22:36 +10:00
Damien George cbc9f944c4 tests,tools: Update path to unix micropython executable.
These were missed by 47c84286e8

Signed-off-by: Damien George <damien@micropython.org>
2022-08-18 11:47:58 +10:00
Scott Shawcroft 1ae88c3131
Merge remote-tracking branch 'adafruit/main' into design_guide_update 2022-08-11 10:52:06 -07:00
Damien George d53c3b6ade unix/variants: Remove variant suffix from executable filename.
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-11 13:34:34 +10:00
Daniel Jour 409995ac68 tools/ci.sh: Force mpy-cross build for samd and teensy. 2022-08-11 13:33:51 +10:00
Daniel Jour 47c84286e8 all: Fix paths to mpy-cross and micropython binaries.
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory.  Thus
some paths had to be adjusted.
2022-08-11 13:31:13 +10:00
Scott Shawcroft 9c6c8b5d62
Don't build boards for docs changes 2022-08-10 16:31:50 -07:00
Dan Ellis 9b5e00fcc5 py/formatfloat: Format all whole-number floats exactly.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place.  This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point.  The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.

Fixes issue #4212.

Signed-off-by: Dan Ellis dan.ellis@gmail.com
2022-08-09 13:43:47 -04:00
Scott Shawcroft 167108a91f
Fix builds with displayio but not terminalio 2022-08-04 16:34:25 -07:00
Scott Shawcroft 3a2bcbc5c7
Enable title bar on all builds
* Tweak scroll area position so last line is complete and top is
  under the title bar.
* Pick Blinka size based on the font to minimize unused space in
  title bar. Related to #2791
* Update the title bar after terminal is started. Fixes #6078

Fixes #6668
2022-08-04 16:33:10 -07:00
Jim Mussared 58bed5ec14 tools/ci.sh: Initialise submodules for more ports.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-03 00:08:28 +10:00
Scott Shawcroft 51006f5eeb
Merge branch 'main' into webflow_edit_page 2022-07-29 10:03:07 -07:00
Scott Shawcroft bb734c75d7
Improve build matrix computation
If a changed file is in frozen, supervisor, shared-bindings,
shared-module or common-hal, then compute board settings to
determine if it is impacted.

This should reduce the boards built for changes that are in those
directories.
2022-07-26 15:33:13 -07:00
Neradoc f472996e80 move get_board_mapping to shared_binding_matrix because it just uses a list from there
uppercase some of the configuration lists and dicts
properly split the extensions into a list
2022-07-26 18:15:59 +02:00
Neradoc eabe8b971a list extensions instead of macros names ("bin,uf2" not BIN_UF2)
the modules_support_matrix usees a dictionnary per board instead of a list
optionally include the frozen modules URLs in it
2022-07-26 18:15:59 +02:00
Neradoc d021d9ae4a Move build extensions to board directories
- define CIRCUITPY_BUILD_EXTENSIONS to predefined values
- set CIRCUITPY_BUILD_EXTENSIONS in port and board config
- reuse the support matrix "get_settings_from_makefile" to get it
- move the existing port and board specific values
- remove the C3 specific board values because it's not the default
- update build_release_files.py to use get_settings_from_makefile
2022-07-26 18:15:50 +02:00
Dan Ellis f9cbe6bc47 py/formatfloat: Format all whole-number floats exactly.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place.  This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point.  The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.

Fixes issue #4212.

Signed-off-by: Dan Ellis dan.ellis@gmail.com
2022-07-26 22:23:47 +10:00
foamyguy d0782b60cb Merge branch 'main' into webflow_edit_page
# Conflicts:
#	supervisor/shared/web_workflow/web_workflow.c
2022-07-25 19:20:05 -05:00
Scott Shawcroft 47f718aa92
Merge pull request #6615 from askpatrickw/beetle-esp32-c3
Beetle esp32 c3
2022-07-25 15:00:05 -07:00
Patrick 71c22232f0 board config compelted 2022-07-17 11:48:45 -07:00
foamyguy 584e27139e add back /edit/ handler and specify ES6 quote chars for jsmin 2022-07-15 19:49:03 -05:00
Scott Shawcroft bad080a7bb
Merge pull request #6588 from jepler/signal-errors-ci-fetch-deps
signal errors in ci_fetch_deps subprocesses
2022-07-15 11:17:59 -07:00
Jeff Epler 8cdbeb62f6
Pass check=False for an expected-fail command 2022-07-14 15:16:28 -05:00
Scott Shawcroft ac460dd1e1
Merge branch 'main' into esp32 2022-07-13 15:30:53 -07:00
Jeff Epler 104ac5b965
signal errors in ci_fetch_deps subprocesses
A recent build failed. The original error seemed to be during ci_fetch_deps
where a build message said
```
  fatal: reference is not a tree: 346c936e14c6ea3a8d3d65cb1fa46202dc92999d
  fatal: Unable to checkout '346c936e14c6ea3a8d3d65cb1fa46202dc92999d' in submodule path 'extmod/ulab'
```
(along with other problems), but this step didn't signal failure to
github actions.

By adding the check= parameter, a failure of the subprocess will cause
a CalledProcessError to be raised, which will make ci_fetch_deps exit with
nonzero status. In turn, this should let Actions understand that something
went wrong with this step, instead of waiting for some subsequent step(s)
to go wrong.
2022-07-13 14:56:22 -05:00
iabdalkader 2076f2efcc tools/autobuild: Add nrf port to autobuild scripts. 2022-07-13 16:19:22 +10:00
Scott Shawcroft 23bbb19138
Merge remote-tracking branch 'adafruit/main' into websocket_serial 2022-07-12 13:15:38 -07:00
Scott Shawcroft 15fe386457
Fix build and minify html and js 2022-07-12 11:13:17 -07:00
Dan Halbert fa62c87536 ESP32 .bin only; update nvm.toml 2022-07-08 20:37:47 -04:00
Dan Halbert 2c7ad7a39f make translate and pre-commit formatting fixes 2022-07-08 16:53:53 -04:00
Dan Halbert a9d4f09360 Update ci_check_duplicate_usb_vid_pid.py for ESP32 2022-07-08 16:11:01 -04:00
Damien George 80bc9b3dee tools/mpremote: Don't be verbose when using cat command.
Fixes issue #8828.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-08 22:28:26 +10:00
Damien George 7e5137e0ae tools/pyboard.py: Add verbose option to filesystem_command.
Signed-off-by: Damien George <damien@micropython.org>
2022-07-08 22:26:41 +10:00
Patrick 03c1871d6b Folders have underscores not dashes 2022-07-01 14:50:46 -07:00
Patrick f95edd93a0 C3 boards are Bin only 2022-07-01 14:50:46 -07:00
Scott Shawcroft 3cd05291d0
Static files + welcome page 2022-06-27 13:39:09 -07:00
Scott Shawcroft 7543dd9af0
CORS works and autoreload too 2022-06-27 13:39:08 -07:00
Scott Shawcroft eeb9b7881e
Disable/enable buttons 2022-06-27 13:39:08 -07:00
Scott Shawcroft d6eaaafee8
Basic JS 2022-06-27 13:39:08 -07:00
Matthew McGowan d03bba3adb feat(swan_r5): Enable UF2 support for Swan R5 board 2022-06-21 14:47:33 -07:00
David Lechner 17f0297c1a tools/ci.sh: Drop ppa requirement for code formatting CI.
The CI scripts were using a PPA to get a backported version of uncrustify
on Ubuntu 20.04.  However, this causes CI to intermittently fail due to
connection issues to launchpad.net or the key server.

Ubuntu 22.04 has a newer version of uncrustify removing the need for the
PPA.  Ubuntu 22.04 is now in beta on GitHub actions, so it can be used.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-17 16:56:41 +10:00
Maureen Helm bada8b3cad zephyr: Upgrade to Zephyr v3.1.0.
Updates the Zephyr port build instructions and CI to use the latest Zephyr
release tag.

Tested on frdm_k64f.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2022-06-17 16:51:34 +10:00
Damien George dea1ba3309 tools/metrics.py: Add mimxrt and renesas-ra to list of available ports.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-17 14:57:58 +10:00
Scott Shawcroft 4e70781c45
Remove extra glyph padding 2022-06-13 09:21:41 -07:00
Damien George 1f1afae622 examples/natmod/features3: Add example to test more natmod features.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Damien George 17a0d65ee4 tools/mpy_ld.py: Support GOT entries that reference inside mp_fun_table.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Scott Shawcroft 6446010753
Wi-Fi autoconnect and title bar status
This adds support for CIRCUITPY_WIFI_SSID and CIRCUITPY_WIFI_PASSWORD
in `/.env`. When both are defined, CircuitPython will attempt to
connect to the network even when user code isn't running. If the
user code attempts to a network with the same SSID, it will return
immediately. Connecting to another SSID will disconnect from the
auto-connected network. If the user code initiates the connection,
then it will be shutdown after user code exits. (Should match <8
behavior.)

This PR also reworks the default displayio terminal. It now supports
a title bar TileGrid in addition to the (newly renamed) scroll area.
The default title bar is the top row of the display and is positioned
to the right of the Blinka logo when it is enabled. The scroll area
is now below the Blinka logo.

The Wi-Fi auto-connect code now uses the title bar to show its
state including the IP address when connected. It does this through
the "standard" OSC control sequence `ESC ] 0 ; <s> ESC \` where <s>
is the title bar string. This is commonly supported by terminals
so it should work over USB and UART as well.

Related to #6174
2022-06-09 14:55:54 -07:00
Damien George 313f082896 tools/ci.sh: Build native .mpy examples for armv7emsp, xtensawin in CI.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 13:24:08 +10:00
Damien George abb3850398 tools/mpy_ld.py: Support R_XTENSA_PDIFF32 relocation.
Newer versions of the ESP-IDF's toolchain use this relocation.

Fixes issue #8436.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 11:57:22 +10:00
Damien George 6a1dbaad1c tools/ci.sh: Build NUCLEO_H743ZI with -O2 optimisation level.
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-08 15:00:59 +10:00
Damien George 5290bfaefe tools/mpremote: Bump version to 0.3.0.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:23:14 +10:00
Damien George 646fcdadbf tools/mpremote: Add command to print the version.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:22:04 +10:00
Damien George 2fb413b265 tools/mpy-tool.py: Improve generated frozen identifiers.
Frozen identifiers now include their full name hierarchy, eg their class
name.  This makes it easier to understand the generated code.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George 9670a156da all: Rename MICROPY_PY_WIZNET5K to MICROPY_PY_NETWORK_WIZNET5K.
To match MICROPY_PY_NETWORK_CYW43 and MICROPY_PY_NETWORK_NINAW10.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George bf92b0cbf2 tools/ci.sh: Add test for mpy-tool's merging feature.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:51:45 +10:00
Damien George 599a22e569 tools/mpy-tool.py: Rework .mpy merging feature.
Now that the native qstr link table is gone, merging a native .mpy file
with a bytecode .mpy file is not as simple as concatenating the .mpy data.
The qstr_table and obj_table tables from all merged .mpy files must now be
joined together, because they are global to the .mpy file (and hence global
to the merged .mpy file).  This means the bytecode needs to be be decoded,
qstr_table and obj_table indices updated to point to the correct entries in
the new tables, and then the bytecode re-encoded.

This commit makes this change to the merging feature in mpy-tool.py.  This
can now merge an arbitrary number of bytecode .mpy files, and up to one
native .mpy file.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:51:45 +10:00
Damien George f506bf342a py/bc: Remove unused mp_opcode_format function.
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:32:38 +10:00
Damien George b37b578214 py/persistentcode: Remove remaining native qstr linking support.
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table.  The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py.  But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.

This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table.  mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:19:55 +10:00
Nathan Young 761e66e85b Enable UF2 builds for STM32 Thing Plus 2022-06-04 15:21:06 +02:00
Andrew Leech b92e51efe7 tools/ci.sh: Add build of W5100S_EVB_PICO board to rp2 CI.
This exercises the inclusion of a number of libraries in the rp2 port
including mbedtls and lwip.

Signed-off-by: Andrew Leech <andrew@alelec.net>
2022-06-03 14:35:52 +10:00
Damien George 1d047617bb tools/mpy-tool.py: Remove obsolete unicode flag in .mpy header.
This was removed in c49d5207e9

Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 11:43:46 +10:00
David Lechner d42d35f56d tests/run-tests.py: Enable `-X realtime` option for macOS tests.
This enables the new `-X realtime` runtime option when running tests on
macOS.  This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers.  This
makes tests that depend on the passage of time more likely to succeed.

CI tests that were disabled because of this are now enabled again.

Signed-off-by: David Lechner <david@pybricks.com>
2022-05-24 00:52:44 +10:00
Damien George c1b9d2259e py/dynruntime.mk: Add basic support for armv6m architecture.
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms.  More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 23:01:25 +10:00
Damien George d4d53e9e11 py/emitnative: Access qstr values using indirection table qstr_table.
This changes the native emitter to access qstr values using the qstr
indirection table qstr_table, but only when generating native code that
will be saved to a .mpy file.  This makes the resulting native code fully
static, ie it does not require any fix-ups or rewriting when it is
imported.

The performance of native code is more or less unchanged.  Benchmark
results on PYBv1.0 (using --via-mpy and --emit native) are:

N=100 M=100          baseline -> this-commit     diff      diff% (error%)
bm_chaos.py            407.16 ->     411.85 :   +4.69 =  +1.152% (+/-0.01%)
bm_fannkuch.py         100.89 ->     101.20 :   +0.31 =  +0.307% (+/-0.01%)
bm_fft.py             3521.17 ->    3441.72 :  -79.45 =  -2.256% (+/-0.00%)
bm_float.py           6707.29 ->    6644.83 :  -62.46 =  -0.931% (+/-0.00%)
bm_hexiom.py            55.91 ->      55.41 :   -0.50 =  -0.894% (+/-0.00%)
bm_nqueens.py         5343.54 ->    5326.17 :  -17.37 =  -0.325% (+/-0.00%)
bm_pidigits.py         603.89 ->     632.79 :  +28.90 =  +4.786% (+/-0.33%)
core_qstr.py            64.18 ->      64.09 :   -0.09 =  -0.140% (+/-0.01%)
core_yield_from.py     313.61 ->     311.11 :   -2.50 =  -0.797% (+/-0.03%)
misc_aes.py            654.29 ->     659.75 :   +5.46 =  +0.834% (+/-0.02%)
misc_mandel.py        4205.10 ->    4272.08 :  +66.98 =  +1.593% (+/-0.01%)
misc_pystone.py       3077.79 ->    3128.39 :  +50.60 =  +1.644% (+/-0.01%)
misc_raytrace.py       388.45 ->     393.71 :   +5.26 =  +1.354% (+/-0.01%)
viper_call0.py         576.83 ->     566.76 :  -10.07 =  -1.746% (+/-0.05%)
viper_call1a.py        550.39 ->     540.12 :  -10.27 =  -1.866% (+/-0.11%)
viper_call1b.py        438.32 ->     432.09 :   -6.23 =  -1.421% (+/-0.11%)
viper_call1c.py        442.96 ->     436.11 :   -6.85 =  -1.546% (+/-0.08%)
viper_call2a.py        536.31 ->     527.37 :   -8.94 =  -1.667% (+/-0.04%)
viper_call2b.py        378.99 ->     377.50 :   -1.49 =  -0.393% (+/-0.08%)

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 15:43:06 +10:00
Damien George 1fb01bd6c5 py/emitnative: Put a pointer to the native prelude in child_table array.
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory.  This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise.  Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime.  The prelude was
then placed in a bytes object, pointed to by the module's constant table.

This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0.  The reasons
for doing this are:

1. It decouples the native emitter from runtime requirements, the emitted
   code no longer needs to know if the system it runs on can/can't read
   byte-wise from executable memory.

2. It makes all ports have the same emitter behaviour, there is no longer
   the N_PRELUDE_AS_BYTES_OBJ option.

3. The module's constant table is now used only for actual constants in the
   Python code.  This allows further optimisations to be done with the
   constants (eg constant deduplication).

Code size change for those ports that enable the native emitter:
   unix x64:   +80 +0.015%
      stm32:   +24 +0.004% PYBV10
    esp8266:   +88 +0.013% GENERIC
      esp32:   -20 -0.002% GENERIC[incl -112(data)]
        rp2:   +32 +0.005% PICO

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:44:49 +10:00
Damien George 8725a32f41 tools/ci.sh: For esp8266 CI install esptool 3.3.1.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:01:56 +10:00
Damien George f8d3956c63 tools/ci.sh: Run full test suite on stackless and settrace builds.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00
Damien George c49d5207e9 py/persistentcode: Remove unicode feature flag from .mpy file.
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.

The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.

With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this).  The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.

This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Damien George 2ed4f7a130 tools/mpy_ld.py: Remove obsolete QSTR_WINDOW_SIZE constant.
This was made obsolete in f2040bfc7e

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 12:51:54 +10:00
Neradoc bb4c63699a Fetch the submodules in readthedocs with ci_fetch_deps.py 2022-05-14 23:17:49 +02:00