Note: at this time, the ssl module on pico_w never verifies the server
certificate. This means it does not actually provide a higher security
level than regular socket / http protocols.
Originally, black_bindings found each contiguous "//|" block and sent
it to black independently. This was slower than it needed to be.
Instead, swap the comment prefix: when running black, take off
"//|" prefixes and put "##|" prefixes on all un-prefixed lines.
Then, after black is run, do the opposite operation
This more than doubles the overall speed of "pre-commit run --all",
from 3m20s to 55s CPU time on my local machine (32.5s to under 10s
"elapsed" time)
It also causes a small amount of churn in the bindings, because
black now sees enough context to know whether one 'def' follows another
or ends the 'def's in a 'class'. In the latter case, it adds an extra
newline, which becomes a "//|" line.
I'm less sure why a trailing comma was omitted before down in
rp2pio/StateMachine.c but let's roll with it.
My pings go out, and then they come back
```py
import os
import wifi
import ipaddress
wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
ipv4 = ipaddress.ip_address("8.8.4.4")
print("Ping google.com: %f ms" % (wifi.radio.ping(ipv4)*1000))
```
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
* Tweak scroll area position so last line is complete and top is
under the title bar.
* Pick Blinka size based on the font to minimize unused space in
title bar. Related to #2791
* Update the title bar after terminal is started. Fixes#6078Fixes#6668
If a changed file is in frozen, supervisor, shared-bindings,
shared-module or common-hal, then compute board settings to
determine if it is impacted.
This should reduce the boards built for changes that are in those
directories.
- define CIRCUITPY_BUILD_EXTENSIONS to predefined values
- set CIRCUITPY_BUILD_EXTENSIONS in port and board config
- reuse the support matrix "get_settings_from_makefile" to get it
- move the existing port and board specific values
- remove the C3 specific board values because it's not the default
- update build_release_files.py to use get_settings_from_makefile
A recent build failed. The original error seemed to be during ci_fetch_deps
where a build message said
```
fatal: reference is not a tree: 346c936e14c6ea3a8d3d65cb1fa46202dc92999d
fatal: Unable to checkout '346c936e14c6ea3a8d3d65cb1fa46202dc92999d' in submodule path 'extmod/ulab'
```
(along with other problems), but this step didn't signal failure to
github actions.
By adding the check= parameter, a failure of the subprocess will cause
a CalledProcessError to be raised, which will make ci_fetch_deps exit with
nonzero status. In turn, this should let Actions understand that something
went wrong with this step, instead of waiting for some subsequent step(s)
to go wrong.
This adds support for CIRCUITPY_WIFI_SSID and CIRCUITPY_WIFI_PASSWORD
in `/.env`. When both are defined, CircuitPython will attempt to
connect to the network even when user code isn't running. If the
user code attempts to a network with the same SSID, it will return
immediately. Connecting to another SSID will disconnect from the
auto-connected network. If the user code initiates the connection,
then it will be shutdown after user code exits. (Should match <8
behavior.)
This PR also reworks the default displayio terminal. It now supports
a title bar TileGrid in addition to the (newly renamed) scroll area.
The default title bar is the top row of the display and is positioned
to the right of the Blinka logo when it is enabled. The scroll area
is now below the Blinka logo.
The Wi-Fi auto-connect code now uses the title bar to show its
state including the IP address when connected. It does this through
the "standard" OSC control sequence `ESC ] 0 ; <s> ESC \` where <s>
is the title bar string. This is commonly supported by terminals
so it should work over USB and UART as well.
Related to #6174
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.
Signed-off-by: Damien George <damien@micropython.org>
Frozen identifiers now include their full name hierarchy, eg their class
name. This makes it easier to understand the generated code.
Signed-off-by: Damien George <damien@micropython.org>
Now that the native qstr link table is gone, merging a native .mpy file
with a bytecode .mpy file is not as simple as concatenating the .mpy data.
The qstr_table and obj_table tables from all merged .mpy files must now be
joined together, because they are global to the .mpy file (and hence global
to the merged .mpy file). This means the bytecode needs to be be decoded,
qstr_table and obj_table indices updated to point to the correct entries in
the new tables, and then the bytecode re-encoded.
This commit makes this change to the merging feature in mpy-tool.py. This
can now merge an arbitrary number of bytecode .mpy files, and up to one
native .mpy file.
Signed-off-by: Damien George <damien@micropython.org>
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.
Signed-off-by: Damien George <damien@micropython.org>
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table. The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py. But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.
This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table. mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.
Signed-off-by: Damien George <damien@micropython.org>
This enables the new `-X realtime` runtime option when running tests on
macOS. This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers. This
makes tests that depend on the passage of time more likely to succeed.
CI tests that were disabled because of this are now enabled again.
Signed-off-by: David Lechner <david@pybricks.com>
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms. More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.
Signed-off-by: Damien George <damien@micropython.org>
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory. This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise. Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was
then placed in a bytes object, pointed to by the module's constant table.
This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons
for doing this are:
1. It decouples the native emitter from runtime requirements, the emitted
code no longer needs to know if the system it runs on can/can't read
byte-wise from executable memory.
2. It makes all ports have the same emitter behaviour, there is no longer
the N_PRELUDE_AS_BYTES_OBJ option.
3. The module's constant table is now used only for actual constants in the
Python code. This allows further optimisations to be done with the
constants (eg constant deduplication).
Code size change for those ports that enable the native emitter:
unix x64: +80 +0.015%
stm32: +24 +0.004% PYBV10
esp8266: +88 +0.013% GENERIC
esp32: -20 -0.002% GENERIC[incl -112(data)]
rp2: +32 +0.005% PICO
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
- fix the list of frozen modules in the board info, separate it from the builtin modules
- frozen modules are listed along with the link to their repository
- get the repository by running the git command in the frozen directory
- frozen modules are listed at the end
- they copy the style of the other modules
- frozen modules in build_board_info don't need the URLs, they are filtered out
This will add a space after a comma if it doesn't have one, but will allow
more than one space if the spaces are already there.
Signed-off-by: Damien George <damien@micropython.org>
This adds a rule to cover all ports/**/*.[ch] file to the code formatting
list. Explicit exclusions are also added for code in ports/ which is third
party, or which requires a lot of reformatting.
Signed-off-by: Damien George <damien@micropython.org>
To keep the separate parts of the code that use these values in sync. And
make it easier to add new object types.
Signed-off-by: Damien George <damien@micropython.org>
Specifying the option `--unsafe-links` (or `-l`) to `mpremote mount` will
allow symlinks to be followed in the local directory that point outside of
the base directory path.
For the unsafe case the `path_check()` method of `PyboardCommand` still
checks for a common path but without expanding symlinks. While this check
is currently redundant, it makes the purpose of the method clearer for
possible future uses or extensions.
With the existing code problems can occur with remounting, the "if t -
t_last_activity > QUIET_TIMEOUT:" check can be triggered early before the
REPL string comes through, meaning that the remount doesn't happen.
On certain boards the "MPY: soft reboot" line comes through immediately
(getting the routine past initial timeout) but then there's a slightly
longer delay while the board restarts before it prints out the startup
header and the REPL prompt.
This commit adds some extra pattern monitoring during the timeout loop to
track the state if a soft restart is actually started.
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument. These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
Since cpydiff is code used as documentation, there are cases where we may
want to use Black's `fmt: on/off/skip` comments to avoid automatic
formatting. However, we don't want these comments to be distracting in the
generated documentation.
This rewrites the code to omit these comments when generating the docs.
Signed-off-by: David Lechner <david@pybricks.com>
To me, it made more sense to track which boards go together in a cluster;
when reviewing a request to actually use a duplicate vid/pid, you want
to know what board(s) it is aliasing.
I also revamped the detection of non-USB boards so that a board .mk file
that couldn't be parsed by the code here would raise a problem instead
of just being skipped for the purposes of checking.
There were some lines with comments on the end, and some variation in
capitalization of the IDs. These are all normalized and a (sometimes
unfriendly!) error printed when it's incorrect.
Before this, here were some ways to trick the duplicate vid/pid checker:
```
USB_PID = 0XABCD
USB_PID = 0xAbCd
USB_PID = 0xABCD # harmless comment?
```
None of these things were ever done on purpose.
Tested on PYBV10 and PYBD_SF6, with MBOOT_FSLOAD enabled and programming
new firmware from a .dfu.gz file stored on the SD card.
Signed-off-by: Damien George <damien@micropython.org>
That caused the compile of frozen_content.c to fail if characters like
backslash were in a short string. Thanks to @hippy for identifying the
spot to change.
This makes the auto soft-reset behaviour of mpremote more logical, and now
configurable via these new commands.
Signed-off-by: Damien George <damien@micropython.org>
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
Scott Shawcroft
Dominic Davis-Foster
Jeff Epler
microDev
Dan Halbert
Dan Halbert
Kyle Mohr
Dan Ellis
Neradoc
foamyguy
Patrick
Matthew McGowan
Damien George
Nathan Young
Andrew Leech
David Lechner
SimonVail
Takeo Takahashi
Rob Knegjens
Andrew Leech
Waterlens
Michael Himing
Maureen Helm
robert-hh