Commit Graph

513 Commits

Author SHA1 Message Date
Jim Mussared
5852fd7708 tools/manifestfile.py: Allow manifests to set metadata.
The metadata can be version, description, and license.

After executing a manifest, the top-level metadata can be queried, and also
each file output from the manifest will have the metadata of the
containing manifest.

Use the version metadata to "tag" files before freezing such that they have
__version__ available.
2022-09-05 17:07:13 +10:00
Jim Mussared
bc23f207ce tools/manifestfile.py: Allow require() to specify unix packages.
By default, don't include micropython-lib/unix-ffi in the search.

If unix_ffi=True is passed to require(), then include unix-ffi and make it
take precedence over the other locations (e.g. python-stdlib).

This does two things:
 - Prevents non-unix builds from using unix-only packages.
 - Allows the unix build to optionally use a more full-featured (e.g. ffi)
   based package, even with the same name as one from e.g. stdlib.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:06:52 +10:00
Jim Mussared
e9a28ce312 tools/manifestfile.py: Allow include of directory path.
If an include path is a directory, then it implicitly grabs the manifest.py
file inside that directory. This simplifies most manifest.py files.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:06:52 +10:00
Jim Mussared
6bd0ec7a70 tools/makemanifest.py: Update to use mpy_cross module.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:06:43 +10:00
Jim Mussared
f3cdb052db tools/manifestfile.py: Add library for working with manifests.
This splits the manifest file loading logic from makemanifest.py and
updates makemanifest.py to use it.

This will allow non-freezing uses of manifests, such as defining packages
and dependencies in micropython-lib.

Also adds additional methods to the manifest "API":
 - require() - to get a package from micropython-lib.
 - module() - to define a single-file module
 - package() - to define a multi-file package

module() and package() should replace most uses of freeze() and can also
be also used in non-freezing scenarios.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-05 17:00:43 +10:00
Damien George
730e975091 esp32/boards: Merge manifest_release modules into standard manifest.
Having two separate manifests is confusing.  It's simpler to have the daily
builds use the same configuration as the stable, release builds.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-30 11:34:34 +10:00
Damien George
2e386bcf76 tools/mpremote: Print nicer errors for unsupported 'cp -r' arguments.
Also document support for 'cp :a :b'.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George
f5fedf4676 tools/pyboard.py: Add fs_cp function for direct device-to-device copy.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George
24f1161fe2 tools/pyboard.py: Remove implicit fs_put if source starts with ./.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George
858707181d tools/mpremote: Print a nicer error when a filesystem command fails.
Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 13:30:03 +10:00
Damien George
1855df6361 stm32: Remove support for CC3000 WiFi driver.
It has been about 8 years since support for this chip was added.  Reasons
to remove it are:
- It is no longer easy to obtain this part.
- There are now many other options for WiFi.
- It's not a good use of developer time to maintain it.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-26 12:51:37 +10:00
Jim Mussared
492ba5eaf2 stm32: Move board variant config to mpconfigboard.mk.
Rather than having the autobuild know about the particular variants, have
the mpconfigboard.mk describe them and make autobuild discover them
automatically.

Adds a "query-variants" target to stm32/Makefile to allow the set of
possible variants to be queried.

Removes pybv3 from the autobuild as this isn't use by the downloads page.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 09:38:38 +10:00
robert-hh
f6ec01d1da tools/ci.sh: Split the stm32 builds for wiznet5k and cc3k.
- Add lib/wiznet5k into the 'make submodules' step.
- Split the stm32 builds for wiznet5k and cc3k.
- Run 'make .... clean' after making the wiznet5k build.
2022-08-23 15:00:00 +10:00
Nicholas H.Tollervey
af54d2ce9f
javascript: Rename this port to 'webassembly'. 2022-08-22 12:03:39 +01:00
Jim Mussared
59e3348c10 tools/mpremote: Add "edit" command.
This allows a remote file to be edited locally by copying it over, running
the local editor, then copying it back.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:29:13 +10:00
Jim Mussared
263737ecfe tools/pyboard.py: Add "touch" filesystem command.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:24:25 +10:00
Jim Mussared
a311e9e3d4 tools/mpremote: Allow + terminator for fs commands.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-18 22:22:36 +10:00
Damien George
cbc9f944c4 tests,tools: Update path to unix micropython executable.
These were missed by 47c84286e8

Signed-off-by: Damien George <damien@micropython.org>
2022-08-18 11:47:58 +10:00
Damien George
d53c3b6ade unix/variants: Remove variant suffix from executable filename.
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-11 13:34:34 +10:00
Daniel Jour
409995ac68 tools/ci.sh: Force mpy-cross build for samd and teensy. 2022-08-11 13:33:51 +10:00
Daniel Jour
47c84286e8 all: Fix paths to mpy-cross and micropython binaries.
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory.  Thus
some paths had to be adjusted.
2022-08-11 13:31:13 +10:00
Jim Mussared
58bed5ec14 tools/ci.sh: Initialise submodules for more ports.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-03 00:08:28 +10:00
Dan Ellis
f9cbe6bc47 py/formatfloat: Format all whole-number floats exactly.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place.  This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point.  The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.

Fixes issue #4212.

Signed-off-by: Dan Ellis dan.ellis@gmail.com
2022-07-26 22:23:47 +10:00
iabdalkader
2076f2efcc tools/autobuild: Add nrf port to autobuild scripts. 2022-07-13 16:19:22 +10:00
Damien George
80bc9b3dee tools/mpremote: Don't be verbose when using cat command.
Fixes issue #8828.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-08 22:28:26 +10:00
Damien George
7e5137e0ae tools/pyboard.py: Add verbose option to filesystem_command.
Signed-off-by: Damien George <damien@micropython.org>
2022-07-08 22:26:41 +10:00
David Lechner
17f0297c1a tools/ci.sh: Drop ppa requirement for code formatting CI.
The CI scripts were using a PPA to get a backported version of uncrustify
on Ubuntu 20.04.  However, this causes CI to intermittently fail due to
connection issues to launchpad.net or the key server.

Ubuntu 22.04 has a newer version of uncrustify removing the need for the
PPA.  Ubuntu 22.04 is now in beta on GitHub actions, so it can be used.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-17 16:56:41 +10:00
Maureen Helm
bada8b3cad zephyr: Upgrade to Zephyr v3.1.0.
Updates the Zephyr port build instructions and CI to use the latest Zephyr
release tag.

Tested on frdm_k64f.

Signed-off-by: Maureen Helm <maureen.helm@intel.com>
2022-06-17 16:51:34 +10:00
Damien George
dea1ba3309 tools/metrics.py: Add mimxrt and renesas-ra to list of available ports.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-17 14:57:58 +10:00
Damien George
1f1afae622 examples/natmod/features3: Add example to test more natmod features.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Damien George
17a0d65ee4 tools/mpy_ld.py: Support GOT entries that reference inside mp_fun_table.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-10 16:42:43 +10:00
Damien George
313f082896 tools/ci.sh: Build native .mpy examples for armv7emsp, xtensawin in CI.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 13:24:08 +10:00
Damien George
abb3850398 tools/mpy_ld.py: Support R_XTENSA_PDIFF32 relocation.
Newer versions of the ESP-IDF's toolchain use this relocation.

Fixes issue #8436.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-09 11:57:22 +10:00
Damien George
6a1dbaad1c tools/ci.sh: Build NUCLEO_H743ZI with -O2 optimisation level.
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-08 15:00:59 +10:00
Damien George
5290bfaefe tools/mpremote: Bump version to 0.3.0.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:23:14 +10:00
Damien George
646fcdadbf tools/mpremote: Add command to print the version.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:22:04 +10:00
Damien George
2fb413b265 tools/mpy-tool.py: Improve generated frozen identifiers.
Frozen identifiers now include their full name hierarchy, eg their class
name.  This makes it easier to understand the generated code.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George
9670a156da all: Rename MICROPY_PY_WIZNET5K to MICROPY_PY_NETWORK_WIZNET5K.
To match MICROPY_PY_NETWORK_CYW43 and MICROPY_PY_NETWORK_NINAW10.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 16:55:18 +10:00
Damien George
bf92b0cbf2 tools/ci.sh: Add test for mpy-tool's merging feature.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:51:45 +10:00
Damien George
599a22e569 tools/mpy-tool.py: Rework .mpy merging feature.
Now that the native qstr link table is gone, merging a native .mpy file
with a bytecode .mpy file is not as simple as concatenating the .mpy data.
The qstr_table and obj_table tables from all merged .mpy files must now be
joined together, because they are global to the .mpy file (and hence global
to the merged .mpy file).  This means the bytecode needs to be be decoded,
qstr_table and obj_table indices updated to point to the correct entries in
the new tables, and then the bytecode re-encoded.

This commit makes this change to the merging feature in mpy-tool.py.  This
can now merge an arbitrary number of bytecode .mpy files, and up to one
native .mpy file.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:51:45 +10:00
Damien George
f506bf342a py/bc: Remove unused mp_opcode_format function.
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:32:38 +10:00
Damien George
b37b578214 py/persistentcode: Remove remaining native qstr linking support.
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table.  The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py.  But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.

This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table.  mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 13:19:55 +10:00
Andrew Leech
b92e51efe7 tools/ci.sh: Add build of W5100S_EVB_PICO board to rp2 CI.
This exercises the inclusion of a number of libraries in the rp2 port
including mbedtls and lwip.

Signed-off-by: Andrew Leech <andrew@alelec.net>
2022-06-03 14:35:52 +10:00
Damien George
1d047617bb tools/mpy-tool.py: Remove obsolete unicode flag in .mpy header.
This was removed in c49d5207e9

Signed-off-by: Damien George <damien@micropython.org>
2022-05-26 11:43:46 +10:00
David Lechner
d42d35f56d tests/run-tests.py: Enable -X realtime option for macOS tests.
This enables the new `-X realtime` runtime option when running tests on
macOS.  This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers.  This
makes tests that depend on the passage of time more likely to succeed.

CI tests that were disabled because of this are now enabled again.

Signed-off-by: David Lechner <david@pybricks.com>
2022-05-24 00:52:44 +10:00
Damien George
c1b9d2259e py/dynruntime.mk: Add basic support for armv6m architecture.
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms.  More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 23:01:25 +10:00
Damien George
d4d53e9e11 py/emitnative: Access qstr values using indirection table qstr_table.
This changes the native emitter to access qstr values using the qstr
indirection table qstr_table, but only when generating native code that
will be saved to a .mpy file.  This makes the resulting native code fully
static, ie it does not require any fix-ups or rewriting when it is
imported.

The performance of native code is more or less unchanged.  Benchmark
results on PYBv1.0 (using --via-mpy and --emit native) are:

N=100 M=100          baseline -> this-commit     diff      diff% (error%)
bm_chaos.py            407.16 ->     411.85 :   +4.69 =  +1.152% (+/-0.01%)
bm_fannkuch.py         100.89 ->     101.20 :   +0.31 =  +0.307% (+/-0.01%)
bm_fft.py             3521.17 ->    3441.72 :  -79.45 =  -2.256% (+/-0.00%)
bm_float.py           6707.29 ->    6644.83 :  -62.46 =  -0.931% (+/-0.00%)
bm_hexiom.py            55.91 ->      55.41 :   -0.50 =  -0.894% (+/-0.00%)
bm_nqueens.py         5343.54 ->    5326.17 :  -17.37 =  -0.325% (+/-0.00%)
bm_pidigits.py         603.89 ->     632.79 :  +28.90 =  +4.786% (+/-0.33%)
core_qstr.py            64.18 ->      64.09 :   -0.09 =  -0.140% (+/-0.01%)
core_yield_from.py     313.61 ->     311.11 :   -2.50 =  -0.797% (+/-0.03%)
misc_aes.py            654.29 ->     659.75 :   +5.46 =  +0.834% (+/-0.02%)
misc_mandel.py        4205.10 ->    4272.08 :  +66.98 =  +1.593% (+/-0.01%)
misc_pystone.py       3077.79 ->    3128.39 :  +50.60 =  +1.644% (+/-0.01%)
misc_raytrace.py       388.45 ->     393.71 :   +5.26 =  +1.354% (+/-0.01%)
viper_call0.py         576.83 ->     566.76 :  -10.07 =  -1.746% (+/-0.05%)
viper_call1a.py        550.39 ->     540.12 :  -10.27 =  -1.866% (+/-0.11%)
viper_call1b.py        438.32 ->     432.09 :   -6.23 =  -1.421% (+/-0.11%)
viper_call1c.py        442.96 ->     436.11 :   -6.85 =  -1.546% (+/-0.08%)
viper_call2a.py        536.31 ->     527.37 :   -8.94 =  -1.667% (+/-0.04%)
viper_call2b.py        378.99 ->     377.50 :   -1.49 =  -0.393% (+/-0.08%)

Signed-off-by: Damien George <damien@micropython.org>
2022-05-23 15:43:06 +10:00
Damien George
1fb01bd6c5 py/emitnative: Put a pointer to the native prelude in child_table array.
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory.  This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise.  Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime.  The prelude was
then placed in a bytes object, pointed to by the module's constant table.

This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0.  The reasons
for doing this are:

1. It decouples the native emitter from runtime requirements, the emitted
   code no longer needs to know if the system it runs on can/can't read
   byte-wise from executable memory.

2. It makes all ports have the same emitter behaviour, there is no longer
   the N_PRELUDE_AS_BYTES_OBJ option.

3. The module's constant table is now used only for actual constants in the
   Python code.  This allows further optimisations to be done with the
   constants (eg constant deduplication).

Code size change for those ports that enable the native emitter:
   unix x64:   +80 +0.015%
      stm32:   +24 +0.004% PYBV10
    esp8266:   +88 +0.013% GENERIC
      esp32:   -20 -0.002% GENERIC[incl -112(data)]
        rp2:   +32 +0.005% PICO

Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:44:49 +10:00
Damien George
8725a32f41 tools/ci.sh: For esp8266 CI install esptool 3.3.1.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 16:01:56 +10:00
Damien George
f8d3956c63 tools/ci.sh: Run full test suite on stackless and settrace builds.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-17 14:25:51 +10:00