Commit Graph

69 Commits

Author SHA1 Message Date
Damien George de43b500bd py/runtime: Allow initialising sys.path/argv with defaults.
If MICROPY_PY_SYS_PATH_ARGV_DEFAULTS is enabled (which it is by default)
then sys.path and sys.argv will be initialised and populated with default
values.  This keeps all bare-metal ports aligned.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-18 00:08:07 +11:00
Jim Mussared a7fa18c203 py/builtinimport: Refactor module importing.
Simplify and document/comment the handling of builtin import for:
- already-loaded modules
- built-in modules
- built-in umodules (formerly weak links)
- filesystem modules

Retains existing functionality with smaller code size but should also
facilitate potential new features (built-in packages, controlling the
frozen path).

Also makes the (unix-only) -m behavior a bit more obvious and configurable.

Code size change with this commit:

   bare-arm:    +0 +0.000%
minimal x86:   -64 -0.039%
   unix x64:   -32 -0.006%
unix nanbox:    -4 -0.001%
      stm32:  -184 -0.047% PYBV10
     cc3200:  -120 -0.065%
    esp8266:  -228 -0.033% GENERIC
      esp32:  -268 -0.018% GENERIC[incl +16(data)]
        nrf:  -152 -0.087% pca10040
        rp2:  -256 -0.052% PICO
       samd:   -80 -0.057% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2021-12-01 13:23:34 +11:00
retsyo e5f9e2febc windows/mpconfigport.h: Enable help and help("modules").
Following a similar change to the unix port in
6430cd3e02
2021-11-17 14:52:39 +11:00
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
stijn 318c029d45 windows/README: Remove unsupported Python instructions for Cygwin.
It's not possible anymore to build MicroPython on Cygwin using a
standard Windows installation of Python so don't advertise that.
Specifically: preprocessing in makeqstrdefs.py fails on the subprocess
call with 'gcc: fatal error: no input files' because one of the flags
contains double quotes and that somehow messes up the commandline.
2021-09-10 15:19:25 +10:00
Jim Mussared 692d36d779 py: Implement partial PEP-498 (f-string) support.
This implements (most of) the PEP-498 spec for f-strings and is based on
https://github.com/micropython/micropython/pull/4998 by @klardotsh.

It is implemented in the lexer as a syntax translation to `str.format`:
  f"{a}" --> "{}".format(a)

It also supports:
  f"{a=}" --> "a={}".format(a)

This is done by extracting the arguments into a temporary vstr buffer,
then after the string has been tokenized, the lexer input queue is saved
and the contents of the temporary vstr buffer are injected into the lexer
instead.

There are four main limitations:
- raw f-strings (`fr` or `rf` prefixes) are not supported and will raise
  `SyntaxError: raw f-strings are not supported`.

- literal concatenation of f-strings with adjacent strings will fail
    "{}" f"{a}" --> "{}{}".format(a)    (str.format will incorrectly use
                                         the braces from the non-f-string)
    f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate)

- PEP-498 requires the full parser to understand the interpolated
  argument, however because this entirely runs in the lexer it cannot
  resolve nested braces in expressions like
    f"{'}'}"

- The !r, !s, and !a conversions are not supported.

Includes tests and cpydiffs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-14 16:58:40 +10:00
Jim Mussared 22fdb21302 windows/appveyor: Update to VS 2017 and use Python 3.8 for build/test.
MicroPython implements some 3.5+ features, and this change helps to reduce
the need for some .exp files in the test suite.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-07-15 00:10:52 +10:00
Damien George 136369d72f all: Update to point to files in new shared/ directory.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-12 17:08:10 +10:00
Bryan Tong Minh 3d9af87721 windows/Makefile: Add .exe extension to executables name.
Uses the same logic applied in 5b57ae985f
to determine when to add .exe.

See related: #3310, #3361, #3370, #4143, #5727.
2021-07-08 12:35:08 +10:00
David Lechner 259d9b69fe py/mpstate: Schedule KeyboardInterrupt on main thread.
This introduces a new macro to get the main thread and uses it to ensure
that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only
scheduled on the main thread. This is more deterministic than being
scheduled on a random thread and is more in line with CPython that only
allow signal handlers to run on the main thread.

Fixes issue #7026.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-19 09:49:00 +10:00
David Lechner ca920f7218 py/mpstate: Make exceptions thread-local.
This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t.
This allows exceptions to be scheduled on a specific thread.

Signed-off-by: David Lechner <david@pybricks.com>
2021-06-19 09:43:44 +10:00
Damien George e9e9c76ddf all: Rename mp_keyboard_interrupt to mp_sched_keyboard_interrupt.
To match mp_sched_exception() and mp_sched_schedule().

Signed-off-by: Damien George <damien@micropython.org>
2021-04-30 15:13:43 +10:00
stijn bb2007b05c windows/mpconfigport.h: Enable features also present in unix port. 2021-04-23 23:15:10 +10:00
Damien George 6129b8e401 tests: Rename run-tests to run-tests.py for consistency.
Signed-off-by: Damien George <damien@micropython.org>
2021-03-12 19:56:09 +11:00
stijn 81a4d96aed windows/msvc: Use same default python command as core. 2021-02-02 21:33:18 +11:00
stijn 0397448501 tests/run-tests: Change default Python command used on Windows.
Default to just calling python since that is most commonly available: the
official installer or zipfiles from python.org, anaconda, nupkg all result
in python being available but not python3.  In other words: the default
used so far is wrong.  Note that os.name is 'posix' when running the python
version which comes with Cygwin or MSys2 so they are not affected by this.
However of all possible ways to get Python on Windows, only Cygwin provides
no python command so update the default way for running tests in the
README.
2021-02-02 21:32:20 +11:00
Jim Mussared 4559bcb467 unix: Make mp_hal_delay_ms run MICROPY_EVENT_POLL_HOOK.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-13 17:19:05 +11:00
Damien George 905a18aafe unix,windows: Implement mp_hal_time_ns using gettimeofday.
This provides microsecond accuracy.

Signed-off-by: Damien George <damien@micropython.org>
2020-10-01 14:20:42 +10:00
stijn bada8c9231 windows: Update build instructions in README.
Make the instructions more complete by documenting all needed steps for
starting from scratch.  Also add a section for MSYS2 since the Travis build
uses it as well and it's a good alternative for Cygwin.  Remove the mingw32
reference since it's not readily available anymore in most Linux distros
nor compiles successfully.
2020-09-23 22:22:13 +10:00
stijn 70bec41089 windows: Show test failures in the Appveyor builds. 2020-09-11 10:56:02 +10:00
stijn 5b94c61097 windows/Makefile: Support freezing modules.
Alter the build flags as needed to support freezing modules with a
manifest.  This makes freezing works just like it does for e.g. the unix
port.
2020-09-11 10:52:52 +10:00
stijn 2a9ea69fa9 windows/msvc: Support freezing modules.
Support freezing modules via manifest.py for consistency with the other
ports.  In essence this comes down to calling makemanifest.py and adding
the resulting .c file to the build.  Note the file with preprocessed qstrs
has been renamed to match what makemanifest.py expects and which is also
the name all other ports use.
2020-09-11 10:52:35 +10:00
stijn 2e54d9d146 py: Fix handling of NaN in certain pow implementations.
Adds a new compile-time option MICROPY_PY_MATH_POW_FIX_NAN for use with
toolchains that don't handle pow-of-NaN correctly.
2020-09-11 10:04:57 +10:00
Damien George 2a72e90ab8 extmod/vfs: Add option to use 1970 as Epoch.
By setting MICROPY_EPOCH_IS_1970 a port can opt to use 1970/1/1 as the
Epoch for timestamps returned by stat().  And this setting is enabled on
the unix and windows ports because that's what they use.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-01 12:36:28 +10:00
stijn 9523ca92e0 windows: Make appveyor.yml self-contained.
Add configuration which otherwise has to be set via the UI so the file is
more self-contained, and remove configuration which is not needed because
it's the same as the default.  The major change here is that for a while
now Appveyor has been using Visual Studio 2015 by default while we still
want to support 2013.
2020-05-28 09:56:35 +10:00
stijn 81db22f693 py/modmath: Work around msvc float bugs in atan2, fmod and modf.
Older implementations deal with infinity/negative zero incorrectly.  This
commit adds generic fixes that can be enabled by any port that needs them,
along with new tests cases.
2020-05-28 09:54:54 +10:00
Jim Mussared 710426024a all: Factor gchelper code to one place and use it for unix & ARM ports.
No functionality change is intended with this commit, it just consolidates
the separate implementations of GC helper code to the lib/utils/ directory
as a general set of helper functions useful for any port.  This reduces
duplication of code, and makes it easier for future ports or embedders to
get the GC implementation correct.

Ports should now link against gchelper_native.c and either gchelper_m0.s or
gchelper_m3.s (currently only Cortex-M is supported but other architectures
can follow), or use the fallback gchelper_generic.c which will work on
x86/x64/ARM.

The gc_helper_get_sp function from gchelper_m3.s is not really GC related
and was only used by cc3200, so it has been moved to that port and renamed
to cortex_m3_get_sp.
2020-04-29 23:45:19 +10:00
stijn 84fa3312cf all: Format code to add space after C++-style comment start.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
2020-04-23 11:24:25 +10:00
stijn 30840ebc99 all: Enable extra conversion warnings where applicable.
Add -Wdouble-promotion and -Wfloat-conversion for most ports to ban out
implicit floating point conversions, and add extra Travis builds using
MICROPY_FLOAT_IMPL_FLOAT to uncover warnings which weren't found
previously.  For the unix port -Wsign-comparison is added as well but only
there since only clang supports this but gcc doesn't.
2020-04-18 22:42:28 +10:00
stijn 70affd9ba2 all: Fix implicit floating point to integer conversions.
These are found when building with -Wfloat-conversion.
2020-04-18 22:42:24 +10:00
Jim Mussared 92c83bd16b windows: Update genhdr.targets to match makeqstrdefs.py args. 2020-04-05 14:29:49 +10:00
Damien George 1a3e386c67 all: Remove spaces inside and around parenthesis.
Using new options enabled in the uncrustify configuration.
2020-03-28 23:36:44 +11:00
David Lechner d0edaf88a3 windows/windows_mphal: Fix missing semicolon. 2020-03-25 00:59:34 +11:00
David Lechner 3b07736b6d unix,windows: Use STDIN_FILENO, STDOUT_FILENO macros where appropriate.
This replaces 0 and 1 with STDIN_FILENO and STDOUT_FILENO to make the
intention of the code clearer.
2020-03-25 00:59:05 +11:00
David Lechner 100012bec6 windows: Remove custom definition of MP_PLAT_PRINT_STRN.
This removes the port-specific definition of MP_PLAT_PRINT_STRN on the
windows port, so that the default mp_hal_stdout_tx_strn_cooked() is always
used.  This fixes releasing the GIL during the call to write() (this was
missed in bc3499f010).

Also, mp_hal_dupterm_tx_strn() was defined but never used anywhere so it is
safe to delete it.
2020-03-25 00:56:38 +11:00
stijn f62cc41fac windows/msvc: Fix warnings regarding function declarations.
Fix missing mkdir and gettimeofday declarations, then silence msvc-specific
compiler warning C4996: 'The POSIX name for this item is deprecated'.
2020-03-25 00:38:11 +11:00
Damien George 2cdf1d25f5 unix: Remove custom file implementation to use extmod's VFS POSIX one.
The implementation in extmod/vfs_posix_file.c is now equivalent to that in
ports/unix/file.c, so remove the latter and use the former instead.
2020-03-18 21:01:07 +11:00
Damien George 69661f3343 all: Reformat C and Python source code with tools/codeformat.py.
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-28 10:33:03 +11:00
stijn 5f91933e54 windows: Improve default search path.
The default value for MICROPYPATH used in unix/main.c is
"~/.micropython/lib:/usr/lib/micropython" which has 2 problems when used in
the Windows port:
- it has a ':' as path separator but the port uses ';' so the entire string
  is effectively discarded since it gets interpreted as a single path which
  doesn't exist
- /usr/lib/micropython is not a valid path in a standard Windows
  environment

Override the value with a suitable default.
2020-02-11 13:34:35 +11:00
Damien George 046ae80bdf unix, windows: Use mp_keyboard_interrupt instead of custom code.
The mp_keyboard_interrupt() function does exactly what is needed here, and
using it gets ctrl-C working when MICROPY_ENABLE_SCHEDULER is enabled on
these ports (and MICROPY_ASYNC_KBD_INTR is disabled).
2020-02-07 16:08:33 +11:00
David Lechner bc3499f010 windows/windows_mphal: Release GIL during system calls.
This releases the GIL during syscalls that could block.
2020-01-26 23:27:40 +11:00
stijn 599371b133 windows: Support word-based move/delete key sequences for REPL.
Translate common Ctrl-Left/Right/Delete/Backspace to the EMACS-style
sequences (i.e. Alt key based) for forward-word, backward-word, forwad-kill
and backward-kill.  Requires MICROPY_REPL_EMACS_WORDS_MOVE to be defined so
the readline implementation interprets these.
2020-01-22 16:54:19 +11:00
Damien George 1582c7eeb0 unix,windows: Enable module weak links. 2019-10-22 16:23:43 +11:00
stijn d96391aca3 mpy-cross,windows: Add msvc build of mpy-cross.
Add the project file to the mpy-cross directory, which is also where the
executable ends up, and change the Appveyor settings to build mpy-cross
with both msvc and mingw-w64 and verify this all works by running tests
with --via-mpy.
2019-09-18 22:16:36 +10:00
stijn bc86c6252a windows: Default to binary mode for files.
If this is not set it might default to calls to open() to use text mode
which is usually not wanted, and even wrong and leading to incorrect
results when loading binary .mpy files.

This also means that text files written and read will not have line-ending
translation from \n to \r\n and vice-versa anymore.  This shouldn't be much
of a problem though since most tools dealing with text files adapt
automatically to any of the 2 formats.
2019-09-18 22:15:48 +10:00
stijn 6957939604 windows/msvc: Change the way sources are listed.
Reserve sources.props for listing just the MicroPython core and extmod
files, similar to how py.mk lists port-independent source files.  This
allows reusing the source list, for instance for building mpy-cross.  The
sources for building the executable itself are listed in the corresponding
project file, similar to how the other ports specify the source files in
their Makefile.
2019-09-18 22:15:10 +10:00
stijn 146c32a141 windows/msvc: Enable overriding directories used in the build.
Append to PyIncDirs, used to define include directories specific to
MicroPython, instead of just overwriting it so project files importing this
file can define additional directories.  And allow defining the target
directory for the executable instead of hardcoding it to the windows
directory.  Main reason for this change is that it will allow building
mpy-cross with msvc.
2019-09-18 22:14:37 +10:00
stijn 94873a4826 windows/msvc: Move build options from .vcxproj to .props files.
We want the .vcxproj to be just a container with the minimum content for
making it work as a project file for Visual Studio and MSBuild, whereas the
actual build options and actions get placed in separate reusable files.
This was roughly the case already except some compiler options were
overlooked; fix this here: we'll need those common options when adding a
project file for building mpy-cross.
2019-09-18 22:14:11 +10:00
stijn b2b21839d3 windows/msvc: Remove unneeded definitions for qstr generation.
These were probably added to detect more qstrs but as long as the
micropython executable itself doesn't use the same build options the qstrs
would be unused anyway.  Furthermore these definitions are for internal use
and get enabled when corresponding MICROPY_EMIT_XXX are defined, in which
case the compiler would warn about symbol redefinitions since they'd be
defined both here and in the source.
2019-09-18 22:13:26 +10:00
Andrew Leech 380048df64 windows/Makefile: Make use of CFLAGS_EXTRA, LDFLAGS_EXTRA and SRC_MOD.
To be consistent with the unix port.
2019-09-10 11:20:36 +10:00