Commit Graph

11648 Commits

Author SHA1 Message Date
Damien George b0932fcf2e all: Bump version to 1.13.
Signed-off-by: Damien George <damien@micropython.org>
2020-09-02 12:01:26 +10:00
Damien George 0e6ef40359 tests/extmod: Add tests for verifying FAT and littlefs mtime values.
Verifies mtime timestamps on files match the value returned by time.time().

Also update vfs_fat_ramdisk.py so it doesn't check FAT timestamp of the
root, because that may change across runs/ports.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-02 00:19:38 +10:00
Damien George a909c21587 unix/fatfs_port: Fix month offset in timestamp calculation.
Signed-off-by: Damien George <damien@micropython.org>
2020-09-02 00:19:06 +10:00
Damien George 2a72e90ab8 extmod/vfs: Add option to use 1970 as Epoch.
By setting MICROPY_EPOCH_IS_1970 a port can opt to use 1970/1/1 as the
Epoch for timestamps returned by stat().  And this setting is enabled on
the unix and windows ports because that's what they use.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-01 12:36:28 +10:00
Damien George 0385b21597 unix/modos: Support larger integer range in uos.stat fields.
On 32-bit builds these stat fields will overflow a small-int, so use
mp_obj_new_int_from_uint to construct the int object.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-01 12:36:28 +10:00
Damien George c70e599659 extmod/vfs: Support larger integer range in VFS stat time fields.
On ports like unix where the Epoch is 1970/1/1 and atime/mtime/ctime are in
seconds since the Epoch, this value will overflow a small-int on 32-bit
systems.  So far this is only an issue on 32-bit unix builds that use the
VFS layer (eg dev and coverage unix variants) but the fix (using
mp_obj_new_int_from_uint instead of MP_OBJ_NEW_SMALL_INT) is there for all
ports so as to not complicate the code, and because they will need the
range one day.

Also apply a similar fix to other fields in VfsPosix.stat because they may
also be large.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-01 12:36:28 +10:00
Damien George 40153b800a esp32/mphalport: Fix mp_hal_time_ns offset.
gettimeofday returns seconds since 2000/1/1 so needs to be adjusted to
seconds since 1970/1/1 to give the correct return value of mp_hal_time_ns.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-31 00:49:58 +10:00
Damien George 836bca9956 unix/variants: Fix fast and freedos variants so they build again.
This regressed in bd2fff6687

Signed-off-by: Damien George <damien@micropython.org>
2020-08-30 13:48:26 +10:00
Damien George d1995e50eb extmod/modlwip: Fix error return for TCP recv when not connected.
This commit fixes the cases when a TCP socket is in STATE_NEW,
STATE_LISTENING or STATE_CONNECTING and recv() is called on it.  It now
raises ENOTCONN instead of a random error code due to it previously
indexing beyond the start of error_lookup_table[].

Signed-off-by: Damien George <damien@micropython.org>
2020-08-30 13:20:51 +10:00
Damien George 06659077a8 all: Update Python code to conform to latest black formatting.
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:

- If there is a trailing comma in a list (eg [], () or function call) then
  that list is now written out with one line per element.  So remove such
  trailing commas where the list should stay on one line.

- Spaces at the start of """ doc strings are removed.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 15:18:01 +10:00
Damien George 0c7354afaf tests: Split out complex reverse-op tests to separate test file.
So they can be skipped if __rOP__'s are not supported on the target.  Also
fix the typo in the complex_special_methods.py filename.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 14:12:20 +10:00
Damien George 40d174ac7d stm32/powerctrl.h: Include stdbool.h to get definition of bool.
Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 14:00:24 +10:00
Damien George 338b12d3c8 LICENSE,docs: Update copyright year range to include 2020.
Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 13:58:20 +10:00
Damien George a93a378e93 zephyr/README: Update required Zephyr version and mention new features.
Signed-off-by: Damien George <damien@micropython.org>
2020-08-28 16:42:32 +10:00
Michael Buesch cef678b2db extmod/machine_i2c: Fix buffer overrun if 'addrsize' is bigger than 32.
The memory operation functions read_mem() and write_mem() create a
temporary buffer on the local C stack for the address bytes with the size
of 4 bytes.  This buffer is filled in a loop from the user supplied address
and address length.  If the user supplied 'addrsize' is bigger than 32, the
local buffer is overrun.

Fix this by raising an exception for invalid 'addrsize' values.

Signed-off-by: Michael Buesch <m@bues.ch>
2020-08-27 12:39:11 +10:00
stijn 0c3f9d58a5 tests/run-tests: Make test output directory configurable.
A configurable result directory is advantageous because it enables
using a dedicated location, eventually outside of the source tree,
instead of forcing the output files into a fixed directory which might
also contain other files already. For that reason the default output
directory also has been changed to tests/results/.
2020-08-27 11:12:08 +10:00
stijn 405893afc6 tests/run-tests: Use absolute paths where possible.
Replace some usages of paths relative to the current working directory
with absolute paths relative to the tests directory.

Fixes and resulting changes:
- default values of MICROPYTHON and MPYCROSS are absolute paths and
  always correct
- likewise, the correct full paths for tools and extmod directories
  are appended to sys.path
- printing/cleaning failures works properly since it expects the .exp
  and .out files in the tests directory which is also where they
  are written to now, plus no more need for changing directories

This fixes #5872 and allows running custom tests which use run-tests
without having to cd to the tests directory first, and the test output
still is in the tests/ directory instead of the current working directory.

Discovery of tests and all skip test logic based on paths relative to
the current working directory remains unchanged which essentially means
that for running most of MicroPython's own tests, run-tests must still
be ran from within it's directory, so document that.
2020-08-27 11:12:08 +10:00
Roberto Colistete Jr 91c5d168c0 nrf/Makefile: Improve user C modules support.
Add CFLAGS_EXTRA to CFLAGS. Include LDFLAGS_MOD to the compilation.
And, add SRC_MOD to SRC_QSTR.
2020-08-26 22:14:40 +02:00
Damien George 5fb276de33 tests/extmod: Make uasyncio_fair test more reliable by adjusting sleeps.
With sleep(0.2) a multiple of sleep(0.1), the order of task 2 and 3
execution is not well defined, and depends on the precision of the system
clock and how fast the rest of the code runs.  So change 0.2 to 0.18 to
make the test more reliable.

Also fix a typo of t3/t4, and cancel t4 at the end.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-26 17:05:52 +10:00
Andrew Leech a80a146858 extmod/bluetooth: Support active scanning in BLE.gap_scan().
This adds an additional optional parameter to gap_scan() to select active
scanning, where scan responses are returned as well as normal scan results.
This parameter is False by default which retains the existing behaviour.
2020-08-26 15:00:11 +10:00
Jim Mussared 0bc2c1c105 extmod/modbluetooth: Fix race between READ_REQUEST and other IRQs.
The READ_REQUEST callback is handled as a hard interrupt (because the BLE
stack needs an immediate response from it so it can continue) and so calls
to Python require extra protection:

- the caller-owned tuple passed into the callback must be separate from the
  tuple used by other callback events (which are soft interrupts);

- the GC and scheduler must be locked during callback execution.
2020-08-26 14:57:36 +10:00
Jim Mussared 3d9a7ed02f extmod/btstack: Implement GAP scan duration_ms parameter.
This commit makes scanning work when duration_ms is set to zero.  Prior to
this it would not work with duration_ms set to zero.
2020-08-26 14:55:52 +10:00
Damien George 2acc087880 extmod/vfs_lfs: Add mtime support to littlefs files.
This commit adds support for modification time of files on littlefs v2
filesystems, using file attributes.  For some background see issue #6114.

Features/properties of this implementation:
- Only supported on littlefs2 (not littlefs1).
- Uses littlefs2's general file attributes to store the timestamp.
- The timestamp is 64-bits and stores nanoseconds since 1970/1/1 (if the
  range to the year 2554 is not enough then additional bits can be added to
  this timestamp by adding another file attribute).
- mtime is enabled by default but can be disabled in the constructor, eg:
  uos.mount(uos.VfsLfs2(bdev, mtime=False), '/flash')
- It's fully backwards compatible, existing littlefs2 filesystems will work
  without reformatting and timestamps will be added transparently to
  existing files (once they are opened for writing).
- Files without timestamps will open correctly, and stat will just return 0
  for their timestamp.
- mtime can be disabled or enabled each mount time and timestamps will only
  be updated if mtime is enabled (otherwise they will be untouched).

Signed-off-by: Damien George <damien@micropython.org>
2020-08-25 17:35:19 +10:00
Damien George ee50a6effe py/mphal.h: Introduce mp_hal_time_ns and implement on various ports.
This should return a 64-bit value being the number of nanoseconds since
1970/1/1.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 16:13:44 +10:00
Damien George badd351150 lib/timeutils: Add helper functions to deal with nanosecs since 1970.
Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 15:41:10 +10:00
Damien George 92899354d9 unix/fatfs_port: Implement get_fattime.
Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 14:45:57 +10:00
Damien George 55c76eaac1 extmod/uasyncio: Truncate negative sleeps to 0.
Otherwise a task that continuously awaits on a large negative sleep can
monopolise the scheduler (because its wake time is always less than
everything else in the pairing heap).

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 12:17:06 +10:00
Damien George 20948a3d54 tests/extmod: Add test for uasyncio.sleep of a negative time.
It should take 0 time to await on a negative sleep.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 12:17:06 +10:00
Damien George 5f9b105244 py/runtime: Fix builtin compile() in "single" mode so it prints exprs.
As per CPython behaviour, compile(stmt, "file", "single") should create
code which prints to stdout (via __repl_print__) the results of any
expressions in stmt.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 11:38:46 +10:00
Damien George 448319a745 tools/makemanifest.py: Use os.makedirs to make path for generated files.
The existing implementation of mkdir() in this file is not sophisticated
enough to work correctly on all operating systems (eg Mac can raise
EISDIR).  Using the standard os.makedirs() function handles all cases
correctly.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 11:18:48 +10:00
Martin Milata 492cf34fd8 tools/mpy-tool.py: Fix offset of line number info.
Signed-off-by: Martin Milata <martin@martinmilata.cz>
2020-08-21 16:17:07 +10:00
Michael Buesch 60cf2c0959 tools/pyboard.py: Replace eval() of received data with alternative.
Prior to this commit, pyboard.py used eval() to "parse" file data received
from the board.  Using eval() on received data from a device is dangerous,
because a malicious device may inject arbitrary code execution on the PC
that is doing the operation.

Consider the following scenario:

Eve may write a malicious script to Bob's board in his absence.  On return
Bob notices that something is wrong with the board, because it doesn't work
as expected anymore.  He wants to read out boot.py (or any other file) to
see what is wrong.  What he gets is a remote code execution on his PC.

Proof of concept:

Eve:

  $ cat boot.py
  _print = print
  print = lambda *x, **y: _print("os.system('ls /; echo Pwned!')", end="\r\n\x04")
  $ ./pyboard.py -f cp boot.py :
  cp boot.py :boot.py

Bob:

  $ ./pyboard.py -f cp :boot.py /tmp/foo
  cp :boot.py /tmp/foo
  bin   chroot  dev  home  lib32  media  opt   root  sbin  sys  usr
  boot  config  etc  lib   lib64  mnt    proc  run   srv   tmp  var
  Pwned!

There's also the possibility that the device is malfunctioning and sends
random and possibly dangerous data back to the PC, to be eval'd.

Fix this problem by using ast.literal_eval() to parse the received bytes,
instead of eval().

Signed-off-by: Michael Buesch <m@bues.ch>
2020-08-21 16:08:03 +10:00
Dave Hylands 8727c4e2ec stm32/pin_defs_stm32: Fix pin printing to show IN mode correctly.
Prior to this commit, if you configure a pin as an output type (I2C in this
example) and then later configure it back as an input, then it will report
the type incorrectly.  Example:

    >>> import machine
    >>> b6 = machine.Pin('B6')
    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.IN)
    >>> machine.I2C(1)
    I2C(1, scl=B6, sda=B7, freq=420000)
    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.ALT_OPEN_DRAIN, pull=Pin.PULL_UP, af=Pin.AF4_I2C1)
    >>> b6.init(machine.Pin.IN)
    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.ALT_OPEN_DRAIN, af=Pin.AF4_I2C1)

With this commit the last print now works:

    >>> b6
    Pin(Pin.cpu.B6, mode=Pin.IN)
2020-08-21 13:42:47 +10:00
Zenix27 e76c7466b6 docs: Change `\*` to `*` in argument lists.
Latest versions of Sphinx (at least 3.1.0) do not need the `*` escaped and
will render the `\` in the output if it is there, so remove it.

Fixes issue #6209.
2020-08-21 12:24:13 +10:00
Maureen Helm 17689a71f6 travis: Add zephyr build to CI.
Adds a job to build the zephyr port in CI using the same docker container
that the zephyr project uses for its own CI.

Always make clean zephyr builds to ensure we don't just rebuild C code, but
we also rebuild Kconfig and dts.  This is required when switching between
boards, which have different Kconfigs and device trees.

Uses the tagged zephyr 2.3.0 release.

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
2020-08-21 11:37:10 +10:00
Maureen Helm ac94e06f0b zephyr: Include storage/flash_map.h unconditionally.
Include storage/flash_map.h unconditionally so we always have access to the
FLASH_AREA_LABEL_EXISTS macro, even if CONFIG_FLASH_MAP is not defined.

This fixes a build error for the qemu_x86 board:

main.c:108:63: error: missing binary operator before token "("
  108 |     #elif defined(CONFIG_FLASH_MAP) && FLASH_AREA_LABEL_EXISTS(storage)
      |                                                               ^
../../py/mkrules.mk:88: recipe for target 'build/genhdr/qstr.i.last' failed

Signed-off-by: Maureen Helm <maureen.helm@nxp.com>
2020-08-21 11:23:43 +10:00
Damien George 60f5b941e0 extmod/vfs_reader: Fix mp_reader_new_file to open file in "rb" mode.
mp_reader_new_file() is used to read in files for importing, either .py or
.mpy files, for the lexer and persistent code loader respectively.  In both
cases the file should be opened in raw bytes mode: the lexer handles
unicode characters itself, and .mpy files contain 8-bit bytes by nature.

Before this commit importing was working correctly because, although the
file was opened in text mode, all native filesystem implementations (POSIX,
FAT, LFS) would access the file in raw bytes mode via mp_stream_rw()
calling mp_stream_p_t.read().  So it was only an issue for non-native
filesystems, such as those implemented in Python.  For Python-based
filesystem implementations, a call to mp_stream_rw() would go via IOBase
and then to readinto() at the Python level, and readinto() is only defined
on files opened in raw bytes mode.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-12 23:40:50 +10:00
Damien George b731bd0ce6 tools/makemanifest.py: Print nicely formatted errors from mpy-cross.
If mpy-cross exits with an error be sure to print that error in a way that
is readable, instead of a long bytes object.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-08 14:46:05 +10:00
Damien George 9883d8e818 py/persistentcode: Maintain root ptr list of imported native .mpy code.
On ports where normal heap memory can contain executable code (eg ARM-based
ports such as stm32), native code loaded from an .mpy file may be reclaimed
by the GC because there's no reference to the very start of the native
machine code block that is reachable from root pointers (only pointers to
internal parts of the machine code block are reachable, but that doesn't
help the GC find the memory).

This commit fixes this issue by maintaining an explicit list of root
pointers pointing to native code that is loaded from an .mpy file.  This
is not needed for all ports so is selectable by the new configuration
option MICROPY_PERSISTENT_CODE_TRACK_RELOC_CODE.  It's enabled by default
if a port does not specify any special functions to allocate or commit
executable memory.

A test is included to test that native code loaded from an .mpy file does
not get reclaimed by the GC.

Fixes #6045.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-02 22:34:09 +10:00
Damien George 8da40baa47 tests/micropython: Improve .mpy import tests to run on more targets.
All imports are now tested to see if the test should be skipped,
UserFile.read is removed, and UserFile.readinto is made more efficient.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-26 22:44:59 +10:00
Damien George 0c0cef9870 tests: Move .mpy import tests from import/ to micropython/ dir.
These tests are specific to MicroPython so have a better home in the
micropython/ test subdir, and putting them here allows them to be run by
all targets, not just those that have access to the local filesystem (eg
the unix port).

Signed-off-by: Damien George <damien@micropython.org>
2020-07-26 22:04:31 +10:00
Damien George 952de5cb77 tools/makemanifest.py: Use errno.EEXIST instead of number 17.
To make this code more portable, across different platforms.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-26 10:56:24 +10:00
Damien George 441460d81f extmod/uasyncio: Add StreamReader.readexactly(n) method.
It raises on EOFError instead of an IncompleteReadError (which is what
CPython does).  But the latter is derived from EOFError so code compatible
with MicroPython and CPython can be written by catching EOFError (eg see
included test).

Fixes issue #6156.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-25 23:10:05 +10:00
Damien George fd2ff867a0 stm32/usbdev: Fix calculation of SCSI LUN size with multiple LUNs.
The SCSI driver calls GetCapacity to get the block size and number of
blocks of the underlying block-device/LUN.  It caches these values and uses
them later on to verify that reads/writes are within the bounds of the LUN.
But, prior to this commit, there was only one set of cached values for all
LUNs, so the bounds checking for a LUN could use incorrect values, values
from one of the other LUNs that most recently updated the cached values.
This would lead to failed SCSI requests.

This commit fixes this issue by having separate cached values for each LUN.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-25 01:12:07 +10:00
Jonathan Hogg 37e1b5c891 py/compile: Don't await __aiter__ special method in async-for.
MicroPython's original implementation of __aiter__ was correct for an
earlier (provisional) version of PEP492 (CPython 3.5), where __aiter__ was
an async-def function.  But that changed in the final version of PEP492 (in
CPython 3.5.2) where the function was changed to a normal one.  See
https://www.python.org/dev/peps/pep-0492/#why-aiter-does-not-return-an-awaitable
See also the note at the end of this subsection in the docs:
https://docs.python.org/3.5/reference/datamodel.html#asynchronous-iterators
And for completeness the BPO: https://bugs.python.org/issue27243

To be consistent with the Python spec as it stands today (and now that
PEP492 is final) this commit changes MicroPython's behaviour to match
CPython:  __aiter__ should return an async-iterable object, but is not
itself awaitable.

The relevant tests are updated to match.

See #6267.
2020-07-25 00:58:18 +10:00
Josh Lloyd fe7d47971f docs/esp32: Fix machine.Timer quickref to specify HW timers.
Also remove trailing spaces on other lines.
2020-07-25 00:50:06 +10:00
Howard Lovatt 47289f4bc9 docs/library: Update pyb.UART to correct pyboard UART availability.
On original pyboard UART 5 isn't available; added pyboard D availability.
2020-07-25 00:46:29 +10:00
Howard Lovatt 4a7c2731c5 docs/library: Update pyb.Timer to add missing args and defaults to init. 2020-07-25 00:34:48 +10:00
Howard Lovatt cf9be201d7 docs/library: Update pyb.SPI init method to add descr about "ti" arg. 2020-07-25 00:28:16 +10:00
Glenn Ruben Bakke caaaa2b1f4 nrf: Enable more features for all targets.
Enabling the following features for all targets, except for nrf51
targets compiled to be used with SoftDevice:

- MICROPY_PY_ARRAY_SLICE_ASSIGN
- MICROPY_PY_SYS_STDFILES
- MICROPY_PY_UBINASCII
2020-07-22 12:54:20 +02:00