Commit Graph

4593 Commits

Author SHA1 Message Date
Dan Halbert 8b7c23c1ee address review comments 2020-12-01 20:01:14 -05:00
Mark 237385798c
Merge branch 'main' into bus_device 2020-12-01 15:47:16 -06:00
Scott Shawcroft a975ef4971
Merge pull request #3695 from cwalther/movable
Add movable supervisor allocations
2020-11-30 16:00:55 -08:00
Dan Halbert 9768951a2a Disable complex arithmetic on SAMD21 builds to make space 2020-11-29 15:34:38 -05:00
Damien George be24e6a53f py/mpprint: Prevent case fall-through when assert is disabled.
Signed-off-by: Damien George <damien@micropython.org>
2020-11-29 17:31:24 +11:00
Christian Walther bde1c4166d Revert "Prevent exceptions from accumulating in REPL"
This reverts commit 0cd951fb73.

It is not a correct solution because it prevents printing the same exception twice.
2020-11-28 23:10:17 +01:00
Christian Walther 2ba9805f84 Use movable allocation system for terminal tilegrid.
Moving memory is now done by the infrastructure and neither necessary nor correct here anymore.
2020-11-28 17:54:34 +01:00
Christian Walther c7404a3ff8 Add movable allocation system.
This allows calls to `allocate_memory()` while the VM is running, it will then allocate from the GC heap (unless there is a suitable hole among the supervisor allocations), and when the VM exits and the GC heap is freed, the allocation will be moved to the bottom of the former GC heap and transformed into a proper supervisor allocation. Existing movable allocations will also be moved to defragment the supervisor heap and ensure that the next VM run gets as much memory as possible for the GC heap.

By itself this breaks terminalio because it violates the assumption that supervisor_display_move_memory() still has access to an undisturbed heap to copy the tilegrid from. It will work in many cases, but if you're unlucky you will get garbled terminal contents after exiting from the vm run that created the display. This will be fixed in the following commit, which is separate to simplify review.
2020-11-28 17:50:23 +01:00
Dan Halbert 28d9e9186e Disable complex arithmetic on SAMD21 builds to make space 2020-11-28 10:12:46 -05:00
Dan Halbert ef0830bfe2 merge from upstream + wip 2020-11-25 17:52:06 -05:00
Dan Halbert 9dbea36eac changed alarm.time API 2020-11-25 15:09:27 -05:00
Jeff Epler e778fc1f87
Merge pull request #3741 from hathach/fix-cdc-connection-race
update tinyusb to fix cdc connection race
2020-11-24 19:11:41 -06:00
Jeff Epler c451b22255 Disable 3-arg pow() function on m0 boards
`pow(a, b, c)` can compute `(a ** b) % c` efficiently (in time and memory).
This can be useful for extremely specific applications, like implementing
the RSA cryptosystem.  For typical uses of CircuitPython, this is not an
important feature.  A survey of the bundle and learn system didn't find
any uses.

Disable it on M0 builds so that we can fit in needed upgrades to the USB
stack.
2020-11-24 16:54:33 -06:00
Dan Halbert 7a45afc549 working, but need to avoid deep sleeping too fast before USB ready 2020-11-23 22:44:53 -05:00
Scott Shawcroft f8dcb25170
Merge pull request #3694 from jepler/update-ulab2
ulab: Update to release tag 1.1.0
2020-11-23 15:17:46 -08:00
Jeff Epler 9d8be648ee ulab: Update to release tag 1.1.0
Disable certain classes of diagnostic when building ulab.  We should
submit patches upstream to (A) fix these errors and (B) upgrade their
CI so that the problems are caught before we want to integrate with
CircuitPython, but not right now.
2020-11-23 10:23:50 -06:00
Jim Mussared 0e8af2b370 extmod/modbluetooth: Add API for L2CAP channels.
Also known as L2CAP "connection oriented channels". This provides a
socket-like data transfer mechanism for BLE.

Currently only implemented for NimBLE on STM32 / Unix.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-24 01:07:17 +11:00
Dan Halbert 75559f35cc wip: ResetReason to microcontroller.cpu 2020-11-21 23:29:52 -05:00
Dan Halbert e4c66990e2 compiles 2020-11-20 23:33:39 -05:00
Jeff Epler 982bce7259 py.mk: allow translation to be overriden in GNUmakefile
I like to use local makefile overrides, in the file GNUmakefile
(or, on case-sensitive systems, makefile) to set compilation choices.
However, writing
    TRANSLATION := de_DE
    include Makefile
did not work, because py.mk would override the TRANSLATION := specified
in an earlier part of the makefiles (but not from the commandline).

By using ?= instead of := the local makefile override works, but when
TRANSLATION is not specified it continues to work as before.
2020-11-19 16:23:35 -06:00
Jeff Epler b2b8520880 Always use preprocessor for MICROPY_ERROR_REPORTING
This ensures that only the translate("") alternative that will be used
is seen after preprocessing.  Improves the quality of the Huffman encoding
and reduces binary size slightly.

Also makes one "enhanced" error message only occur when ERROR_REPORTING_DETAILED:
Instead of the word-for-word python3 error message
"Type object has no attribute '%q'", the message will be
"'type' object has no attribute '%q'".  Also reduces binary size.
(that's rolled into this commit as it was right next to a change to
use the preprocessor for MICROPY_ERROR_REPORTING)

Note that the odd semicolon after "value_error:" in parsenum.c is necessary
due to a detail of the C grammar, in which a declaration cannot follow
a label directly.
2020-11-19 16:18:52 -06:00
Jeff Epler c06fc8e02d Introduce, use mp_raise_arg1
This raises an exception with a given object value.  Saves a bit of
code size.
2020-11-19 16:15:06 -06:00
Jeff Epler d5f6748d1b Use mp_raise instead of nlr_raise(new_exception) where possible
This saves a bit of code space
2020-11-19 16:13:01 -06:00
Jeff Epler 0556f9f851 Revert "samd21: Enable terse error reporting on resource constrained chip family"
This reverts commit 9a642fc049.
2020-11-19 15:12:56 -06:00
Dan Halbert 649c930536 wip 2020-11-19 15:43:39 -05:00
Dan Halbert 5bb3c321e9 merge from main 2020-11-19 00:29:14 -05:00
Dan Halbert 682054a216 WIP: redo API; not compiled yet 2020-11-19 00:23:27 -05:00
Jeff Epler 9a642fc049 samd21: Enable terse error reporting on resource constrained chip family
This reclaims over 1kB of flash space by simplifying certain exception
messages.  e.g., it will no longer display the requested/actual length
when a fixed list/tuple of N items is needed:

        if (MICROPY_ERROR_REPORTING == MICROPY_ERROR_REPORTING_TERSE) {
            mp_raise_ValueError(translate("tuple/list has wrong length"));
        } else {
            mp_raise_ValueError_varg(translate("requested length %d but object has length %d"),
                (int)len, (int)seq_len);

Other chip families including samd51 keep their current error reporting
capabilities.
2020-11-18 20:37:36 -06:00
Dan Halbert ffff02c053 Merge remote-tracking branch 'adafruit/main' into sleep 2020-11-16 12:06:11 -05:00
Dan Halbert bb77f1d130 wip: initial code changes, starting from @tannewt's sleepio branch 2020-11-16 11:56:20 -05:00
root 0cd951fb73 Prevent exceptions from accumulating in REPL 2020-11-16 10:36:05 -06:00
Scott Shawcroft bda3267432
Save flash space
* No weak link for modules. It only impacts _os and _time and is
  already disabled for non-full builds.
* Turn off PA00 and PA01 because they are the crystal on the Metro
  M0 Express.
* Change ejected default to false to move it to BSS. It is set on
  USB connection anyway.
* Set sinc_filter to const. Doesn't help flash but keeps it out of
  RAM.
2020-11-13 18:57:52 -08:00
Jim Mussared a7932ae4e6 tools/makeqstrdefs.py: Run qstr preprocessing in parallel.
This gives a substantial speedup of the preprocessing step, i.e. the
generation of qstr.i.last.  For example on a clean build, making
qstr.i.last:

    21s -> 4s on STM32 (WB55)
    8.9 -> 1.8s on Unix (dev).

Done in collaboration with @stinos.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-11-12 15:04:53 +11:00
Damien George d7e1526593 py/binary: Fix sign extension setting wide integer on 32-bit archs.
Signed-off-by: Damien George <damien@micropython.org>
2020-11-11 22:18:24 +11:00
Damien George 1fef5662ab py/mpz: Do sign extension in mpz_as_bytes for negative values.
Signed-off-by: Damien George <damien@micropython.org>
2020-11-11 22:18:24 +11:00
gamblor21 4c93db3595 Renamed to adafruit_bus_device 2020-11-03 18:35:20 -06:00
Dan Halbert 72b829dff0 add binascii to most builds 2020-11-01 14:52:03 -05:00
gamblor21 78477a374a Initial SPI commit 2020-10-31 12:17:29 -05:00
stijn 0153148fd2 py/py.mk: Support C++ code for user C modules.
Support C++ code in .cpp files by providing CXX counterparts of the
_USERMOD_ flags we have for C already.  This merely enables the Makefile of
user C modules to use variables specific to C++ compilation, it is still up
to each port's main Makefile to also include these in the build.
2020-10-29 15:29:20 +11:00
stijn e498a8bd13 py: Workaround clang error when building misc.h with C++ compiler. 2020-10-29 15:28:14 +11:00
stijn 8e94fa0d2e py/makeqstrdefs.py: Support preprocessing C++ files for QSTR generation.
When SCR_QSTR contains C++ files they should be preprocessed with the same
compiler flags (CXXFLAGS) as they will be compiled with, to make sure code
scanned for QSTR occurrences is effectively the code used in the rest of
the build.  The 'split SCR_QSTR in .c and .cpp files and process each with
different flags' logic isn't trivial to express in a Makefile and the
existing principle for deciding which files to preprocess was already
rather complicated, so the actual preprocessing is moved into
makeqstrdefs.py completely.
2020-10-29 15:27:30 +11:00
stijn f1666419a8 py/mkrules.mk: Add target for compiling C++ files.
Move the target from the ESP32 Makefile since that does what is needed
already, but also include files from user C modules as is done for the C
files.
2020-10-29 15:27:18 +11:00
stijn 2b9f0586e7 py/makeqstrdefs.py: Process C++ files as well.
Preprocessed C++ code isn't different from C code when it comes to QSTR
instances so process it as well.
2020-10-29 15:27:11 +11:00
stijn 1b723937e3 py/makeqstrdefs.py: Fix beaviour when scanning non-C preprocessed files.
When process_file() is passed a preprocessed C++ file for instance it won't
find any lines containing .c files and the last_fname variable remains
None, so handle that gracefully.
2020-10-29 15:26:35 +11:00
microDev 930cf14dce
Add check for invalid io, function to disable all alarms 2020-10-27 16:17:26 -07:00
microDev 59df1a11ad
Add alarm_touch module 2020-10-27 16:16:52 -07:00
microDev da449723df
Fix build error 2020-10-27 16:16:15 -07:00
microDev 4d8ffdca8d
restructure alarm modules 2020-10-27 16:15:09 -07:00
microDev e5ff55b15c
Renamed alarm modules 2020-10-27 16:13:25 -07:00
microDev e310b871c8
Get io wake working 2020-10-27 16:13:25 -07:00
microDev 90b9ec6f2c
Initial Sleep Support 2020-10-27 16:13:22 -07:00
gamblor21 b637d3911e Initial commit 2020-10-24 20:48:35 -05:00
Emil Renner Berthing ccd92335a1 py, extmod: Introduce and use MP_FALLTHROUGH macro.
Newer GCC versions are able to warn about switch cases that fall
through.  This is usually a sign of a forgotten break statement, but in
the few cases where a fall through is intended we annotate it with this
macro to avoid the warning.
2020-10-22 11:53:16 +02:00
Emil Renner Berthing f1f6ef7b17 py/vmentrytable: Ignore GCC -Woverride-init.
Like Clang, GCC warns about this file, but only with -Woverride-init
which is enabled by -Wextra. Disable the warnings for this file just
like we do for Clang to make -Wextra happy.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing 9aa58cf8ba py, extmod: Add explicit initializers for default values.
When compiling with -Wextra which includes -Wmissing-field-initializers
GCC will warn that the defval field of mp_arg_val_t is not initialized.
This is just a warning as it is defined to be zero initialized, but since
it is a union it makes sense to be explicit about which member we're
going to use, so add the explicit initializers and get rid of the
warning.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing fdd6fa389e py: Use unsigned comparison of chars.
On x86 chars are signed, but we're comparing a char to '0' + unsigned int,
which is promoted to an unsigned int. Let's promote the char to unsigned
before doing the comparison to avoid weird corner cases.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing 6d3aa16443 py/objexcept: Compare mp_emergency_exception_buf_size signed.
mp_emergency_exception_buf_size is signed, so let's make sure we compare
it as such.
2020-10-22 11:47:36 +02:00
Emil Renner Berthing 6324c3e054 py/scope: Name and use id_kind_type_t.
The function scope_find_or_add_id used to take a scope_kind_t enum and
save it in an uint8_t. Saving an enum in a uint8_t is fine, but
everywhere this function is called it is not actually given a
scope_kind_t but an anonymous enum instead. Let's give this enum a name
and use that as the argument type.

This doesn't change the generated code, but is a C type mismatch that
unfortunately doesn't show up unless you enable -Wenum-conversion.
2020-10-22 11:40:56 +02:00
Christian Walther 1eab0692b5 Fix missing `nproc` on macOS.
396979a breaks building on macOS: `nproc` is a Linux thing, use a cross-platform alternative.
2020-10-20 16:39:32 +02:00
Dan Halbert 82b49afe43 enable CIRCUITPY_BLEIO_HCI on non-nRF boards where it will fit 2020-10-15 11:27:21 -04:00
Dan Halbert f1e8f2b404
Merge pull request #3554 from gamblor21/move_ordereddict
Moved ORDEREDDICT define to central location
2020-10-14 22:39:04 -04:00
gamblor21 f6f89565d8 Remove ordered dict from SAMD21 2020-10-14 20:18:49 -05:00
gamblor21 e6d0b207ec Removed MICROPY_PY_COLLECTIONS_NAMEDTUPLE__ASDICT from unix coverage 2020-10-14 14:06:34 -05:00
gamblor21 4270061db4 Moved ORDEREDDICT define to central location 2020-10-13 18:52:27 -05:00
Scott Shawcroft 9de96786ad
Merge pull request #3538 from jepler/parallel-qstrlast
build: parallelize the qstr build steps
2020-10-12 15:52:43 -07:00
Scott Shawcroft 1eb1434fc9
Merge pull request #3537 from jepler/update-protomatter-2
rgbmatrix: update protomatter to 1.0.5 tag
2020-10-12 15:45:51 -07:00
Scott Shawcroft 179e13f103
Merge pull request #3539 from jepler/lto-type-mismatch
remove warning-disable flag that seems unneeded now
2020-10-12 15:44:25 -07:00
Kenny 94beeabc51 remove unnecessary board configuration and address feedback 2020-10-11 22:42:59 -07:00
Jeff Epler 479552ce56 build: Make genlast write the "split" files
This gets a further speedup of about 2s (12s -> 9.5s elapsed build time)
for stm32f405_feather

For what are probably historical reasons, the qstr process involves
preprocessing a large number of source files into a single "qstr.i.last"
file, then reading this and splitting it into one "qstr" file for each
original source ("*.c") file.

By eliminating the step of writing qstr.i.last as well as making the
regular-expression-matching part be parallelized, build speed is further
improved.

Because the step to build QSTR_DEFS_COLLECTED does not access
qstr.i.last, the path is replaced with "-" in the Makefile.
2020-10-11 21:18:03 -05:00
Jeff Epler 607e4a905a build: parallelize the creation of qstr.i.last
Rather than simply invoking gcc in preprocessor mode with a list of files, use
a Python script with the (python3) ThreadPoolExecutor to invoke the
preprocessor in parallel.

The amount of concurrency is the number of system CPUs, not the makefile "-j"
parallelism setting, because there is no simple and correct way for a Python
program to correctly work together with make's idea of parallelism.

This reduces the build time of stm32f405 feather (a non-LTO build) from 16s to
12s on my 16-thread Ryzen machine.
2020-10-11 20:19:59 -05:00
Jeff Epler c139eccc92 remove warning that seems unneeded now 2020-10-11 16:23:02 -05:00
Kenny 98aa4b7943 update async tests with less upython workaround and more cpython compatibility 2020-10-10 23:39:32 -07:00
Kenny 5d96afc5c2 i do not know if this is needed but this is not the vm i use anymore 2020-10-10 15:45:08 -07:00
Kenny bf849ff674 async def syntax rigor and __await__ magic method
Some examples of improved compliance with CPython that currently
have divergent behavior in CircuitPython are listed below:

* yield from is not allowed in async methods
```
>>> async def f():
...     yield from 'abc'
...
Traceback (most recent call last):
  File "<stdin>", line 2, in f
SyntaxError: 'yield from' inside async function
```

* await only works on awaitable expressions
```
>>> async def f():
...     await 'not awaitable'
...
>>> f().send(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
AttributeError: 'str' object has no attribute '__await__'
```

* only __await__()able expressions are awaitable
Okay this one actually does not work in circuitpython at all today.
This is how CPython works though and pretending __await__ does not
exist will only bite users who write both.
```
>>> class c:
...     pass
...
>>> def f(self):
...     yield
...     yield
...     return 'f to pay respects'
...
>>> c.__await__ = f  # could just as easily have put it on the class but this shows how it's wired
>>> async def g():
...     awaitable_thing = c()
...     partial = await awaitable_thing
...     return 'press ' + partial
...
>>> q = g()
>>> q.send(None)
>>> q.send(None)
>>> q.send(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: press f to pay respects
```
2020-10-10 15:45:08 -07:00
warriorofwire 5cadf525bd fix missing cflag defeating the board gating 2020-10-10 15:45:08 -07:00
warriorofwire d94d2d2975 Add async/await syntax to FULL_BUILD
This adds the `async def` and `await` verbs to valid CircuitPython syntax using the Micropython implementation.

Consider:
```
>>> class Awaitable:
...     def __iter__(self):
...         for i in range(3):
...             print('awaiting', i)
...             yield
...         return 42
...
>>> async def wait_for_it():
...     a = Awaitable()
...     result = await a
...     return result
...
>>> task = wait_for_it()
>>> next(task)
awaiting 0
>>> next(task)
awaiting 1
>>> next(task)
awaiting 2
>>> next(task)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  StopIteration: 42
>>>
```

and more excitingly:
```
>>> async def it_awaits_a_subtask():
...     value = await wait_for_it()
...     print('twice as good', value * 2)
...
>>> task = it_awaits_a_subtask()
>>> next(task)
awaiting 0
>>> next(task)
awaiting 1
>>> next(task)
awaiting 2
>>> next(task)
twice as good 84
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  StopIteration:
```

Note that this is just syntax plumbing, not an all-encompassing implementation of an asynchronous task scheduler or asynchronous hardware apis.
  uasyncio might be a good module to bring in, or something else - but the standard Python syntax does not _strictly require_ deeper hardware
  support.
Micropython implements the await verb via the __iter__ function rather than __await__.  It's okay.

The syntax being present will enable users to write clean and expressive multi-step state machines that are written serially and interleaved
  according to the rules provided by those users.

Given that this does not include an all-encompassing C scheduler, this is expected to be an advanced functionality until the community settles
  on the future of deep hardware support for async/await in CircuitPython.  Users will implement yield-based schedulers and tasks wrapping
  synchronous hardware APIs with polling to avoid blocking, while their application business logic gets simple `await` statements.
2020-10-10 15:38:40 -07:00
Jeff Epler 5e38bb98cb rgbmatrix: update protomatter to 1.0.5 tag
this is compile-tested on
 stm32f405 feather
 matrixportal
 nrf52840 feather

but not actually tested-tested.
2020-10-10 14:30:37 -05:00
Jim Mussared b137d064e9 py/objtype: Handle __dict__ attribute when type has no locals. 2020-10-10 00:16:32 +11:00
Jim Mussared 880875bea1 py/objdict: Add mp_const_empty_dict_obj, use it for mp_const_empty_map. 2020-10-10 00:16:26 +11:00
Damien George 843dcd4f85 py/parse: Expose rule-name printing as MICROPY_DEBUG_PARSE_RULE_NAME.
So it can be enabled without modifying the source.

Signed-off-by: Damien George <damien@micropython.org>
2020-10-01 15:26:43 +10:00
Mike Wadsten c711c0049e py/makeversionhdr.py: Match only git tags which look like versions.
Some downstream projects may use tags in their repositories for more than
just designating MicroPython releases.  In those cases, the
makeversionhdr.py script would end up using a different tag than intended.
So tell `git describe` to only match tags that look like a MicroPython
version tag, such as `v1.12` or `v2.0`.
2020-10-01 11:01:43 +10:00
Jeff Epler a4cc3ad6cb canio: RemoteTransmissionRequest: Split implementation, keep one structure
This already begins obscuring things, because now there are two sets of
shared-module functions for manipulating the same structure, e.g.,
common_hal_canio_remote_transmission_request_get_id and
common_hal_canio_message_get_id
2020-09-28 17:22:00 -05:00
Damien George 50e34f979c py/objarray.h: Add mp_obj_memoryview_init() helper function.
Signed-off-by: Damien George <damien@micropython.org>
2020-09-25 12:23:11 +10:00
Iyassou Shimels ca017841d6 py/objstr: Make bytes(bytes_obj) return bytes_obj.
Calling the bytes constructor on a bytes object returns the original bytes
object.  This saves allocating a new instance, and matches CPython.

Signed-off-by: Iyassou Shimels <s.iyassou@gmail.com>
2020-09-24 11:04:58 +10:00
Scott Shawcroft a8558a48ed
Merge pull request #3456 from jepler/qstr-and-or-demagic
makeqstrdefs: don't make _and_, _or_ poisoned substrings for QSTRs
2020-09-23 12:24:02 -07:00
Jeff Epler 28e80e47d7 makeqstrdefs: don't make _and_, _or_ poisoned substrings for QSTRs
New contributor @mdroberts1243 encountered an interesting problem in
which the argument they had named "column_underscore_and_page_addressing"
simply couldn't be used; I discovered that internally this had been
transformed into "column_underscore∧page_addressing", because QSTR
makes _ENTITY_ stand for the same thing as &ENTITY; does in HTML.

This might be nice for some things, but we don't want it here!
I was unable to find a sensible way to "escape" and prevent this entity
coding, so instead I ripped out support for the _and_ and _or_ escapes.
2020-09-22 17:39:44 -05:00
Jeff Epler 4869dbdc67 canio: rename from _canio
This reflects our belief that the API is stable enough to avoid incompatible changes during 6.x.
2020-09-21 16:44:26 -05:00
Jeff Epler a2e1867f69 _canio: Minimal implementation for SAM E5x MCUs
Tested & working:

 * Send standard packets
 * Receive standard packets (1 FIFO, no filter)

Interoperation between SAM E54 Xplained running this tree and
MicroPython running on STM32F405 Feather with an external
transceiver was also tested.

Many other aspects of a full implementation are not yet present,
such as error detection and recovery.
2020-09-21 16:44:26 -05:00
Jeff Epler e7a213a114 py: Add enum helper code
This makes it much easier to implement enums, and the printing code is
shared.  We might want to convert other enums to this in the future.
2020-09-21 16:44:26 -05:00
Jeff Epler 0318eb359f makeqstrdata: Work around python3.6 compatibility problem
Discord user Folknology encountered a problem building with Python 3.6.9,
`TypeError: ord() expected a character, but string of length 0 found`.

I was able to reproduce the problem using Python3.5*, and discovered that
the meaning of the regular expression `"|."` had changed in 3.7.  Before,
```
>>> [m.group(0) for m in re.finditer("|.", "hello")]
['', '', '', '', '', '']
```
After:
```
>>> [m.group(0) for m in re.finditer("|.", "hello")]
['', 'h', '', 'e', '', 'l', '', 'l', '', 'o', '']
```
Check if `words` is empty and if so use `"."` as the regular expression
instead.  This gives the same result on both versions:
```
['h', 'e', 'l', 'l', 'o']
```
and fixes the generation of the huffman dictionary.

Folknology verified that this fix worked for them.

 * I could easily install 3.5 but not 3.6.  3.5 reproduced the same problem
2020-09-21 10:03:07 -05:00
Jeff Epler bfbbbd6c5c makeqstrdata: Work with older Python
This construct (which I added without sufficient testing,
apparently) is only supported in Python 3.7 and newer.  Make it
optional so that this script works on other Python versions.  This
means that if you have a system with non-UTF-8 encoding you will
need to use Python 3.7.

In particular, this affects a problem building circuitpython in
github's ubuntu-18.04 virtual environment when Python 3.7 is not
explicitly installed.  cookie-cuttered libraries call for Python
3.6:
```
    - name: Set up Python 3.6
      uses: actions/setup-python@v1
      with:
        python-version: 3.6
```
Since CircuitPython's own build calls for 3.8, this problem was not
detected.

This problem was also encountered by discord user mdroberts1243.

The failure I encountered was here:
https://github.com/jepler/Jepler_CircuitPython_udecimal/runs/1138045020?check_suite_focus=true
.. while my step of "clone and build circuitpython unix port" is
unusual, I think the same problem would have affected "build assets"
if that step had been reached.
2020-09-19 10:16:13 -05:00
Jim Mussared 9d1983f078 py/dynruntime.h: Add mp_import_* and mp_load/store_*.
These functions already exist in the fun table, and this commit just adds
convenience macros for them.
2020-09-18 18:34:02 +10:00
Damien George 8f20cdc353 all: Rename absolute time-based functions to include "epoch".
For time-based functions that work with absolute time there is the need for
an Epoch, to set the zero-point at which the absolute time starts counting.
Such functions include time.time() and filesystem stat return values.  And
different ports may use a different Epoch.

To make it clearer what functions use the Epoch (whatever it may be), and
make the ports more consistent with their use of the Epoch, this commit
renames all Epoch related functions to include the word "epoch" in their
name (and remove references to "2000").

Along with this rename, the following things have changed:

- mp_hal_time_ns() is now specified to return the number of nanoseconds
  since the Epoch, rather than since 1970 (but since this is an internal
  function it doesn't change anything for the user).

- littlefs timestamps on the esp8266 have been fixed (they were previously
  off by 30 years in nanoseconds).

Otherwise, there is no functional change made by this commit.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-18 17:20:34 +10:00
Scott Shawcroft 750bc1e04a
Merge pull request #3398 from jepler/better-dictionary-compression
compression: Implement @ciscorn's dictionary approach
2020-09-16 11:10:22 -07:00
Jeff Epler a8e98cda83 makeqstrdata: comment my understanding of @ciscorn's code 2020-09-16 08:28:15 -05:00
Kamil Tomaszewski c2fc592c2c camera: Change API 2020-09-14 13:11:15 +02:00
Kamil Tomaszewski 064c597b60 camera: Implement new library for camera 2020-09-14 13:11:15 +02:00
Jeff Epler 90f7340bfc move implicit-fallthrough warning enable to defns.mk 2020-09-13 13:13:09 -05:00
Taku Fukada d18d79ac47 Small improvements to the dictionary compression 2020-09-14 01:50:01 +09:00
Jeff Epler 15964a4750 makeqstrdata: Avoid encoding problems
Most users and the CI system are running in configurations where Python
configures stdout and stderr in UTF-8 mode.  However, Windows is different,
setting values like CP1252.  This led to a build failure on Windows, because
makeqstrdata printed Unicode strings to its stdout, expecting them to be
encoded as UTF-8.

This script is writing (stdout) to a compiler input file and potentially
printing messages (stderr) to a log or console.  Explicitly configure stdout to
use utf-8 to get consistent behavior on all platforms, and configure stderr so
that if any log/diagnostic messages are printed that cannot be displayed
correctly, they are still displayed instead of creating an error while trying
to print the diagnostic information.

I considered setting the encodings both to ascii, but this would just be
occasionally inconvenient to developers like me who want to show diagnostic
info on stderr and in comments while working with the compression code.

Closes: #3408
2020-09-12 19:43:08 -05:00
Jeff Epler 12d826d941 Add FALLTHROUGH comments as needed
I investigated these cases and confirmed that the fallthrough behavior
was intentional.
2020-09-12 15:11:29 -05:00
Jeff Epler 54d97251fe modstruct: Improve compliance with python3
While checking whether we can enable -Wimplicit-fallthrough, I encountered
a diagnostic in mp_binary_set_val_array_from_int which led to discovering
the following bug:
```
>>> struct.pack("xb", 3)
b'\x03\x03'
```
That is, the next value (3) was used as the value of a padding byte, while
standard Python always fills "x" bytes with zeros.  I initially thought
this had to do with the unintentional fallthrough, but it doesn't.
Instead, this code would relate to an array.array with a typecode of
padding ('x'), which is ALSO not desktop Python compliant:
```
>>> array.array('x', (1, 2, 3))
array('x', [1, 0, 0])
```
Possibly this is dead code that used to be shared between struct-setting
and array-setting, but it no longer is.

I also discovered that the argument list length for struct.pack
and struct.pack_into were not checked, and that the length of binary data
passed to array.array was not checked to be a multiple of the element
size.

I have corrected all of these to conform more closely to standard Python
and revised some tests where necessary.  Some tests for micropython-specific
behavior that does not conform to standard Python and is not present
in CircuitPython was deleted outright.
2020-09-12 14:07:23 -05:00
Jeff Epler 40ab5c6b21 compression: Implement ciscorn's dictionary approach
Massive savings.  Thanks so much @ciscorn for providing the initial
code for choosing the dictionary.

This adds a bit of time to the build, both to find the dictionary
but also because (for reasons I don't fully understand), the binary
search in the compress() function no longer worked and had to be
replaced with a linear search.

I think this is because the intended invariant is that for codebook
entries that encode to the same number of bits, the entries are ordered
in ascending value.  However, I mis-placed the transition from "words"
to "byte/char values" so the codebook entries for words are in word-order
rather than their code order.

Because this price is only paid at build time, I didn't care to determine
exactly where the correct fix was.

I also commented out a line to produce the "estimated total memory size"
-- at least on the unix build with TRANSLATION=ja, this led to a build
time KeyError trying to compute the codebook size for all the strings.
I think this occurs because some single unicode code point ('ァ') is
no longer present as itself in the compressed strings, due to always
being replaced by a word.

As promised, this seems to save hundreds of bytes in the German translation
on the trinket m0.

Testing performed:
 - built trinket_m0 in several languages
 - built and ran unix port in several languages (en, de_DE, ja) and ran
   simple error-producing codes like ./micropython -c '1/0'
2020-09-12 10:10:45 -05:00
Damien George acdb0608b7 py/parse: Pass in an mp_print_t to mp_parse_node_print.
So the output can be redirected if needed.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-11 23:00:03 +10:00
Damien George 85f2b239d8 py/showbc: Pass in an mp_print_t struct to all bytecode-print functions.
So the output can be redirected if needed.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-11 17:22:28 +10:00
stijn 2e54d9d146 py: Fix handling of NaN in certain pow implementations.
Adds a new compile-time option MICROPY_PY_MATH_POW_FIX_NAN for use with
toolchains that don't handle pow-of-NaN correctly.
2020-09-11 10:04:57 +10:00
Damien George 8d5a40c86e py/objfloat: Fix handling of negative float to power of nan.
Prior to this commit, pow(-2, float('nan')) would return (nan+nanj), or
raise an exception on targets that don't support complex numbers.  This is
fixed to return simply nan, as CPython does.

Signed-off-by: Damien George <damien@micropython.org>
2020-09-11 10:03:57 +10:00
Scott Shawcroft 1ba28b3edc
Merge pull request #3370 from jepler/compression-bigrams
add bigram compression to makeqstrdata (save ~100 bytes on trinket m0 de_DE)
2020-09-10 11:44:56 -07:00
Scott Shawcroft 683462c1b1
Merge pull request #3326 from tannewt/native_wifi
Add native wifi API with ESP32S2 support
2020-09-10 11:20:44 -07:00
Jeff Epler bdb07adfcc translations: Make decompression clearer
Now this gets filled in with values e.g., 128 (0x80) and 159 (0x9f).
2020-09-08 19:07:53 -05:00
Jeff Epler 73858ea682 circuitpy_mpconfig: enable 3-arg pow() with CIRCUITPY_FULL_BUILD
This is needed for a port of python3's decimal.py module.
2020-09-06 10:07:57 -05:00
Jeff Epler 20c2dd0c08 core: add int.bit_length() when MICROPY_CYPTHON_COMPAT is enabled
This method of integer objects is needed for a port of python3's
decimal.py module.

MICROPY_CPYTHON_COMPAT is enabled by CIRCUITPY_FULL_BUILD.
2020-09-06 09:53:16 -05:00
Scott Shawcroft 96cf60fbbd
Merge remote-tracking branch 'adafruit/main' into native_wifi 2020-09-03 16:34:56 -07:00
Scott Shawcroft 0b94638aeb
Changes based on Dan's feedback 2020-09-03 16:32:12 -07:00
stijn 40ad8f1666 all: Rename "sys" module to "usys".
This is consistent with the other 'micro' modules and allows implementing
additional features in Python via e.g. micropython-lib's sys.

Note this is a breaking change (not backwards compatible) for ports which
do not enable weak links, as "import sys" must now be replaced with
"import usys".
2020-09-04 00:10:24 +10:00
Jeff Epler cbfd38d1ce Rename functions to encode_ngrams / decode_ngrams 2020-09-02 19:09:23 -05:00
Jeff Epler c34cb82ecb makeqstrdata: correct range of low code points to 0x80..0x9f inclusive
The previous range was unintentionally big and overlaps some characters
we'd like to use (and also 0xa0, which we don't intentionally use)
2020-09-02 15:52:02 -05:00
Damien George b0932fcf2e all: Bump version to 1.13.
Signed-off-by: Damien George <damien@micropython.org>
2020-09-02 12:01:26 +10:00
Jeff Epler 07740d19f3 add bigram compression to makeqstrdata
Compress common unicode bigrams by making code points in the range
0x80 - 0xbf (inclusive) represent them.  Then, they can be greedily
encoded and the substituted code points handled by the existing Huffman
compression.  Normally code points in the range 0x80-0xbf are not used
in Unicode, so we stake our own claim.  Using the more arguably correct
"Private Use Area" (PUA) would mean that for scripts that only use
code points under 256 we would use more memory for the "values" table.

bigram means "two letters", and is also sometimes called a "digram".
It's nothing to do with "big RAM".  For our purposes, a bigram represents
two successive unicode code points, so for instance in our build on
trinket m0 for english the most frequent are:
['t ', 'e ', 'in', 'd ', ...].

The bigrams are selected based on frequency in the corpus, but the
selection is not necessarily optimal, for these reasons I can think of:
 * Suppose the corpus was just "tea" repeated 100 times.  The
   top bigrams would be "te", and "ea".  However,
   overlap, "te" could never be used.  Thus, some bigrams might actually
   waste space
    * I _assume_ this has to be why e.g., bigram 0x86 "s " is more
      frequent than bigram 0x85 " a" in English for Trinket M0, because
      sequences like "can't add" would get the "t " digram and then
      be unable to use the " a" digram.

 * And generally, if a bigram is frequent then so are its constituents.
   Say that "i" and "n" both encode to just 5 or 6 bits, then the huffman
   code for "in" had better compress to 10 or fewer bits or it's a net
   loss!
    * I checked though!  "i" is 5 bits, "n" is 6 bits (lucky guess)
      but the bigram 0x83 also just 6 bits, so this one is a win of
      5 bits for every "it" minus overhead.  Yay, this round goes to team
      compression.
    * On the other hand, the least frequent bigram 0x9d " n" is 10 bits
      long and its constituent code points are 4+6 bits so there's no
      savings, but there is the cost of the table entry.
    * and somehow 0x9f 'an' is never used at all!

With or without accounting for overlaps, there is some optimum number
of bigrams.  Adding one more bigram uses at least 2 bytes (for the
entry in the bigram table; 4 bytes if code points >255 are in the
source text) and also needs a slot in the Huffman dictionary, so
adding bigrams beyond the optimim number makes compression worse again.

If it's an improvement, the fact that it's not guaranteed optimal
doesn't seem to matter too much.  It just leaves a little more fruit
for the next sweep to pick up.  Perhaps try adding the most frequent
bigram not yet present, until it doesn't improve compression overall.

Right now, de_DE is again the "fullest" build on trinket_m0.  (It's
reclaimed that spot from the ja translation somehow)  This change saves
104 bytes there, increasing free space about 6.8%.  In the larger
(but not critically full) pyportal build it saves 324 bytes.

The specific number of bigrams used (32) was chosen as it is the max
number that fit within the 0x80..0xbf range.  Larger tables would
require the use of 16 bit code points in the de_DE build, losing savings
overall.

(Side note: The most frequent letters in English have been said
to be: ETA OIN SHRDLU; but we have UAC EIL MOPRST in our corpus)
2020-09-01 17:12:22 -05:00
Scott Shawcroft f0e60da51f
Merge pull request #3310 from dhalbert/ble_hci
_bleio HCI implementation
2020-09-01 11:28:05 -07:00
Dan Halbert 6dbd369272 merge from upstream 2020-08-30 14:39:03 -04:00
Dan Halbert b27d511251 address review; use constructor for HCI Adapter 2020-08-30 14:06:48 -04:00
Jeff Epler 455226ffde builtinimport: Fix a crash with 'import ulab.linalg' on unix port only
A crash like the following occurs in the unix port:
```
Program received signal SIGSEGV, Segmentation fault.
0x00005555555a2d7a in mp_obj_module_set_globals (self_in=0x55555562c860 <ulab_user_cmodule>, globals=0x55555562c840 <mp_module_ulab_globals>) at ../../py/objmodule.c:145
145	    self->globals = globals;
(gdb) up
#1  0x00005555555b2781 in mp_builtin___import__ (n_args=5, args=0x7fffffffdbb0) at ../../py/builtinimport.c:496
496	                mp_obj_module_set_globals(outer_module_obj,
(gdb)
#2  0x00005555555940c9 in mp_import_name (name=824, fromlist=0x555555621f10 <mp_const_none_obj>, level=0x1) at ../../py/runtime.c:1392
1392	    return mp_builtin___import__(5, args);
```

I don't understand how it doesn't happen on the embedded ports, because
the module object should reside in ROM and the assignment of self->globals
should trigger a Hard Fault.

By checking VERIFY_PTR, we know that the pointed-to data is on the heap
so we can do things like mutate it.
2020-08-30 11:09:49 -05:00
Damien George 06659077a8 all: Update Python code to conform to latest black formatting.
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:

- If there is a trailing comma in a list (eg [], () or function call) then
  that list is now written out with one line per element.  So remove such
  trailing commas where the list should stay on one line.

- Spaces at the start of """ doc strings are removed.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-29 15:18:01 +10:00
Scott Shawcroft 767ca5c3dc
Merge remote-tracking branch 'adafruit/main' into native_wifi 2020-08-27 11:42:31 -07:00
Jeff Epler 2e0a109331
Merge pull request #3318 from jepler/interrupt-serial-rx
supervisor: check for interrupt during rx_chr
2020-08-25 21:01:33 -05:00
Scott Shawcroft 8b71e26abd
Merge remote-tracking branch 'adafruit/main' into native_wifi 2020-08-25 16:39:23 -07:00
Jeff Epler c0753c1afb mp_obj_print_helper: Handle a ctrl-c that comes in during printing
In #2689, hitting ctrl-c during the printing of an object with a lot of sub-objects could cause the screen to stop updating (without showing a KeyboardInterrupt).  This makes the printing of such objects acutally interruptable, and also correctly handles the KeyboardInterrupt:

```
>>> l = ["a" * 100] * 200
>>> l
['aaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaa', Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyboardInterrupt:
>>>
```
2020-08-25 11:47:50 -05:00
Damien George ee50a6effe py/mphal.h: Introduce mp_hal_time_ns and implement on various ports.
This should return a 64-bit value being the number of nanoseconds since
1970/1/1.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 16:13:44 +10:00
Damien George 5f9b105244 py/runtime: Fix builtin compile() in "single" mode so it prints exprs.
As per CPython behaviour, compile(stmt, "file", "single") should create
code which prints to stdout (via __repl_print__) the results of any
expressions in stmt.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 11:38:46 +10:00
Scott Shawcroft 701e80a025
Make socket reads interruptable 2020-08-21 11:00:02 -07:00
Dan Halbert 0e30dd8bcc merge from upstream; working; includes debug_out code for debugging via Saleae for posterity 2020-08-20 20:29:57 -04:00
Scott Shawcroft eb8b42aff1
Add basic error handling 2020-08-19 14:23:28 -07:00
Scott Shawcroft 1034cc1217
Add espidf module. 2020-08-19 14:23:28 -07:00
Scott Shawcroft 430530c74b
SSL works until it runs out of memory 2020-08-19 14:23:28 -07:00
Scott Shawcroft c9ece21c28
SocketPool stubbed out 2020-08-19 14:22:13 -07:00
Scott Shawcroft 3860991111
Ping work and start to add socketpool 2020-08-19 14:22:13 -07:00
Scott Shawcroft c53a72d3f5
Fix ipaddress import and parse ipv4 strings 2020-08-19 14:22:13 -07:00
Scott Shawcroft c62ab6e09a
Add ipaddress 2020-08-19 14:22:12 -07:00
Scott Shawcroft 1a6f4e0fe0
Scanning WIP. Need to sort out supervisor memory 2020-08-19 14:22:12 -07:00
Scott Shawcroft c5b8401a15
First crack at native wifi API 2020-08-19 14:21:59 -07:00
Scott Shawcroft 6857f98426
Split pulseio.PWMOut into pwmio
This gives us better granularity when implementing new ports because
PWMOut is commonly implemented before PulseIn and PulseOut.

Fixes #3211
2020-08-18 13:08:33 -07:00
Scott Shawcroft 24ca5c0218
Merge pull request #3295 from tannewt/turn_off_terminalio
Turn off terminalio for ja and ko
2020-08-18 12:10:31 -07:00
Taku Fukada 79a3796b1c Calculate the Huffman codebook without MP_QSTRs 2020-08-18 23:21:14 +09:00
Scott Shawcroft d01f5dc0bd
Turn off terminalio for ja and ko
The font is missing many characters and the build needs the space.
We can optimize font storage when we get a good font.

The serial output will work as usual.
2020-08-17 17:17:59 -07:00
Jeff Epler 08ed09acc6 makeqstrdata: don't print "compression incrased length" messages
This check as implemented is misleading, because it compares the
compressed size in bytes (including the length indication) with the source
string length in Unicode code points.  For English this is approximately
fair, but for Japanese this is quite unfair and produces an excess of
"increased length" messages.

This message might have existed for one of two reasons:
 * to alert to an improperly function huffman compression
 * to call attention to a need for a "string is stored uncompressed" case
We know by now that the huffman compression is functioning as designed and
effective in general.

Just to be on the safe side, I did some back-of-the-envelope estimates.
I considered these three replacements for "the true source string size, in bytes":
+    decompressed_len_utf8 = len(decompressed.encode('utf-8'))
+    decompressed_len_utf16 = len(decompressed.encode('utf-16be'))
+    decompressed_len_bitsize = ((1+len(decompressed)) * math.ceil(math.log(1+len(values), 2)) + 7) // 8

The third counts how many bits each character requires (fewer than 128
characters in the source character set = 7, fewer than 256 = 8, fewer than 512
= 9, etc, adding a string-terminating value) and is in some way representative
of the best way we would be able to store "uncompressed strings".  The Japanese
translation (largest as of writing) has just a few strings which increase by
this metric.  However, the amount of loss due to expansion in those cases is
outweighed by the cost of adding 1 bit per string to indicate whether it's
compressed or not.  For instance, in the BOARD=trinket_m0 TRANSLATION=ja build
the loss is 47 bytes over 300 strings.  Adding 1 bit to each of 300 strings will
cost about 37 bytes, leaving just 5 Thumb instructions to implement the code to
check and decode "uncompressed" strings in order to break even.
2020-08-16 20:50:48 -05:00
Jeff Epler cff448205f Don't define SHARPDISPLAY when !DISPLAYIO
.. even if FULL_BUILD
2020-08-12 07:39:28 -05:00
Jeff Epler c1400bae9b sharpmemory: Implement support for Sharp Memory Displays in framebufferio 2020-08-12 07:32:18 -05:00
Jeff Epler 93b373d617 "pop from empty %q"
Saves 12 bytes code on trinket m0
2020-08-04 18:42:09 -05:00
Jeff Epler 65e26f4a06 py: mp_obj_get_type_qstr as macro saves 24 bytes 2020-08-04 14:45:45 -05:00
Jeff Epler 024c8da578 Combine some "can't convert" messages 2020-08-04 14:45:45 -05:00
Jeff Epler c849b781c0 Combine 'index out of range' messages 2020-08-04 14:45:45 -05:00
Jeff Epler 89797fd3f9 various: Use mp_obj_get_type_qstr more widely
This removes runtime allocations of the cstring version of the qstring.

It is not a size improvement
2020-08-04 14:45:45 -05:00
Jeff Epler c37a25f0e5 Use qstrs to save an additional 4 bytes 2020-08-04 14:45:45 -05:00
Jeff Epler 92917b84f1 fix exception type for pop from empty set 2020-08-04 13:58:29 -05:00
Jeff Epler 67eb93fc98 py: introduce, use mp_raise_msg_vlist
This saves a very small amount of flash, 8 bytes on trinket_m0
2020-08-04 13:34:29 -05:00
Jeff Epler dddd25a776 Combine similar strings to reduce size of translations
This is a slight trade-off with code size, in places where a "_varg"
mp_raise variant is now used.  The net savings on trinket_m0 is
just 32 bytes.

It also means that the translation will include the original English
text, and cannot be translated.  These are usually names of Python
types such as int, set, or dict or special values such as "inf" or
"Nan".
2020-08-04 13:34:29 -05:00
Dan Halbert 0a60aee3e4 wip: compiles 2020-08-02 11:36:38 -04:00
Damien George 9883d8e818 py/persistentcode: Maintain root ptr list of imported native .mpy code.
On ports where normal heap memory can contain executable code (eg ARM-based
ports such as stm32), native code loaded from an .mpy file may be reclaimed
by the GC because there's no reference to the very start of the native
machine code block that is reachable from root pointers (only pointers to
internal parts of the machine code block are reachable, but that doesn't
help the GC find the memory).

This commit fixes this issue by maintaining an explicit list of root
pointers pointing to native code that is loaded from an .mpy file.  This
is not needed for all ports so is selectable by the new configuration
option MICROPY_PERSISTENT_CODE_TRACK_RELOC_CODE.  It's enabled by default
if a port does not specify any special functions to allocate or commit
executable memory.

A test is included to test that native code loaded from an .mpy file does
not get reclaimed by the GC.

Fixes #6045.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-02 22:34:09 +10:00
Jeff Epler d69f081c04 Merge remote-tracking branch 'origin/main' into blm_badge 2020-07-30 07:24:48 -05:00
Scott Shawcroft 61d1148bb3
Merge pull request #3222 from WarriorOfWire/pick_micropython
py/compile: Don't await __aiter__ special method in async-for.
2020-07-29 10:54:37 -07:00
Jeff Epler 9b8df7f635 Upgrade ulab
This version
 * moves source files to reflect module structure
 * adds inline documentation suitable for extract_pyi
 * incompatibly moves spectrogram to fft
 * incompatibly removes "extras"

There are some remaining markup errors in the specific revision of
extmod/ulab but they do not prevent the doc building process from
completing.
2020-07-28 16:57:48 -05:00
Dan Halbert aa97ea2501 Merge remote-tracking branch 'adafruit/main' into blm_badge 2020-07-28 14:15:02 -04:00
Dan Halbert e5e132a364 add blm_badge; add CIRCUITPY_AUDIOBUSIOIO_I2SOUT 2020-07-28 11:49:54 -04:00
Jerry Needell 2bdd62220e adjust stack for SAMD21 to accomodate larger pystack -- update frozen module adafruit_busdevice 2020-07-27 21:50:36 -04:00
Jonathan Hogg 901f3dce6e py/compile: Don't await __aiter__ special method in async-for.
MicroPython's original implementation of __aiter__ was correct for an
earlier (provisional) version of PEP492 (CPython 3.5), where __aiter__ was
an async-def function.  But that changed in the final version of PEP492 (in
CPython 3.5.2) where the function was changed to a normal one.  See
https://www.python.org/dev/peps/pep-0492/#why-aiter-does-not-return-an-awaitable
See also the note at the end of this subsection in the docs:
https://docs.python.org/3.5/reference/datamodel.html#asynchronous-iterators
And for completeness the BPO: https://bugs.python.org/issue27243

To be consistent with the Python spec as it stands today (and now that
PEP492 is final) this commit changes MicroPython's behaviour to match
CPython:  __aiter__ should return an async-iterable object, but is not
itself awaitable.

The relevant tests are updated to match.

See #6267.
2020-07-24 22:55:37 -07:00
Kenny 9a1f1236cc require async for and async with to actually be in an async def method instead of just a generator 2020-07-24 19:47:34 -07:00
Jonathan Hogg 37e1b5c891 py/compile: Don't await __aiter__ special method in async-for.
MicroPython's original implementation of __aiter__ was correct for an
earlier (provisional) version of PEP492 (CPython 3.5), where __aiter__ was
an async-def function.  But that changed in the final version of PEP492 (in
CPython 3.5.2) where the function was changed to a normal one.  See
https://www.python.org/dev/peps/pep-0492/#why-aiter-does-not-return-an-awaitable
See also the note at the end of this subsection in the docs:
https://docs.python.org/3.5/reference/datamodel.html#asynchronous-iterators
And for completeness the BPO: https://bugs.python.org/issue27243

To be consistent with the Python spec as it stands today (and now that
PEP492 is final) this commit changes MicroPython's behaviour to match
CPython:  __aiter__ should return an async-iterable object, but is not
itself awaitable.

The relevant tests are updated to match.

See #6267.
2020-07-25 00:58:18 +10:00
Kenny e9b4e0bd35 remove new char*s because m0 is way oversubscribed 2020-07-23 20:41:10 -07:00
Kenny 51a79b1af7 add coroutine behavior for generators
coroutines don't have __next__; they also call themselves coroutines.
This does not change the fact that `async def` methods are generators,
but it does make them behave more like CPython.
2020-07-23 20:40:16 -07:00
Dan Halbert f6f45c82a1 wip: ATT protocol 2020-07-23 18:54:26 -04:00
Damien George f43834aba2
py/py.mk: Use additional CFLAGS to compile string0.c.
Otherwise functions like memset might get optimised to call themselves (eg
with gcc 10).  And provide CFLAGS_BUILTIN so these options can be changed
by a port if needed.

Fixes issue #6053.
2020-07-22 16:26:47 -07:00
Scott Shawcroft 2bd6d05663
Add externs. GCC10 complains about duplicate defines 2020-07-22 16:26:46 -07:00
Scott Shawcroft 1ec3580946
Merge branch 'main' into memmonitor 2020-07-22 12:24:19 -07:00
Scott Shawcroft 6db10f9c7d
Turn off find when CPYTHON_COMPAT is off 2020-07-21 15:40:51 -07:00
Scott Shawcroft 1b3bc98068
Merge remote-tracking branch 'adafruit/main' into bytearray_find 2020-07-21 15:12:04 -07:00
Scott Shawcroft cb7df2e1ed
Merge pull request #3160 from tannewt/enable_pystack
Enable PYSTACK to keep function state out of the heap
2020-07-21 13:38:52 -07:00
Scott Shawcroft 900edb2b8e
Only add .find without CPYTHON_COMPAT 2020-07-21 13:08:42 -07:00
Damien George a853fff838 py/obj.h: Fix mp_seq_replace_slice_no_grow to use memmove not memcpy.
Because the argument arrays may overlap, as show by the new tests in this
commit.

Also remove the debugging comments for these macros, add a new comment
about overlapping regions, and separate the macros by blank lines to make
them easier to read.

Fixes issue #6244.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-21 01:05:59 +10:00
Scott Shawcroft f38a9c8d29
Add cast for mpy-cross warning 2020-07-17 17:55:06 -07:00
Scott Shawcroft a1e4814a27
Get AllocationAlarm working 2020-07-17 17:15:03 -07:00
Scott Shawcroft 518d909b2c
Add memorymonitor module 2020-07-17 17:15:03 -07:00
Scott Shawcroft 4b157aa6b8
Add find varients to bytearray 2020-07-17 17:07:37 -07:00
Scott Shawcroft 1160635608
Enable PYSTACK to keep function state out of the heap 2020-07-17 17:03:42 -07:00
DavePutz e5f7adcf5d
Fix to pass mpy-cross build 2020-07-13 22:54:52 -05:00
DavePutz 14799f9ee6
more formatting fix 2020-07-13 22:43:46 -05:00
DavePutz 30361f6f2a
Fix formatting 2020-07-13 22:39:30 -05:00
DavePutz e963cff72d
Issue #2949 Run background checks during long multiplications 2020-07-13 22:27:06 -05:00
Scott Shawcroft 51c888d4be
Merge pull request #3003 from Flameeyes/master
License tagging according to REUSE specifications.
2020-07-13 16:28:49 -07:00
Jeff Epler 6d97f6fccc audioio: Remove compatibility code
These items were aliased from audiocore to audioio for compatibility
with 4.x, but according to our deprecation schedule can be removed
in 6.0.
2020-07-08 20:31:35 -05:00
Dan Halbert a5ab2829eb use zephyr include files; wip: compiles 2020-07-08 09:33:41 -04:00
Diego Elio Pettenò 34b4993d63 Add license to some obvious files. 2020-07-06 19:16:25 +01:00
Dan Halbert 11cb3e3b4b hci skeleton done; not working yet 2020-06-30 23:19:40 -04:00
Damien George 332d83343f py: Rework mp_convert_member_lookup to properly handle built-ins.
This commit fixes lookups of class members to make it so that built-in
functions that are used as methods/functions of a class work correctly.

The mp_convert_member_lookup() function is pretty much completely changed
by this commit, but for the most part it's just reorganised and the
indenting changed.  The functional changes are:

- staticmethod and classmethod checks moved to later in the if-logic,
  because they are less common and so should be checked after the more
  common cases.

- The explicit mp_obj_is_type(member, &mp_type_type) check is removed
  because it's now subsumed by other, more general tests in this function.

- MP_TYPE_FLAG_BINDS_SELF and MP_TYPE_FLAG_BUILTIN_FUN type flags added to
  make the checks in this function much simpler (now they just test this
  bit in type->flags).

- An extra check is made for mp_obj_is_instance_type(type) to fix lookup of
  built-in functions.

Fixes #1326 and #6198.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-30 23:55:32 +10:00
Damien George d06ae1d2b1 py/obj.h: Make existing MP_TYPE_FLAG_xxx macros sequential.
There's no reason to have them non-sequential, this was likely a typo from
commit 9ec1caf42e.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-30 23:55:32 +10:00
Dan Halbert 1bc2e979eb wip; compiles; much commented out 2020-06-26 17:23:20 -04:00
Jeff Epler 05837b2841 sdioio: Add shared-bindings
There is no implementation yet.
2020-06-26 11:50:25 -05:00
Jeff Epler 57fde2e07b sdcardio: implement new library for SD card I/O
Testing performed: That a card is successfully mounted on Pygamer with
the built in SD card slot

This module is enabled for most FULL_BUILD boards, but is disabled for
samd21 ("M0"), litex, and pca10100 for various reasons.
2020-06-26 11:50:23 -05:00
Damien George 9f911d822e py/objcomplex: Add mp_obj_get_complex_maybe for use in complex bin-op.
This allows complex binary operations to fail gracefully with unsupported
operation rather than raising an exception, so that special methods work
correctly.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 01:03:10 +10:00
Damien George 41fa8b5482 py/emitnative: Implement binary operations for viper uint operands.
uint types in viper mode can now be used for all binary operators except
floor-divide and modulo.

Fixes issue #1847 and issue #6177.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 00:24:04 +10:00
Damien George b3b8706d27 py/asm: Add condition codes for signed comparisons.
Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 00:24:04 +10:00
Damien George aa26fe62d8 py/asm: Add funcs/macros to emit machine code for logical-shift-right.
Signed-off-by: Damien George <damien@micropython.org>
2020-06-27 00:24:04 +10:00
Dan Halbert 759929c24a hci early wip; refactor supervisor bluetooth.c for nrf: tested 2020-06-25 20:57:17 -04:00
Jeff Epler 1d2cc0b968 I2CPeripheral: Rename class and its module
This is an incompatible change.
2020-06-25 11:44:19 -05:00
Kamil Tomaszewski 84f424f631 gnss: Implement new library for GNSS 2020-06-24 11:14:44 +02:00
Damien George 457fdf61c3 py/objtype: Support passing in an OrderedDict to type() as the locals.
An OrderedDict can now be used for the locals when creating a type
explicitly via type(name, bases, locals).

Signed-off-by: Damien George <damien@micropython.org>
2020-06-24 12:05:40 +10:00
Damien George 456a3abe8d py/obj.h: Add public mp_obj_is_dict_or_ordereddict() helper macro.
And use it in py/objdict.c instead of mp_obj_is_dict_type.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-24 12:05:00 +10:00
Jeff Epler a7b36f724f circuitpy_defns.mk: Sort several lists of source files
and regularize whitespace in one spot
2020-06-22 10:45:27 -05:00
Jeff Epler a580f0f1c4 _pew: move to common-hal
I noticed that this code was referring to samd-specific functionality,
and isn't enabled except in one samd board (pewpew10).  Move it.

There is incomplte support for _pew in mimxrt10xx which then caused build
errors; adding a #if guard to check for _pew being enabled fixes it.
The _pew module is not likely to be important on mimxrt but I'll leave the
choice to remove it to someone else.
2020-06-22 10:45:27 -05:00
Damien George 6164c7e666 py/misc.h: Add missing semi-colon in mp_float_union_t for big-endian.
Fixes issue #6161.

Signed-off-by: Damien George <damien@micropython.org>
2020-06-22 13:42:24 +10:00
Damien George f2e267da68 py/compile: Implement PEP 526, syntax for variable annotations.
This addition to the grammar was introduced in Python 3.6.  It allows
annotating the type of a varilable, like:

    x: int = 123
    s: str

The implementation in this commit is quite simple and just ignores the
annotation (the int and str bits above).  The reason to implement this is
to allow Python 3.6+ code that uses this feature to compile under
MicroPython without change, and for users to use type checkers.

In the future viper could use this syntax as a way to give types to
variables, which is currently done in a bit of an ad-hoc way, eg
x = int(123).  And this syntax could potentially be used in the inline
assembler to define labels in an way that's easier to read.
2020-06-16 23:18:01 +10:00
Damien George 131b0de70a py/grammar.h: Consolidate duplicate sub-rules for :test and =test. 2020-06-16 23:18:01 +10:00
Damien George 1783950311 py/compile: Implement PEP 572, assignment expressions with := operator.
The syntax matches CPython and the semantics are equivalent except that,
unlike CPython, MicroPython allows using := to assign to comprehension
iteration variables, because disallowing this would take a lot of code to
check for it.

The new compile-time option MICROPY_PY_ASSIGN_EXPR selects this feature and
is enabled by default, following MICROPY_PY_ASYNC_AWAIT.
2020-06-16 22:02:24 +10:00
Damien George 0fd91e39b1 py/compile: Convert scope test to SCOPE_IS_COMP_LIKE macro.
This macro can be used elsewhere.
2020-06-16 21:42:37 +10:00
David Lechner ecd7826316 tools/codeformat.py: Remove sizeof fixup.
Formatting for `* sizeof` was fixed in uncrustify v0.71, so we no longer
need the fixups for it.  Also, there was one file where the updated
uncrustify caught a problem that the regex didn't pick up, which is updated
in this commit.

Signed-off-by: David Lechner <david@pybricks.com>
2020-06-14 20:24:18 +10:00
Damien George f3062b5cbd py/obj.h: Clarify comments about mp_map_t is_fixed and is_ordered.
Long ago, prior to 0ef01d0a75, fixed and
ordered maps were the same setting with the "table_is_fixed_array" member
of mp_map_t.  But these settings are actually independent, and it is
possible to have is_fixed=1, is_ordered=0 (although this can currently
only be done by tools/cc1).  So update the comments to reflect this.
2020-06-10 22:27:10 +10:00
Andrew Leech 95cbe6b65e py/objtype: Use mp_obj_dict_copy() for creating obj.__dict__ attribute.
The resulting dict is now marked as read-only (is_fixed=1) to enforce the
fact that changes to this dict will not be reflected in the class instance.

This commit reduces code size by about 20 bytes, and should be more
efficient because it creates a direct copy of the dict rather than
reinserting all elements.
2020-06-10 22:03:39 +10:00
Andrew Leech 28370c0450 py/objtype: Add __dict__ attribute for class objects.
The behavior mirrors the instance object dict attribute where a copy of the
local attributes are provided (unless the dict is read-only, then that dict
itself is returned, as an optimisation).  MicroPython does not support
modifying this dict because the changes will not be reflected in the class.

The feature is only enabled if MICROPY_CPYTHON_COMPAT is set, the same as
the instance version.
2020-06-10 21:58:13 +10:00
Damien George bd06c698f0 py/dynruntime.h: Make mp_obj_str_get_str raise if arg not a str/bytes. 2020-06-08 09:05:03 +02:00
Jim Mussared e6881f0829 extmod/modbluetooth: Make modbluetooth event not a bitfield.
There doesn't appear to be any use for only triggering on specific events,
so it's just easier to number them sequentially.  This makes them smaller
values so they take up only 1 byte in the ringbuf, only 1 byte for the
opcode in the bytecode, and makes room for more events.

Also add a couple of new event types that need to be implemented (to avoid
re-numbering later).

And rename _COMPLETE and _STATUS to _DONE for consistency.

In the future the "trigger" keyword argument can be reinstated by requiring
the user to compute the bitmask, eg:

    ble.irq(handler, 1 << _IRQ_SCAN_RESULT | 1 << _IRQ_SCAN_DONE)
2020-06-05 14:04:20 +10:00
Diego Elio Pettenò dd5d7c86d2 Fix up end of file and trailing whitespace.
This can be enforced by pre-commit, but correct it separately to make it easier to review.
2020-06-03 10:56:35 +01:00
Damien George 203b10703e py/modbuiltins: Fix getattr to work with class raising AttributeError.
Fixes issue #6089.
2020-06-02 15:42:20 +10:00
Scott Shawcroft 959f7297da
Merge remote-tracking branch 'adafruit/master' into wdt-nrf 2020-06-01 17:01:16 -07:00
Jeff Epler f211a090e2 py.mk: Assume we want all C source files in ulab 2020-06-01 08:26:23 -05:00
Jeff Epler 1cc281b6a4 py.mk: Assume we want all C files from ulab 2020-06-01 08:20:23 -05:00
Scott Shawcroft 2fd7a43f2f
Merge remote-tracking branch 'adafruit/master' into wdt-nrf 2020-05-29 10:16:24 -07:00
Dan Halbert 6ace4ee7e8
Merge pull request #2968 from jepler/more-efficient-translation
More efficient translation
2020-05-28 21:37:45 -04:00
Scott Shawcroft 796373b8be
A number of small ESP32S2 fixes:
* Fix flash writes that don't end on a sector boundary. Fixes #2944
* Fix enum incompatibility with IDF.
* Fix printf output so it goes out debug UART.
* Increase stack size to 8k.
* Fix sleep of less than a tick so it doesn't crash.
2020-05-28 15:43:55 -07:00
Jeff Epler d0f9b5901e translations: document the compressed format 2020-05-28 11:30:46 -05:00
Jeff Epler fe3e8d1589 string compression: save a few bits per string
Length was stored as a 16-bit number always.  Most translations have
a max length far less.  For example, US English translation lengths
always fit in just 8 bits.  probably all languages fit in 9 bits.

This also has the side effect of reducing the alignment of
compressed_string_t from 2 bytes to 1.

testing performed: ran in german and english on pyruler, printed messages
looked right.

Firmware size, en_US
Before: 3044 bytes free in flash
After: 3408 bytes free in flash

Firmware size, de_DE (with #2967 merged to restore translations)
Before: 1236 bytes free in flash
After: 1600 bytes free in flash
2020-05-28 08:36:08 -05:00
David Lechner 093fd80760 py/modsys: Use consistent naming pattern for module-level const objects.
This renames a few identifiers to follow the usual naming convention of
mp_<module>_<name>.  This makes them easier to find, e.g. when grep'ing.
2020-05-28 10:02:14 +10:00
stijn 97ccde0c43 py/ringbuf: Fix compilation with msvc.
Older versions do not have "inline" so fetch the definition from
mpconfigport.h.
2020-05-28 09:56:18 +10:00
stijn 81db22f693 py/modmath: Work around msvc float bugs in atan2, fmod and modf.
Older implementations deal with infinity/negative zero incorrectly.  This
commit adds generic fixes that can be enabled by any port that needs them,
along with new tests cases.
2020-05-28 09:54:54 +10:00
Damien George a902b69dd5 py/py.mk: Use additional CFLAGS to compile string0.c.
Otherwise functions like memset might get optimised to call themselves (eg
with gcc 10).  And provide CFLAGS_BUILTIN so these options can be changed
by a port if needed.

Fixes issue #6053.
2020-05-27 23:10:23 +10:00
Sean Cross bd086a102e Revert "add WatchDogTimeout exception"
This reverts commit 561e7e619095869f58fc728d428f3ff20e8bfc40.
2020-05-27 11:28:49 +08:00
Sean Cross 08362c9cab watchdogtimer: refactor to new api
This refactors the WatchDogTimer API to use the format proposed in
https://github.com/adafruit/circuitpython/pull/2933#issuecomment-632268227

Signed-off-by: Sean Cross <sean@xobs.io>
2020-05-27 11:28:49 +08:00
Sean Cross ae950bc050 add WatchDogTimeout exception
This adds an exception to be raised when the WatchDogTimer times out.

Note that this currently causes a HardFault, and it's not clear why it's
not behaving properly.

Signed-off-by: Sean Cross <sean@xobs.io>
2020-05-27 11:28:49 +08:00
Sean Cross 595f6387c2 watchdog: rename module from `wdt` and move to `microcontroller`
This also places it under the `microcontroller` object.

Signed-off-by: Sean Cross <sean@xobs.io>
2020-05-27 11:28:49 +08:00
Sean Cross f4719609f7 wdt: add watchdog support
This adds shared bindings for a watchdog timer, based on the API
provided by micropython.

Signed-off-by: Sean Cross <sean@xobs.io>
2020-05-27 11:28:48 +08:00
Scott Shawcroft acf4b1bede
Remove reverse methods from per-board defs 2020-05-19 11:43:57 -07:00
Scott Shawcroft cf690bd390
Merge remote-tracking branch 'adafruit/master' into esp32s2 2020-05-18 16:46:41 -07:00
Scott Shawcroft 9811c1fe4b
Merge pull request #2896 from theacodes/add-bytearray-decode
Add bytearray.decode() for CPython compatibility
2020-05-18 14:38:45 -07:00
Scott Shawcroft 6aaab005c5
Initial ESP32S2 port.
Basic blinky works but doesn't check pins.
2020-05-15 15:36:16 -07:00
Thea Flowers 0876034adb Make decode only apply to bytearray and not array as well 2020-05-15 14:32:44 -07:00
Thea Flowers fe3e8ee02c Add bytearray.decode() for CPython compatibility
CPython has a `decode()` method on `bytearray`. This adds that method using the code from `bytes.decode`.

Test program:

```python

byte_boi = bytes([0x6D, 0x65, 0x65, 0x70])
print(byte_boi)  # b'meep'
byte_boi_str = byte_boi.decode("utf-8")
print(byte_boi_str)  # meep

byte_array_boi = bytearray(byte_boi)
print(byte_array_boi)  # bytearray(b'meep')
byte_array_boi_str = byte_array_boi.decode("utf-8")
print(byte_array_boi_str)  # meep

print(byte_array_boi_str == byte_boi_str)  # True

```
2020-05-14 21:52:58 -07:00
Damien George 801f7dca78 py/nativeglue.h: Rename "setjmp" entry to "setjmp_" to avoid any clash.
Because some compilers may define setjmp to something.

Fixes issue #6032.
2020-05-14 21:48:05 +10:00
Scott Shawcroft 7546d47f77
Merge remote-tracking branch 'adafruit/master' into improve_verification 2020-05-13 08:23:55 -07:00
Scott Shawcroft 2efe17bcb7
Don't build in empty help.c 2020-05-12 19:01:17 -07:00
Scott Shawcroft 4612270f98
Only enable COUNTIO in full builds 2020-05-12 19:00:51 -07:00
Scott Shawcroft 277e8d528b
Merge branch 'master' into Optical-Encoder-Module 2020-05-12 18:22:57 -07:00