When SCR_QSTR contains C++ files they should be preprocessed with the same
compiler flags (CXXFLAGS) as they will be compiled with, to make sure code
scanned for QSTR occurrences is effectively the code used in the rest of
the build. The 'split SCR_QSTR in .c and .cpp files and process each with
different flags' logic isn't trivial to express in a Makefile and the
existing principle for deciding which files to preprocess was already
rather complicated, so the actual preprocessing is moved into
makeqstrdefs.py completely.
When process_file() is passed a preprocessed C++ file for instance it won't
find any lines containing .c files and the last_fname variable remains
None, so handle that gracefully.
Newer GCC versions are able to warn about switch cases that fall
through. This is usually a sign of a forgotten break statement, but in
the few cases where a fall through is intended we annotate it with this
macro to avoid the warning.
Like Clang, GCC warns about this file, but only with -Woverride-init
which is enabled by -Wextra. Disable the warnings for this file just
like we do for Clang to make -Wextra happy.
When compiling with -Wextra which includes -Wmissing-field-initializers
GCC will warn that the defval field of mp_arg_val_t is not initialized.
This is just a warning as it is defined to be zero initialized, but since
it is a union it makes sense to be explicit about which member we're
going to use, so add the explicit initializers and get rid of the
warning.
On x86 chars are signed, but we're comparing a char to '0' + unsigned int,
which is promoted to an unsigned int. Let's promote the char to unsigned
before doing the comparison to avoid weird corner cases.
The function scope_find_or_add_id used to take a scope_kind_t enum and
save it in an uint8_t. Saving an enum in a uint8_t is fine, but
everywhere this function is called it is not actually given a
scope_kind_t but an anonymous enum instead. Let's give this enum a name
and use that as the argument type.
This doesn't change the generated code, but is a C type mismatch that
unfortunately doesn't show up unless you enable -Wenum-conversion.
This gets a further speedup of about 2s (12s -> 9.5s elapsed build time)
for stm32f405_feather
For what are probably historical reasons, the qstr process involves
preprocessing a large number of source files into a single "qstr.i.last"
file, then reading this and splitting it into one "qstr" file for each
original source ("*.c") file.
By eliminating the step of writing qstr.i.last as well as making the
regular-expression-matching part be parallelized, build speed is further
improved.
Because the step to build QSTR_DEFS_COLLECTED does not access
qstr.i.last, the path is replaced with "-" in the Makefile.
Rather than simply invoking gcc in preprocessor mode with a list of files, use
a Python script with the (python3) ThreadPoolExecutor to invoke the
preprocessor in parallel.
The amount of concurrency is the number of system CPUs, not the makefile "-j"
parallelism setting, because there is no simple and correct way for a Python
program to correctly work together with make's idea of parallelism.
This reduces the build time of stm32f405 feather (a non-LTO build) from 16s to
12s on my 16-thread Ryzen machine.
Some examples of improved compliance with CPython that currently
have divergent behavior in CircuitPython are listed below:
* yield from is not allowed in async methods
```
>>> async def f():
... yield from 'abc'
...
Traceback (most recent call last):
File "<stdin>", line 2, in f
SyntaxError: 'yield from' inside async function
```
* await only works on awaitable expressions
```
>>> async def f():
... await 'not awaitable'
...
>>> f().send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
AttributeError: 'str' object has no attribute '__await__'
```
* only __await__()able expressions are awaitable
Okay this one actually does not work in circuitpython at all today.
This is how CPython works though and pretending __await__ does not
exist will only bite users who write both.
```
>>> class c:
... pass
...
>>> def f(self):
... yield
... yield
... return 'f to pay respects'
...
>>> c.__await__ = f # could just as easily have put it on the class but this shows how it's wired
>>> async def g():
... awaitable_thing = c()
... partial = await awaitable_thing
... return 'press ' + partial
...
>>> q = g()
>>> q.send(None)
>>> q.send(None)
>>> q.send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: press f to pay respects
```
This adds the `async def` and `await` verbs to valid CircuitPython syntax using the Micropython implementation.
Consider:
```
>>> class Awaitable:
... def __iter__(self):
... for i in range(3):
... print('awaiting', i)
... yield
... return 42
...
>>> async def wait_for_it():
... a = Awaitable()
... result = await a
... return result
...
>>> task = wait_for_it()
>>> next(task)
awaiting 0
>>> next(task)
awaiting 1
>>> next(task)
awaiting 2
>>> next(task)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 42
>>>
```
and more excitingly:
```
>>> async def it_awaits_a_subtask():
... value = await wait_for_it()
... print('twice as good', value * 2)
...
>>> task = it_awaits_a_subtask()
>>> next(task)
awaiting 0
>>> next(task)
awaiting 1
>>> next(task)
awaiting 2
>>> next(task)
twice as good 84
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration:
```
Note that this is just syntax plumbing, not an all-encompassing implementation of an asynchronous task scheduler or asynchronous hardware apis.
uasyncio might be a good module to bring in, or something else - but the standard Python syntax does not _strictly require_ deeper hardware
support.
Micropython implements the await verb via the __iter__ function rather than __await__. It's okay.
The syntax being present will enable users to write clean and expressive multi-step state machines that are written serially and interleaved
according to the rules provided by those users.
Given that this does not include an all-encompassing C scheduler, this is expected to be an advanced functionality until the community settles
on the future of deep hardware support for async/await in CircuitPython. Users will implement yield-based schedulers and tasks wrapping
synchronous hardware APIs with polling to avoid blocking, while their application business logic gets simple `await` statements.
Some downstream projects may use tags in their repositories for more than
just designating MicroPython releases. In those cases, the
makeversionhdr.py script would end up using a different tag than intended.
So tell `git describe` to only match tags that look like a MicroPython
version tag, such as `v1.12` or `v2.0`.
This already begins obscuring things, because now there are two sets of
shared-module functions for manipulating the same structure, e.g.,
common_hal_canio_remote_transmission_request_get_id and
common_hal_canio_message_get_id
Calling the bytes constructor on a bytes object returns the original bytes
object. This saves allocating a new instance, and matches CPython.
Signed-off-by: Iyassou Shimels <s.iyassou@gmail.com>
New contributor @mdroberts1243 encountered an interesting problem in
which the argument they had named "column_underscore_and_page_addressing"
simply couldn't be used; I discovered that internally this had been
transformed into "column_underscore∧page_addressing", because QSTR
makes _ENTITY_ stand for the same thing as &ENTITY; does in HTML.
This might be nice for some things, but we don't want it here!
I was unable to find a sensible way to "escape" and prevent this entity
coding, so instead I ripped out support for the _and_ and _or_ escapes.
Tested & working:
* Send standard packets
* Receive standard packets (1 FIFO, no filter)
Interoperation between SAM E54 Xplained running this tree and
MicroPython running on STM32F405 Feather with an external
transceiver was also tested.
Many other aspects of a full implementation are not yet present,
such as error detection and recovery.
Discord user Folknology encountered a problem building with Python 3.6.9,
`TypeError: ord() expected a character, but string of length 0 found`.
I was able to reproduce the problem using Python3.5*, and discovered that
the meaning of the regular expression `"|."` had changed in 3.7. Before,
```
>>> [m.group(0) for m in re.finditer("|.", "hello")]
['', '', '', '', '', '']
```
After:
```
>>> [m.group(0) for m in re.finditer("|.", "hello")]
['', 'h', '', 'e', '', 'l', '', 'l', '', 'o', '']
```
Check if `words` is empty and if so use `"."` as the regular expression
instead. This gives the same result on both versions:
```
['h', 'e', 'l', 'l', 'o']
```
and fixes the generation of the huffman dictionary.
Folknology verified that this fix worked for them.
* I could easily install 3.5 but not 3.6. 3.5 reproduced the same problem