Since CPython 3.8 the optional "sep" argument to hexlify is officially
supported, so update comments in the code and the docs to reflect this.
Signed-off-by: Damien George <damien@micropython.org>
Rather than dealing with the different int types, just pass them all as a
single array of mp_int_t with n_unsigned (before addr) and n_signed (after
addr).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This adds `_IRQ_GET_SECRET` and `_IRQ_SET_SECRET` events to allow the BT
stack to request the Python code retrive/store/delete secret key data. The
actual keys and values are opaque to Python and stack-specific.
Only NimBLE is implemented (pending moving btstack to sync events). The
secret store is designed to be compatible with BlueKitchen's TLV store API.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows the application to be notified if any of encrypted,
authenticated and bonded state change, as well as the encryption key size.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Enable it on STM32/Unix NimBLE only (pairing/bonding requires synchronous
events and full bindings).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Instead of returning None/bool from the IRQ, return None/int (where a zero
value means success). This mirrors how the L2CAP_ACCEPT return value
works.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This widens the characteristic/descriptor flags to 16-bit, to allow setting
encryption/authentication requirements.
Sets the required flags for NimBLE and btstack implementations.
The BLE.FLAG_* constants will eventually be deprecated in favour of copy
and paste Python constants (like the IRQs).
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This allows the application to be notified of changes to the connection
interval, connection latency and supervision timeout.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit switches the roles of the helper task from a cancellation task
to a runner task, to get the correct semantics for cancellation of
wait_for.
Some uasyncio tests are now disabled for the native emitter due to issues
with native code generation of generators and yield-from.
Fixes#5797.
Signed-off-by: Damien George <damien@micropython.org>
This is added because task.coro==None is no longer the way to detect if a
task is finished. Providing a (CPython compatible) function for this
allows the implementation to be abstracted away.
Signed-off-by: Damien George <damien@micropython.org>
When a tasks raises an exception which is uncaught, and no other task
await's on that task, then an error message is printed (or a user function
called) via a call to Loop.call_exception_handler. In CPython this call is
made when the Task object is freed (eg via reference counting) because it's
at that point that it is known that the exception that was raised will
never be handled.
MicroPython does not have reference counting and the current behaviour is
to deal with uncaught exceptions as early as possible, ie as soon as they
terminate the task. But this can be undesirable because in certain cases
a task can start and raise an exception immediately (before any await is
executed in that task's coro) and before any other task gets a chance to
await on it to catch the exception.
This commit changes the behaviour so that tasks which end due to an
uncaught exception are scheduled one more time for execution, and if they
are not await'ed on by the next scheduling loop, then the exception handler
is called (eg the exception is printed out).
Signed-off-by: Damien George <damien@micropython.org>
Disable certain classes of diagnostic when building ulab. We should
submit patches upstream to (A) fix these errors and (B) upgrade their
CI so that the problems are caught before we want to integrate with
CircuitPython, but not right now.
Also known as L2CAP "connection oriented channels". This provides a
socket-like data transfer mechanism for BLE.
Currently only implemented for NimBLE on STM32 / Unix.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Hardware I2C implementations must provide a .init() protocol method if they
want to support reconfiguration. Otherwise the default is that i2c.init()
raises an OSError (currently the case for all ports).
mp_machine_soft_i2c_locals_dict is renamed to mp_machine_i2c_locals_dict to
match the generic SPI bindings.
Fixes issue #6623 (where calling .init() on a HW I2C would crash).
Signed-off-by: Damien George <damien@micropython.org>
This changes stm32 from using PENDSV to run NimBLE to use the MicroPython
scheduler instead. This allows Python BLE callbacks to be invoked directly
(and therefore synchronously) rather than via the ringbuffer.
The NimBLE UART HCI and event processing now happens in a scheduled task
every 128ms. When RX IRQ idle events arrive, it will also schedule this
task to improve latency.
There is a similar change for the unix port where the background thread now
queues the scheduled task.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This requires that the event handlers are called from non-interrupt context
(i.e. the MicroPython scheduler).
This will allow the BLE stack (e.g. NimBLE) to run from the scheduler
rather than an IRQ like PENDSV, and therefore be able to invoke Python
callbacks directly/synchronously. This allows writing Python BLE handlers
for events that require immediate response such as _IRQ_READ_REQUEST (which
was previous a hard IRQ) and future events relating to pairing/bonding.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Using a semaphore (the previous approach) will only run the UART, whereas
for startup we need to also run the event queue.
This change makes it run the full scheduler hook.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Instead of having the stack indicate a "start", "data"..., "end", pass
through the data in one callback as an array of chunks of data.
This is because the upcoming non-ringbuffer modbluetooth implementation
cannot buffer the data in the ringbuffer and requires instead a single
callback with all the data, to pass to the Python callback.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Prior to this change machine.mem32['foo'] (or using any other non-integer
subscript) could result in a fault due to 'foo' being interpreted as an
integer. And when writing code it's hard to tell if the fault is due to a
bad subscript type, or an integer subscript that specifies an invalid
memory address.
The type of the object used in the subscript is now tested to be an
integer by using mp_obj_get_int_truncated instead of
mp_obj_int_get_truncated. The performance hit of this change is minimal,
and machine.memX objects are more for convenience than performance (there
are many other ways to read/write memory in a faster way),
Fixes issue #6588.
If a port provides MICROPY_PY_URANDOM_SEED_INIT_FUNC as a source of
randomness then this will be used when urandom.seed() is called without
an argument (or with None as the argument) to seed the pRNG.
Other related changes in this commit:
- mod_urandom___init__ is changed to call seed() without arguments, instead
of explicitly passing in the result of MICROPY_PY_URANDOM_SEED_INIT_FUNC.
- mod_urandom___init__ will only ever seed the pRNG once (before it could
seed it again if imported by, eg, random and then urandom).
- The Yasmarang state is moved to the BSS for builds where the state is
guaranteed to be initialised on import of the (u)random module.
Signed-off-by: Damien George <damien@micropython.org>
.. pull in various doc build fixes that prevented the previous commit from building. This is still "0.54.5", the tag was updated in micropython-ulab (since no functional difference was introduced, only doc and CI differences, I imagine)
Newer GCC versions are able to warn about switch cases that fall
through. This is usually a sign of a forgotten break statement, but in
the few cases where a fall through is intended we annotate it with this
macro to avoid the warning.
When compiling with -Wextra which includes -Wmissing-field-initializers
GCC will warn that the defval field of mp_arg_val_t is not initialized.
This is just a warning as it is defined to be zero initialized, but since
it is a union it makes sense to be explicit about which member we're
going to use, so add the explicit initializers and get rid of the
warning.
This unifies the flash config to the settings used by the Boot ROM.
This makes the config unique per board which allows for changing
quad enable and status bit differences per flash device. It also
allows for timing differences due to the board layout.
This change also tweaks linker layout to leave more ram space for
the CircuitPython heap.
It requires mp_hal_time_ns() to be provided by a port. This function
allows very accurate absolute timestamps.
Enabled on unix, windows, stm32, esp8266 and esp32.
Signed-off-by: Damien George <damien@micropython.org>
With a warning that this way of constructing software I2C/SPI is
deprecated. The check and warning will be removed in a future release.
This should help existing code to migrate to the new SoftI2C/SoftSPI types.
Signed-off-by: Damien George <damien@micropython.org>
The SoftSPI constructor is now used soley to create SoftSPI instances, it
can no longer delegate to create a hardware-based SPI instance.
Signed-off-by: Damien George <damien@micropython.org>
The SoftI2C constructor is now used soley to create SoftI2C instances, it
can no longer delegate to create a hardware-based I2C instance.
Signed-off-by: Damien George <damien@micropython.org>
Also rename machine_i2c_type to mp_machine_soft_i2c_type. These changes
make it clear that it's a soft-I2C implementation, and match SoftSPI.
Signed-off-by: Damien George <damien@micropython.org>
A read-only memoryview object is a better representation of the data, which
is owned by the ubluetooth module and may change between calls to the
user's irq callback function.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, uos.chdir('/') followed by uos.stat('noexist') would
succeed that stat even though the entry did not exist (some other functions
like listdir would have similar issues). This is because, if the current
directory was the root and the path was relative, mp_vfs_lookup_path would
return success for bad paths.
Signed-off-by: Damien George <damien@micropython.org>
For time-based functions that work with absolute time there is the need for
an Epoch, to set the zero-point at which the absolute time starts counting.
Such functions include time.time() and filesystem stat return values. And
different ports may use a different Epoch.
To make it clearer what functions use the Epoch (whatever it may be), and
make the ports more consistent with their use of the Epoch, this commit
renames all Epoch related functions to include the word "epoch" in their
name (and remove references to "2000").
Along with this rename, the following things have changed:
- mp_hal_time_ns() is now specified to return the number of nanoseconds
since the Epoch, rather than since 1970 (but since this is an internal
function it doesn't change anything for the user).
- littlefs timestamps on the esp8266 have been fixed (they were previously
off by 30 years in nanoseconds).
Otherwise, there is no functional change made by this commit.
Signed-off-by: Damien George <damien@micropython.org>
For the following 3 functions, previously the code relied on whether the
arg was passed at all, to make it optional. Now it allows them to be
explicitly `None` to indicate they are not used:
- gatts_notify(..., [data])
- gattc_discover_services(..., [uuid])
- gattc_discover_characteristics(..., [uuid])
Also ensure that the uuid arguments are actually instances of the uuid
type, and fix handling of the 5th arg in gattc_discover_characteristics().
This allows `ble.active(1)` to fail correctly if the HCI controller is
unavailable.
It also avoids an infine loop in the NimBLE event handler where NimBLE
doesn't correctly detect that the HCI controller is unavailable and keeps
trying to reset.
Furthermore, it fixes an issue where GATT service registrations were left
allocated, which led to a bad realloc if the stack was activated multiple
times.
Generally a controller should either have its own public address hardcoded,
or loaded by the driver (e.g. cywbt43).
However, for a controller that has no public address where you still want a
long-term stable address, this allows you to use a static address generated
by the port. Typically on STM32 this will be an LAA, but a board might
override this.
This commit adds support for using Bluetooth on the unix port via a H4
serial interface (distinct from a USB dongle), with both BTstack and NimBLE
Bluetooth stacks.
Note that MICROPY_PY_BLUETOOTH is now disabled for the coverage variant.
Prior to this commit Bluetooth was anyway not being built on Travis because
libusb was not detected. But now that bluetooth works in H4 mode it will
be built, and will lead to a large decrease in coverage because Bluetooth
tests cannot be run on Travis.
Previously the interaction between the different layers of the Bluetooth
stack was different on each port and each stack. This commit defines
common interfaces between them and implements them for cyw43, btstack,
nimble, stm32, unix.
This is consistent with the other 'micro' modules and allows implementing
additional features in Python via e.g. micropython-lib's sys.
Note this is a breaking change (not backwards compatible) for ports which
do not enable weak links, as "import sys" must now be replaced with
"import usys".
By setting MICROPY_EPOCH_IS_1970 a port can opt to use 1970/1/1 as the
Epoch for timestamps returned by stat(). And this setting is enabled on
the unix and windows ports because that's what they use.
Signed-off-by: Damien George <damien@micropython.org>
On ports like unix where the Epoch is 1970/1/1 and atime/mtime/ctime are in
seconds since the Epoch, this value will overflow a small-int on 32-bit
systems. So far this is only an issue on 32-bit unix builds that use the
VFS layer (eg dev and coverage unix variants) but the fix (using
mp_obj_new_int_from_uint instead of MP_OBJ_NEW_SMALL_INT) is there for all
ports so as to not complicate the code, and because they will need the
range one day.
Also apply a similar fix to other fields in VfsPosix.stat because they may
also be large.
Signed-off-by: Damien George <damien@micropython.org>
This commit fixes the cases when a TCP socket is in STATE_NEW,
STATE_LISTENING or STATE_CONNECTING and recv() is called on it. It now
raises ENOTCONN instead of a random error code due to it previously
indexing beyond the start of error_lookup_table[].
Signed-off-by: Damien George <damien@micropython.org>
Updating to Black v20.8b1 there are two changes that affect the code in
this repository:
- If there is a trailing comma in a list (eg [], () or function call) then
that list is now written out with one line per element. So remove such
trailing commas where the list should stay on one line.
- Spaces at the start of """ doc strings are removed.
Signed-off-by: Damien George <damien@micropython.org>
The memory operation functions read_mem() and write_mem() create a
temporary buffer on the local C stack for the address bytes with the size
of 4 bytes. This buffer is filled in a loop from the user supplied address
and address length. If the user supplied 'addrsize' is bigger than 32, the
local buffer is overrun.
Fix this by raising an exception for invalid 'addrsize' values.
Signed-off-by: Michael Buesch <m@bues.ch>
This adds an additional optional parameter to gap_scan() to select active
scanning, where scan responses are returned as well as normal scan results.
This parameter is False by default which retains the existing behaviour.
The READ_REQUEST callback is handled as a hard interrupt (because the BLE
stack needs an immediate response from it so it can continue) and so calls
to Python require extra protection:
- the caller-owned tuple passed into the callback must be separate from the
tuple used by other callback events (which are soft interrupts);
- the GC and scheduler must be locked during callback execution.
This commit adds support for modification time of files on littlefs v2
filesystems, using file attributes. For some background see issue #6114.
Features/properties of this implementation:
- Only supported on littlefs2 (not littlefs1).
- Uses littlefs2's general file attributes to store the timestamp.
- The timestamp is 64-bits and stores nanoseconds since 1970/1/1 (if the
range to the year 2554 is not enough then additional bits can be added to
this timestamp by adding another file attribute).
- mtime is enabled by default but can be disabled in the constructor, eg:
uos.mount(uos.VfsLfs2(bdev, mtime=False), '/flash')
- It's fully backwards compatible, existing littlefs2 filesystems will work
without reformatting and timestamps will be added transparently to
existing files (once they are opened for writing).
- Files without timestamps will open correctly, and stat will just return 0
for their timestamp.
- mtime can be disabled or enabled each mount time and timestamps will only
be updated if mtime is enabled (otherwise they will be untouched).
Signed-off-by: Damien George <damien@micropython.org>
Otherwise a task that continuously awaits on a large negative sleep can
monopolise the scheduler (because its wake time is always less than
everything else in the pairing heap).
Signed-off-by: Damien George <damien@micropython.org>
mp_reader_new_file() is used to read in files for importing, either .py or
.mpy files, for the lexer and persistent code loader respectively. In both
cases the file should be opened in raw bytes mode: the lexer handles
unicode characters itself, and .mpy files contain 8-bit bytes by nature.
Before this commit importing was working correctly because, although the
file was opened in text mode, all native filesystem implementations (POSIX,
FAT, LFS) would access the file in raw bytes mode via mp_stream_rw()
calling mp_stream_p_t.read(). So it was only an issue for non-native
filesystems, such as those implemented in Python. For Python-based
filesystem implementations, a call to mp_stream_rw() would go via IOBase
and then to readinto() at the Python level, and readinto() is only defined
on files opened in raw bytes mode.
Signed-off-by: Damien George <damien@micropython.org>
This version
* moves source files to reflect module structure
* adds inline documentation suitable for extract_pyi
* incompatibly moves spectrogram to fft
* incompatibly removes "extras"
There are some remaining markup errors in the specific revision of
extmod/ulab but they do not prevent the doc building process from
completing.
It raises on EOFError instead of an IncompleteReadError (which is what
CPython does). But the latter is derived from EOFError so code compatible
with MicroPython and CPython can be written by catching EOFError (eg see
included test).
Fixes issue #6156.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds human readable error messages when mbedtls or axtls raise
an exception. Currently often just an EIO error is raised so the user is
lost and can't tell whether it's a cert error, buffer overrun, connecting
to a non-ssl port, etc. The axtls and mbedtls error raising in the ussl
module is modified to raise:
OSError(-err_num, "error string")
For axtls a small error table of strings is added and used for the second
argument of the OSErrer. For mbedtls the code uses mbedtls' built-in
strerror function, and if there is an out of memory condition it just
produces OSError(-err_num). Producing the error string for mbedtls is
conditional on them being included in the mbedtls build, via
MBEDTLS_ERROR_C.
This commit adds the IRQ_GATTS_INDICATE_DONE BLE event which will be raised
with the status of gatts_indicate (unlike notify, indications require
acknowledgement).
An example of its use is added to ble_temperature.py, and to the multitests
in ble_characteristic.py.
Implemented for btstack and nimble bindings, tested in both directions
between unix/btstack and pybd/nimble.
The goal of this commit is to allow using ble.gatts_notify() at any time,
even if the stack is not ready to send the notification right now. It also
addresses the same issue for ble.gatts_indicate() and ble.gattc_write()
(without response). In addition this commit fixes the case where the
buffer passed to write-with-response wasn't copied, meaning it could be
modified by the caller, affecting the in-progress write.
The changes are:
- gatts_notify/indicate will now run in the background if the ACL buffer is
currently full, meaning that notify/indicate can be called at any time.
- gattc_write(mode=0) (no response) will now allow for one outstanding
write.
- gattc_write(mode=1) (with response) will now copy the buffer so that it
can't be modified by the caller while the write is in progress.
All four paths also now track the buffer while the operation is in
progress, which prevents the GC free'ing the buffer while it's still
needed.
With only `sp_func_proto_paren = remove` set there are some cases where
uncrustify misses removing a space between the function name and the
opening '('. This sets all of the related options to `force` as well.
The ring buffer previously used a single unsigned byte field to save the
length, meaning that it would overflow for large characteristic value
responses.
With this commit it now use a 16-bit length instead and has code to
explicitly truncate at UINT16_MAX (although this should be impossible to
achieve in practice).
This commit makes sure that all discovery complete and read/write status
events set the status to zero on success.
The status value will be implementation-dependent on non-success cases.
On btstack there's no status associated with the read result, it comes
through as a separate event. This allows you to detect read failures or
timeouts.
There doesn't appear to be any use for only triggering on specific events,
so it's just easier to number them sequentially. This makes them smaller
values so they take up only 1 byte in the ringbuf, only 1 byte for the
opcode in the bytecode, and makes room for more events.
Also add a couple of new event types that need to be implemented (to avoid
re-numbering later).
And rename _COMPLETE and _STATUS to _DONE for consistency.
In the future the "trigger" keyword argument can be reinstated by requiring
the user to compute the bitmask, eg:
ble.irq(handler, 1 << _IRQ_SCAN_RESULT | 1 << _IRQ_SCAN_DONE)
Before this change, any NimBLE error that does not appear in the
ble_hs_err_to_errno_table maps to return code 0, meaning success. If we
miss adding an error code to the table we end up returning success in case
of failure.
Instead, handle the zero case explicitly and default to MP_EIO. This
allows removing the now-redundant MP_EIO entries from the mapping.
This commit allows the user to set/get the GAP device name used by service
0x1800, characteristic 0x2a00. The usage is:
BLE.config(gap_name="myname")
print(BLE.config("gap_name"))
As part of this change the compile-time setting
MICROPY_PY_BLUETOOTH_DEFAULT_NAME is renamed to
MICROPY_PY_BLUETOOTH_DEFAULT_GAP_NAME to emphasise its link to GAP and this
new "gap_name" config value. And the default value of this for the NimBLE
bindings is changed from "PYBD" to "MPY NIMBLE" to be more generic.
Ujson should only worry about whitespace before JSON. This becomes apparent when you are using MP stream protocol to read directly from input buffers.
When you attempt to read(1) on a UART (and possibly other protocols) you have to wait for either the byte or the timeout.
Fixes:
- Waiting for a timeout after you have completed reading a correct and complete JSON off the input.
- Raising an OSError after reading a correct and complete JSON off the input.
- Eating more data than semantically owned off the input buffer.
- Blocking to start parsing JSON until the entire JSON body has been loaded into a potentially large, contiguous Python object.
Code you would write before:
```
line = board_busio_uart_port.read_line()
json_dict = json.loads(line)
```
or reaching for fixed buffers and swapping them around in Python.
Code that did not work before that does now:
```
json_dict = json.load(board_busio_uart_port)
```
- This removes the need for intermediate copies of data when reading JSON from micropython stream protocol inputs.
- It also increases total application speed by parsing JSON concurrently with receiving on boards that read from UART via DMA.
- It simplifies code that users write while improving their apps.
If the new name start with '/', cur_dir is not prepened any more, so that
the current working directory is respected. And extend the test cases for
rename to cover this functionality.
This change scans for '.', '..' and multiple '/' and normalizes the new
path name. If the resulting path does not exist, an error is raised.
Non-existing interim path elements are ignored if they are removed during
normalization.
This fixes the bug, that stat(filename) would not consider the current
working directory. So if e.g. the cwd is "lib", then stat("main.py") would
return the info for "/main.py" instead of "/lib/main.py".
For ports that have a system malloc which is not garbage collected (eg
unix, esp32), the stream object for the DB must be retained separately to
prevent it from being reclaimed by the MicroPython GC (because the
berkeley-db library uses malloc to allocate the DB structure which stores
the only reference to the stream).
Although in some cases the user code will explicitly retain a reference to
the underlying stream because it needs to call close() on it, this is not
always the case, eg in cases where the DB is intended to live forever.
Fixes issue #5940.
But only when bluetooth is enabled, i.e. if building the dev or coverage
variants, and we have libusb available.
Update travis to match, i.e. specify the variant when doing
`make submodules`.
This commit adds full support to the unix port for Bluetooth using the
common extmod/modbluetooth Python bindings. This uses the libusb HCI
transport, which supports many common USB BT adaptors.
This change is made for two reasons:
1. A 3rd-party library (eg berkeley-db-1.xx, axtls) may use the system
provided errno for certain errors, and yet MicroPython stream objects
that it calls will be using the internal mp_stream_errno. So if the
library returns an error it is not known whether the corresponding errno
code is stored in the system errno or mp_stream_errno. Using the system
errno in all cases (eg in the mp_stream_posix_XXX wrappers) fixes this
ambiguity.
2. For systems that have threading the system-provided errno should always
be used because the errno value is thread-local.
For systems that do not have an errno, the new lib/embed/__errno.c file is
provided.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
Initially some of these were found building the unix coverage variant on
MacOS because that build uses clang and has -Wdouble-promotion enabled, and
clang performs more vigorous promotion checks than gcc. Additionally the
codebase has been compiled with clang and msvc (the latter with warning
level 3), and with MICROPY_FLOAT_IMPL_FLOAT to find the rest of the
conversions.
Fixes are implemented either as explicit casts, or by using the correct
type, or by using one of the utility functions to handle floating point
casting; these have been moved from nativeglue.c to the public API.
Now that error string compression is supported it's more important to have
consistent error string formatting (eg all lowercase English words,
consistent contractions). This commit cleans up some of the strings to
make them more consistent.
This commit adds Loop.new_event_loop() which is used to reset the singleton
event loop. This functionality is put here instead of in Loop.close() to
make it possible to write code that is compatible with CPython.
The latest version of BTstack has a bug fixed so that it correctly
configures scan parameters if they are set right after activating the
stack. This means that BLE.gap_scan() will correctly set the scanning to
passive and so SCAN_RSP events are not passed through, so we don't need to
explicitly filter them in our bindings.
This commit makes all functions and function wrappers in modubinascii.c
STATIC and conditional on the MICROPY_PY_UBINASCII setting, which will
exclude the file from qstr/ compressed-string searching when ubinascii is
not enabled. The now-unused modubinascii.h header file is also removed.
The cc3200 port is updated accordingly to use this module in its entirety
instead of providing its own top-level definition of ubinascii.
This was originally like this because the cc3200 port has its own ubinascii
module which referenced these methods. The plan appeared to be that the
API might diverge (e.g. hardware crc), but this should be done similar to
I2C/SPI via a port-specific handler, rather than the port having its own
definition of the module. Having a centralised module definition also
enforces consistency of the API among ports.
This commit adds support for global exception handling in uasyncio
according to the CPython error handling:
https://docs.python.org/3/library/asyncio-eventloop.html#error-handling-api
This allows a program to receive exceptions from detached tasks and log
them to an appropriate location, instead of them being printed to the REPL.
The implementation preallocates a context dictionary so in case of an
exception there shouldn't be any RAM allocation.
The approach here is compatible with CPython except that in CPython the
exception handler is called once the task that threw an uncaught exception
is freed, whereas in MicroPython the exception handler is called
immediately when the exception is thrown.
These were found by buiding the unix coverage variant on macOS (so clang
compiler). Mostly, these are fixing implicit cast of float/double to
mp_float_t which is one of those two and one mp_int_t to size_t fix for
good measure.
https://www.python.org/dev/peps/pep-0475/
This implements something similar to PEP 475 on the unix port, and for the
VfsPosix class.
There are a few differences from the CPython implementation:
- Since we call mp_handle_pending() between any ENITR's, additional
functions could be called if MICROPY_ENABLE_SCHEDULER is enabled, not
just signal handlers.
- CPython only handles signal on the main thread, so other threads will
raise InterruptedError instead of retrying. On MicroPython,
mp_handle_pending() will currently raise exceptions on any thread.
A new macro MP_HAL_RETRY_SYSCALL is introduced to reduce duplicated code
and ensure that all instances behave the same. This will also allow other
ports that use POSIX-like system calls (and use, eg, VfsPosix) to provide
their own implementation if needed.
Implements Task and TaskQueue classes in C, using a pairing-heap data
structure. Using this reduces RAM use of each Task, and improves overall
performance of the uasyncio scheduler.
This commit adds a completely new implementation of the uasyncio module.
The aim of this version (compared to the original one in micropython-lib)
is to be more compatible with CPython's asyncio module, so that one can
more easily write code that runs under both MicroPython and CPython (and
reuse CPython asyncio libraries, follow CPython asyncio tutorials, etc).
Async code is not easy to write and any knowledge users already have from
CPython asyncio should transfer to uasyncio without effort, and vice versa.
The implementation here attempts to provide good compatibility with
CPython's asyncio while still being "micro" enough to run where MicroPython
runs. This follows the general philosophy of MicroPython itself, to make it
feel like Python.
The main change is to use a Task object for each coroutine. This allows
more flexibility to queue tasks in various places, eg the main run loop,
tasks waiting on events, locks or other tasks. It no longer requires
pre-allocating a fixed queue size for the main run loop.
A pairing heap is used to queue Tasks.
It's currently implemented in pure Python, separated into components with
lazy importing for optional components. In the future parts of this
implementation can be moved to C to improve speed and reduce memory usage.
But the aim is to maintain a pure-Python version as a reference version.
Also support MP_STREAM_GET_FILENO ioctl. The stdio flush change was done
previously for the unix port in 3e0b46b9af.
These changes make this POSIX file implementation equivalent to the unix
file implementation.
Fixes UDP non-blocking recv so it returns EAGAIN instead of ETIMEDOUT.
Timeout waiting for incoming data is also improved by replacing 100ms delay
with poll_sockets(), as is done in other parts of this module.
Fixes issue #5759.
This commit changes the BLE _IRQ_SCAN_RESULT data from:
addr_type, addr, connectable, rssi, adv_data
to:
addr_type, addr, adv_type, rssi, adv_data
This allows _IRQ_SCAN_RESULT to handle all scan result types (not just
connectable and non-connectable passive scans), and to distinguish between
them using adv_type which is an integer taking values 0x00-0x04 per the BT
specification.
This is a breaking change to the API, albeit a very minor one: the existing
connectable value was a boolean and True now becomes 0x00, False becomes
0x02.
Documentation is updated and a test added.
Fixes#5738.
This commit ensures that the BLE stack is active before allowing operations
that may otherwise crash if it's not active. It also clarifies the state
better (adding the "stopping" state) and renames mp_bluetooth_is_enabled to
the more self-explanatory mp_bluetooth_is_active.
This makes a cleaner separation between the: driver, HCI UART and BT stack.
Also updated the naming to be more consistent (mp_bluetooth_hci_*).
Work done in collaboration with Jim Mussared aka @jimmo.
Move extmod/modbluetooth_nimble.* to extmod/nimble. And move common
Makefile lines to extmod/nimble/nimble.mk (which was previously only used
by stm32). This allows (upcoming) btstack to follow a similar structure.
Work done in collaboration with Jim Mussared aka @jimmo.
This provides a more consistent C-level API to raise exceptions, ie moving
away from nlr_raise towards mp_raise_XXX. It also reduces code size by a
small amount on some ports.
Most types are in rodata/ROM, and mp_obj_base_t.type is a constant pointer,
so enforce this const-ness throughout the code base. If a type ever needs
to be modified (eg a user type) then a simple cast can be used.
The struct member "dest" should never be less than "destStart", so their
difference is never negative. Cast as such to make the comparison
explicitly unsigned, ensuring the compiler produces the correct comparison
instruction, and avoiding any compiler warnings.
Move webrepl support code from ports/esp8266/modules into extmod/webrepl
(to be alongside extmod/modwebrepl.c), and use frozen manifests to include
it in the build on esp8266 and esp32.
A small modification is made to webrepl.py to make it work on non-ESP
ports, i.e. don't call dupterm_notify if not available.
The size of the event ringbuf was previously fixed to compile-time config
value, but it's necessary to sometimes increase this for applications that
have large characteristic buffers to read, or many events at once.
With this commit the size can be set via BLE.config(rxbuf=512), for
example. This also resizes the internal event data buffer which sets the
maximum size of incoming data passed to the event handler.
Protocols are nice, but there is no way for C code to verify whether
a type's "protocol" structure actually implements some particular
protocol. As a result, you can pass an object that implements the
"vfs" protocol to one that expects the "stream" protocol, and the
opposite of awesomeness ensues.
This patch adds an OPTIONAL (but enabled by default) protocol identifier
as the first member of any protocol structure. This identifier is
simply a unique QSTR chosen by the protocol designer and used by each
protocol implementer. When checking for protocol support, instead of
just checking whether the object's type has a non-NULL protocol field,
use `mp_proto_get` which implements the protocol check when possible.
The existing protocols are now named:
protocol_framebuf
protocol_i2c
protocol_pin
protocol_stream
protocol_spi
protocol_vfs
(most of these are unused in CP and are just inherited from MP; vfs and
stream are definitely used though)
I did not find any crashing examples, but here's one to give a flavor of what
is improved, using `micropython_coverage`. Before the change,
the vfs "ioctl" protocol is invoked, and the result is not intelligible
as json (but it could have resulted in a hard fault, potentially):
>>> import uos, ujson
>>> u = uos.VfsPosix('/tmp')
>>> ujson.load(u)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: syntax error in JSON
After the change, the vfs object is correctly detected as not supporting
the stream protocol:
>>> ujson.load(p)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: stream operation not supported
This allows the user to explicitly select the behaviour of the write to the
remote peripheral. This is needed for peripherals that have
characteristics with WRITE_NO_RESPONSE set (instead of normal WRITE). The
function's signature is now:
BLE.gattc_write(conn_handle, value_handle, data, mode=0)
mode=0 means write without response, while mode=1 means write with
response. The latter was the original behaviour so this commit is a change
in behaviour of this method, and one should specify 1 as the 4th argument
to get back the old behaviour.
In the future there could be more modes supported, such as long writes.
The default protection for the BLE ringbuf is to use
MICROPY_BEGIN_ATOMIC_SECTION, which disables all interrupts. On stm32 it
only needs to disable the lowest priority IRQ, pendsv, because that's the
IRQ level at which the BLE stack is driven.
This removes the limit on data coming in from a BLE.gattc_read() request,
or a notify with payload (coming in to a central). In both cases the data
coming in to the BLE callback is now limited only by the available data in
the ringbuf, whereas before it was capped at (default hard coded) 20 bytes.
Instead of enqueue_irq() inspecting the ringbuf to decide whether to
schedule the IRQ callback (if ringbuf is empty), maintain a flag that knows
if the callback is on the schedule queue or not. This saves about 150
bytes of code (for stm32 builds), and simplifies all uses of enqueue_irq()
and schedule_ringbuf().
The address, adv payload and uuid fields of the event are pre-allocated by
modbluetooth, and reused in the IRQ handler. Simplify this and move all
storage into the `mp_obj_bluetooth_ble_t` instance.
This now allows users to hold on to a reference to these instances without
crashes, although they may be overwritten by future events. If they want
to hold onto the values longer term they need to copy them.
Remove existing scan result events from the ringbuf if the ringbuf is full
and we're trying to enqueue any other event. This is needed so that events
such as SCAN_COMPLETE are always put on the ringbuf.
This commit removes the Makefile-level MICROPY_FATFS config and moves the
MICROPY_VFS_FAT config to the Makefile level to replace it. It also moves
the include of the oofatfs source files in the build from each port to a
central place in extmod/extmod.mk.
For a port to enabled VFS FAT support it should now set MICROPY_VFS_FAT=1
at the level of the Makefile. This will include the relevant oofatfs files
in the build and set MICROPY_VFS_FAT=1 at the C (preprocessor) level.
POSIX poll should always return POLLERR and POLLHUP in revents, regardless
of whether they were requested in the input events flags.
See issues #4290 and #5172.
POSIX poll should always return POLLERR and POLLHUP in revents, regardless
of whether they were requested in the input events flags.
See issues #4290 and #5172.
- Adds an explicit way to set the size of a value's internal buffer,
replacing `ble.gatts_write(handle, bytes(size))` (although that
still works).
- Add an "append" mode for values, which means that remote writes
will append to the buffer.
This commit adds helper functions to call readblocks/writeblocks with a
fourth argument, the byte offset within a block.
Although the mp_vfs_blockdev_t struct has grown here by 2 machine words, in
all current uses of this struct within this repository it still fits within
the same number of GC blocks.
For consistency with "umachine". Now that weak links are enabled
by default for built-in modules, this should be a no-op, but allows
extension of the bluetooth module by user code.
Also move registration of ubluetooth to objmodule rather than
port-specific.
NimBLE doesn't actually copy this data, it requires it to stay live.
Only dereference when we register a new set of services.
Fixes#5226
This will allow incrementally adding services in the future, so
rename `reset` to `append` to make it clearer.
Internally change the representation of UUIDs to LE uint8* to simplify this.
This allows UUIDs to be easily used in BLE payloads (such as advertising).
Ref: #5186
This avoids a confusing ENOMEM raised from gap_advertise if there is
currently an active connection. This refers to the static connection
buffer pre-allocated by Nimble (nothing to do with MicroPython heap
memory).
This is to more accurately match the BLE spec, where intervals are
configured in units of channel hop time (625us). When it was
specified in ms, not all "valid" intervals were able to be
specified.
Now that we're also allowing configuration of scan interval, this
commit updates advertising to match.
This adds two additional optional kwargs to `gap_scan()`:
- `interval_us`: How long between scans.
- `window_us`: How long to scan for during a scan.
The default with NimBLE is a 11.25ms window with a 1.28s interval.
Changing these parameters is important for detecting low-frequency
advertisements (e.g. beacons).
Note: these params are in microseconds, not milliseconds in order
to allow the 625us granularity offered by the spec.
On other ports (e.g. ESP32) they provide a complete Nimble implementation
(i.e. we don't need to use the code in extmod/nimble). This change
extracts out the bits that we don't need to use in other ports:
- malloc/free/realloc for Nimble memory.
- pendsv poll handler
- depowering the cywbt
Also cleans up the root pointer management.
With this patch alignment is done relative to the start of the buffer that
is being unpacked, not the raw pointer value, as per CPython.
Fixes issue #3314.
As per the README.md of the upstream source at
https://github.com/B-Con/crypto-algorithms, this source code was released
into the public domain, so make that explicit in the copyright line in the
header.
Enabled via MICROPY_PY_URE_DEBUG, disabled by default (but enabled on unix
coverage build). This is a rarely used feature that costs a lot of code
(500-800 bytes flash). Debugging of regular expressions can be done
offline with other tools.
The helper function exec_user_callback executes within the context of an
lwIP C callback, and the user (Python) callback to be scheduled may want to
perform further TCP/IP actions, so the latter should be scheduled to run
outside the lwIP context (otherwise it's effectively a "hard IRQ" and such
callbacks have lots of restrictions).
If tcp_write returns ERR_MEM then it's not a fatal error but instead means
the caller should retry the write later on (and this is what lwIP's netconn
API does).
This fixes problems where a TCP send would raise OSError(ENOMEM) in
situations where the TCP/IP stack is under heavy load. See eg issues #1897
and #1971.
Setting MICROPY_PY_USSL and MICROPY_SSL_MBEDTLS at the Makefile-level will
now build mbedTLS from source and include it in the build, with the ussl
module using this TLS library. Extra settings like MBEDTLS_CONFIG_FILE may
need to be provided by a given port.
If a port wants to use its own mbedTLS library then it should not set
MICROPY_SSL_MBEDTLS at the Makefile-level but rather set it at the C level,
and provide the library as part of the build in its own way (see eg esp32
port).
In d5f0c87bb9 this call to tcp_poll() was
added to put a timeout on closing TCP sockets. But after calling
tcp_close() the PCB may be freed and therefore invalid, so tcp_poll() can
not be used at that point. As a fix this commit calls tcp_poll() before
closing the TCP PCB. If the PCB is subsequently closed and freed by
tcp_close() or tcp_abort() then the PCB will not be on any active list and
the callback will not be executed, which is the desired behaviour (the
_lwip_tcp_close_poll() callback only needs to be called if the PCB remains
active for longer than the timeout).
Commit 2848a613ac introduced a bug where
lwip_socket_free_incoming() accessed pcb.tcp->state after the PCB was
closed. The state may have changed due to that close call, or the PCB may
be freed and therefore invalid. This commit fixes that by calling
lwip_socket_free_incoming() before the PCB is closed.
For example: i2c.writevto(addr, (buf1, buf2)). This allows to efficiently
(wrt memory) write data composed of separate buffers, such as a command
followed by a large amount of data.
It consists of:
1. "do_handhake" param (default True) to wrap_socket(). If it's False,
handshake won't be performed by wrap_socket(), as it would be done in
blocking way normally. Instead, SSL socket can be set to non-blocking mode,
and handshake would be performed before the first read/write request (by
just returning EAGAIN to these requests, while instead reading/writing/
processing handshake over the connection). Unfortunately, axTLS doesn't
really support non-blocking handshake correctly. So, while framework for
this is implemented on MicroPython's module side, in case of axTLS, it
won't work reliably.
2. Implementation of .setblocking() method. It must be called on SSL socket
for blocking vs non-blocking operation to be handled correctly (for
example, it's not enough to wrap non-blocking socket with wrap_socket()
call - resulting SSL socket won't be itself non-blocking). Note that
.setblocking() propagates call to the underlying socket object, as
expected.
For this, add wrap_socket(do_handshake=False) param. CPython doesn't have
such a param at a module's global function, and at SSLContext.wrap_socket()
it has do_handshake_on_connect param, but that uselessly long.
Beyond that, make write() handle not just MBEDTLS_ERR_SSL_WANT_WRITE, but
also MBEDTLS_ERR_SSL_WANT_READ, as during handshake, write call may be
actually preempted by need to read next handshake message from peer.
Likewise, for read(). And even after the initial negotiation, situations
like that may happen e.g. with renegotiation. Both
MBEDTLS_ERR_SSL_WANT_READ and MBEDTLS_ERR_SSL_WANT_WRITE are however mapped
to the same None return code. The idea is that if the same read()/write()
method is called repeatedly, the progress will be made step by step anyway.
The caveat is if user wants to add the underlying socket to uselect.poll().
To be reliable, in this case, the socket should be polled for both POLL_IN
and POLL_OUT, as we don't know the actual expected direction. But that's
actually problematic. Consider for example that write() ends with
MBEDTLS_ERR_SSL_WANT_READ, but gets converted to None. We put the
underlying socket on pull using POLL_IN|POLL_OUT but that probably returns
immediately with POLL_OUT, as underlyings socket is writable. We call the
same ussl write() again, which again results in MBEDTLS_ERR_SSL_WANT_READ,
etc. We thus go into busy-loop.
So, the handling in this patch is temporary and needs fixing. But exact way
to fix it is not clear. One way is to provide explicit function for
handshake (CPython has do_handshake()), and let *that* return distinct
codes like WANT_READ/WANT_WRITE. But as mentioned above, past the initial
handshake, such situation may happen again with at least renegotiation. So
apparently, the only robust solution is to return "out of bound" special
sentinels like WANT_READ/WANT_WRITE from read()/write() directly. CPython
throws exceptions for these, but those are expensive to adopt that way for
efficiency-conscious implementation like MicroPython.
In CPython the random module is seeded differently on each import, and so
this new macro option MICROPY_PY_URANDOM_SEED_INIT_FUNC allows to implement
such a behaviour.
Since commit da938a83b5 the tcp_arg() that is
set for the new connection is the new connection itself, and the parent
listening socket is found in the pcb->connected entry.
Use uos.dupterm for REPL configuration of the main USB_VCP(0) stream on
dupterm slot 1, if USB is enabled. This means dupterm can also be used to
disable the boot REPL port if desired, via uos.dupterm(None, 1).
For efficiency this adds a simple hook to the global uos.dupterm code to
work with streams that are known to be native streams.
Some users of this module may require the LwIP stack to run at an elevated
priority, to protect against concurrency issues with processing done by the
underlying network interface. Since LwIP doesn't provide such protection
it must be done here (the other option is to run LwIP in a separate thread,
and use thread protection mechanisms, but that is a more heavyweight
solution).
The bug polling for readability was: if alloc==0 and tcp.item==NULL then
the code would incorrectly check tcp.array[iget] which is an invalid
dereference when alloc==0. This patch refactors the code to use a helper
function lwip_socket_incoming_array() to return the correct pointer for the
incomming connection array.
Fixes issue #4511.
This feature is controlled at compile time by MICROPY_PY_URE_SUB, disabled
by default.
Thanks to @dmazzella for the original patch for this feature; see #3770.
This feature is controlled at compile time by
MICROPY_PY_URE_MATCH_SPAN_START_END, disabled by default.
Thanks to @dmazzella for the original patch for this feature; see #3770.
This feature is controlled at compile time by MICROPY_PY_URE_MATCH_GROUPS,
disabled by default.
Thanks to @dmazzella for the original patch for this feature; see #3770.
As mentioned in #4450, `websocket` was experimental with a single intended
user, `webrepl`. Therefore, we'll make this change without a weak
link `websocket` -> `uwebsocket`.
Previously crypto-algorithms impl was included even if MICROPY_SSL_MBEDTLS
was in effect, thus we relied on the compiler/linker to cut out the unused
functions.
Python defines warnings as belonging to categories, where category is a
warning type (descending from exception type). This is useful, as e.g.
allows to disable warnings selectively and provide user-defined warning
types. So, implement this in MicroPython, except that categories are
represented just with strings. However, enough hooks are left to implement
categories differently per-port (e.g. as types), without need to patch each
and every usage.
This header is deprecated as of mbedtls 2.8.0, as shipped with Ubuntu
18.04. Leads to #warning which is promoted to error with uPy compile
options.
Note that the current version of mbedtls is 2.14 at the time of writing.
It's more robust to have the version defined statically in a header file,
rather than dynamically generating it via git using a git tag. In case
git doesn't exist, or a different source control tool is used, it's
important to still have the uPy version number available.
SHORT, INT, LONG, LONGLONG, and unsigned (U*) variants are being defined.
This is done at compile using GCC-style predefined macros like
__SIZEOF_INT__. If the compiler doesn't have such defines, no such types
will be defined.
The recent implementation of the listen backlog meant that the logic to
test for readability of such a socket changed, and this commit updates the
logic to work again.
Array to hold waiting connections is in-place if backlog=1, else is a
dynamically allocated array. Incoming connections are processed FIFO
style to maintain fairness.
sizeof() can work in two ways: a) calculate size of already instantiated
structure ("sizeof variable") - in this case we already no layout; b) size
of structure decsription ("sizeof type"). In the latter case, LAYOUT_NATIVE
was assumed, but there should possibility to calculate size for other
layouts too. So, with this patch, there're now 2 forms:
uctypes.sizeof(struct)
uctypes.sizeof(struct_desc, layout)
This value is unused. It was an artifact of early draft design, but
bitfields were optimized to use scalar one-word encoding, to allow
compact encoding of typical multiple bitfields in MCU control
registers.
Since mbedtls 2.7.0 new digest functions were introduced with a "_ret"
suffix to allow the functions to return an error message (eg, if the
underlying hardware acceleration failed). These new functions must be used
instead of the old ones to prevent deprecation warnings, or link errors for
missing functions, depending on the mbedtls configuration.
This removes the need for a separate axtls build stage, and builds all
axtls object files along with other code. This simplifies and cleans up
the build process, automatically builds axtls when needed, and puts the
axtls object files in the correct $(BUILD) location.
The MicroPython axtls configuration file is provided in
extmod/axtls-include/config.h
Commit 95e70cd0ea 'time: Use 1970 epoch' changed epoch for the time
module, but not for other users. This patch does the same for the only
other core timeutils user: extmod/vfs_fat.c:fat_vfs_stat().
Other timeutils users: cc3200, esp8266 and stm32, are not changed.
Ports that don't use long ints, will still get wrong time values from
os.stat().
This saves code space in builds which use link-time optimization.
The optimization drops the untranslated strings and replaces them
with a compressed_string_t struct. It can then be decompressed to
a c string.
Builds without LTO work as well but include both untranslated
strings and compressed strings.
This work could be expanded to include QSTRs and loaded strings if
a compress method is added to C. Its tracked in #531.
The underlying socket can handling polling, and any other transparent ioctl
requests. Note that CPython handles the case of polling an ssl object by
polling the file descriptor of the underlying socket file, and that
behaviour is emulated here.
Otherwise they may be called on a socket that no longer exists.
For example, if the GC calls the finaliser on the socket and then reuses
its heap memory, the "callback" entry of the old socket may contain invalid
data. If lwIP then calls the TCP callback the code may try to call the
user callback object which is now invalid. The lwIP callbacks must be
deregistered during the closing of the socket, before all the pcb pointers
are set to NULL.
This feature is controlled at compile time by MICROPY_PY_URE_SUB, disabled
by default.
Thanks to @dmazzella for the original patch for this feature; see #3770.
This feature is controlled at compile time by
MICROPY_PY_URE_MATCH_SPAN_START_END, disabled by default.
Thanks to @dmazzella for the original patch for this feature; see #3770.
This feature is controlled at compile time by MICROPY_PY_URE_MATCH_GROUPS,
disabled by default.
Thanks to @dmazzella for the original patch for this feature; see #3770.
This will allow implementations other than axTLS.
This commit includes additions of checks and clarifications of exceptions
related to user input.
To make the interface cleaner, I've disallowed switching from encrypt to
decrypt in the same object, as this is not always possible with other
crypto libraries (not all libraries have AES_convert_key like axTLS).
Allow including crypto consts based on compilation settings. Disabled by
default to reduce code size; if one wants extra code readability, can
enable them.
The API follows guidelines of https://www.python.org/dev/peps/pep-0272/,
but is optimized for code size, with the idea that full PEP 0272
compatibility can be added with a simple Python wrapper mode.
The naming of the module follows (u)hashlib pattern.
At the bare minimum, this module is expected to provide:
* AES128, ECB (i.e. "null") mode, encrypt only
Implementation in this commit is based on axTLS routines, and implements
following:
* AES 128 and 256
* ECB and CBC modes
* encrypt and decrypt
With this patch objects are only checked that they have the stream protocol
at the start of their use as a stream, and afterwards the efficient
mp_get_stream() helper is used to extract the stream protocol C methods.
This patch changes dupterm to call the native C stream methods on the
connected stream objects, instead of calling the Python readinto/write
methods. This is much more efficient for native stream objects like UART
and webrepl and doesn't require allocating a special dupterm array.
This change is a minor breaking change from the user's perspective because
dupterm no longer accepts pure user stream objects to duplicate on. But
with the recent addition of uio.IOBase it is possible to still create such
classes just by inheriting from uio.IOBase, for example:
import uio, uos
class MyStream(uio.IOBase):
def write(self, buf):
# existing write implementation
def readinto(self, buf):
# existing readinto implementation
uos.dupterm(MyStream())
Via the config value MICROPY_PY_UHASHLIB_SHA256. Default to enabled to
keep backwards compatibility.
Also add default value for the sha1 class, to at least document its
existence.
For consistency with other modules, and to help avoid clashes with the
actual underlying functions that do the hashing (eg
crypto-algorithms/sha256.c:sha256_update).
Following other C-level protocols, this VFS protocol is added to help
abstract away implementation details of the underlying VFS in an efficient
way. As a starting point, the import_stat function is put into this
protocol so that the VFS sub-system does not need to know about every VFS
implementation in order to do an efficient stat for importing files.
In the future it might be worth adding other functions to this protocol.
This VFS component allows to mount a host POSIX filesystem within the uPy
VFS sub-system. All traditional POSIX file access then goes through the
VFS, allowing to sandbox a uPy process to a certain sub-dir of the host
system, as well as mount other filesystem types alongside the host
filesystem.
If mbedtls_ctr_drbg_seed() is available in the mbedtls bulid then so should
be mbedtls_entropy_func(). Then it's up to the port to configure a valid
entropy source, eg via MBEDTLS_ENTROPY_HARDWARE_ALT.
Otherwise the "sock" member may have an undefined value if wrap_socket
fails with an exception and exits early, and then if the finaliser runs it
will try to close an invalid stream object.
Fixes issue #3828.
This matches CPython behaviour on Linux: a socket that is new and not
listening or connected is considered "hung up".
Thanks to @rkojedzinszky for the initial patch, PR #3457.
Textualy, the files in lib/uzlib/src were identical to the ones committed
in extmod/uzlib so there should be no behavioral change possible as a
result of this commit.
.. this maybe should be subject to MICROPY_CPYTHON_COMPAT, except that
is not defined in the main circuitpython ports so it would be a change
that makes no difference.
The order of function calls in an arithmetic expression is undefined and so
they must be written out as sequential statements.
Thanks to @dv-extrarius for reporting this issue, see issue #3690.
.. this maybe should be subject to MICROPY_CPYTHON_COMPAT, except that
is not defined in the main circuitpython ports so it would be a change
that makes no difference.
The 2nd and 3rd args of the ternary operator are treated like they are in
the same expression and must have similar types. void is not compatible
with int so that's why the compiler is complaining.
This patch moves the implementation of stream closure from a dedicated
method to the ioctl of the stream protocol, for each type that implements
closing. The benefits of this are:
1. Rounds out the stream ioctl function, which already includes flush,
seek and poll (among other things).
2. Makes calling mp_stream_close() on an object slightly more efficient
because it now no longer needs to lookup the close method and call it,
rather it just delegates straight to the ioctl function (if it exists).
3. Reduces code size and allows future types that implement the stream
protocol to be smaller because they don't need a dedicated close method.
Code size reduction is around 200 bytes smaller for x86 archs and around
30 bytes smaller for the bare-metal archs.
These allow accessing the filesystem label. For instance,
in boot.py, you can set the label on the built-in storage with:
storage.remount('/', False)
storage.getmount('/').label = "NEWLABEL"
storage.remount('/', True)
Users with multiple CIRCUITPY boards may find it desirable to
choose a different label for each board they own.
CPython doesn't allow SEEK_CUR with non-zero offset for files in text mode,
and uPy inherited this behaviour for both text and binary files. It makes
sense to provide full support for SEEK_CUR of binary-mode files in uPy, and
to do this in a minimal way means also allowing to use SEEK_CUR with
non-zero offsets on text-mode files. That seems to be a fair compromise.
This patch takes the software SPI implementation from extmod/machine_spi.c
and moves it to a dedicated file in drivers/bus/softspi.c. This allows the
SPI driver to be used independently of the uPy runtime, making it a more
general component.
This patch eliminates heap allocation in the VFS FAT disk IO layer, when
calling the underlying readblocks/writeblocks methods. The bytearray
object that is passed to these methods is now allocated on the C stack
rather than the heap (it's only 4 words big).
This means that these methods should not retain a pointer to the buffer
object that is passed in, but this was already a restriction because the
original heap-allocated bytearray had its buffer passed by reference.
This patch just moves the definition of the wrapper object fat_vfs_open_obj
to the location of the definition of its function, which matches how it's
done in most other places in the code base.
The fat_vfs_ilistdir2() function was only used by fat_vfs_ilistdir_func()
so moving the former into the same file as the latter allows it to be
placed directly into the latter function, thus saving code size.
CPython doesn't allow SEEK_CUR with non-zero offset for files in text mode,
and uPy inherited this behaviour for both text and binary files. It makes
sense to provide full support for SEEK_CUR of binary-mode files in uPy, and
to do this in a minimal way means also allowing to use SEEK_CUR with
non-zero offsets on text-mode files. That seems to be a fair compromise.
Dramatically improves TCP sending throughput because without an explicit
call to tcp_output() the data is only sent to the lower layers via the
lwIP slow timer which (by default) ticks every 500ms.
Before this patch MP_BINARY_OP_IN had two meanings: coming from bytecode it
meant that the args needed to be swapped, but coming from within the
runtime meant that the args were already in the correct order. This lead
to some confusion in the code and comments stating how args were reversed.
It also lead to 2 bugs: 1) containment for a subclass of a native type
didn't work; 2) the expression "{True} in True" would illegally succeed and
return True. In both of these cases it was because the args to
MP_BINARY_OP_IN ended up being reversed twice.
To fix these things this patch introduces MP_BINARY_OP_CONTAINS which
corresponds exactly to the __contains__ special method, and this is the
operator that built-in types should implement. MP_BINARY_OP_IN is now only
emitted by the compiler and is converted to MP_BINARY_OP_CONTAINS by
swapping the arguments.
It's possible to use the methods (eg ilistdir) of a VFS FatFS object
without it being mounted in the VFS itself. This previously worked but
only because FatFS was "mounting" the filesystem automatically when any
function (eg f_opendir) was called. But it didn't work for ports that used
synchronisation objects (_FS_REENTRANT) because they are only initialised
via a call to f_mount. So, call f_mount explicitly when creating a new
FatFS object so that everything is set up correctly. Then also provide a
finaliser to do the f_umount call, but only if synchronisation objects are
enabled (since otherwise the f_umount call does nothing).
This patch simplifies the str creation API to favour the common case of
creating a str object that is not forced to be interned. To force
interning of a new str the new mp_obj_new_str_via_qstr function is added,
and should only be used if warranted.
Apart from simplifying the mp_obj_new_str function (and making it have the
same signature as mp_obj_new_bytes), this patch also reduces code size by a
bit (-16 bytes for bare-arm and roughly -40 bytes on the bare-metal archs).
