* Always clear the peripheral interrupt so we don't hang when full
* Store the ringbuf in the object so it gets collected when we're alive
* Make UART objects have a finaliser so they are deinit when their
memory is freed
* Copy bytes into the ringbuf from the FIFO after we read to ensure
the interrupt is enabled ASAP
* Copy bytes into the ringbuf from the FIFO before measuring our
rx available because the interrupt is based on a threshold (not
> 0). For example, a single byte won't trigger an interrupt.
Ujson should only worry about whitespace before JSON. This becomes apparent when you are using MP stream protocol to read directly from input buffers.
When you attempt to read(1) on a UART (and possibly other protocols) you have to wait for either the byte or the timeout.
Fixes:
- Waiting for a timeout after you have completed reading a correct and complete JSON off the input.
- Raising an OSError after reading a correct and complete JSON off the input.
- Eating more data than semantically owned off the input buffer.
- Blocking to start parsing JSON until the entire JSON body has been loaded into a potentially large, contiguous Python object.
Code you would write before:
```
line = board_busio_uart_port.read_line()
json_dict = json.loads(line)
```
or reaching for fixed buffers and swapping them around in Python.
Code that did not work before that does now:
```
json_dict = json.load(board_busio_uart_port)
```
- This removes the need for intermediate copies of data when reading JSON from micropython stream protocol inputs.
- It also increases total application speed by parsing JSON concurrently with receiving on boards that read from UART via DMA.
- It simplifies code that users write while improving their apps.
Surely readline() "rtype" is string not int as stated (and not bytes as some might expect).
Also it is not totally unambiguous what happens on a timeout so it would help to clarify in docs that on a timeout
it does NOT return with what it has read so far, rather it leaves all that in the buffer ready for a future read and returns nothing.
Likewise clarify that if timeout=0 but there is no newline it DOES return what it has read so far (NOT None).
At least this is what I think it does and/or is supposed to do!
Python docs are generally not too explicit about what is the proper treatment, so perhaps all the more reason to
clarify the interpretation adopted?
Protocols are nice, but there is no way for C code to verify whether
a type's "protocol" structure actually implements some particular
protocol. As a result, you can pass an object that implements the
"vfs" protocol to one that expects the "stream" protocol, and the
opposite of awesomeness ensues.
This patch adds an OPTIONAL (but enabled by default) protocol identifier
as the first member of any protocol structure. This identifier is
simply a unique QSTR chosen by the protocol designer and used by each
protocol implementer. When checking for protocol support, instead of
just checking whether the object's type has a non-NULL protocol field,
use `mp_proto_get` which implements the protocol check when possible.
The existing protocols are now named:
protocol_framebuf
protocol_i2c
protocol_pin
protocol_stream
protocol_spi
protocol_vfs
(most of these are unused in CP and are just inherited from MP; vfs and
stream are definitely used though)
I did not find any crashing examples, but here's one to give a flavor of what
is improved, using `micropython_coverage`. Before the change,
the vfs "ioctl" protocol is invoked, and the result is not intelligible
as json (but it could have resulted in a hard fault, potentially):
>>> import uos, ujson
>>> u = uos.VfsPosix('/tmp')
>>> ujson.load(u)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: syntax error in JSON
After the change, the vfs object is correctly detected as not supporting
the stream protocol:
>>> ujson.load(p)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: stream operation not supported
Particularly when they have buffers that are written via IRQ or DMA,
UART objects do not relocate gracefully. If such an object is
relocated to the long-lived pool after its original creation, the
IRQ or DMA will write to an unexpected location within the Python
heap, leading to a variety of symptoms. The most frequent symptom
is inability to read from the UART.
Consider the particular case of atmel-samd: usart_uart_obj_t
contains a usart_async_descriptor contains a _usart_async_device.
In _sercom_init_irq_param the address of this contained
_usart_async_device is assigned to a global array
sercom_to_sercom_dev which is later used from the interrupt context
_sercom_usart_interrupt_handler to store the received data in the
right ring buffer.
When the UART object is relocated to the long-lived heap, there's no
mechanism to re-point these internal pointers, so instead take the
cowardly way and allocate the UART object as long-lived.
Happily, almost all UART objects are likely to be long-lived, so
this is unlikely to have a negative effect on memory usage or heap
fragmentation.
Closes: #1056
microDev
Scott Shawcroft
Dan Halbert
Taku Fukada
dherrada
dherrada
Scott Shawcroft
Mark Olsson
warriorofwire
dherrada
AndrewR-L
Dave Marples
Jeff Epler
Lionel Debroux