Two problems: The lead byte for 3-byte sequences was wrong, and one
mid-byte was not even filled in due to a missing "++"!
Apparently this was broken ever since the first "Compress as unicode,
not bytes" commit, but I believed I'd "tested" it by running on the
Pinyin translation.
This rendered at least the Korean and Japanese translations completely
illegible, affecting 5.0 and all later releases.
Compress common unicode bigrams by making code points in the range
0x80 - 0xbf (inclusive) represent them. Then, they can be greedily
encoded and the substituted code points handled by the existing Huffman
compression. Normally code points in the range 0x80-0xbf are not used
in Unicode, so we stake our own claim. Using the more arguably correct
"Private Use Area" (PUA) would mean that for scripts that only use
code points under 256 we would use more memory for the "values" table.
bigram means "two letters", and is also sometimes called a "digram".
It's nothing to do with "big RAM". For our purposes, a bigram represents
two successive unicode code points, so for instance in our build on
trinket m0 for english the most frequent are:
['t ', 'e ', 'in', 'd ', ...].
The bigrams are selected based on frequency in the corpus, but the
selection is not necessarily optimal, for these reasons I can think of:
* Suppose the corpus was just "tea" repeated 100 times. The
top bigrams would be "te", and "ea". However,
overlap, "te" could never be used. Thus, some bigrams might actually
waste space
* I _assume_ this has to be why e.g., bigram 0x86 "s " is more
frequent than bigram 0x85 " a" in English for Trinket M0, because
sequences like "can't add" would get the "t " digram and then
be unable to use the " a" digram.
* And generally, if a bigram is frequent then so are its constituents.
Say that "i" and "n" both encode to just 5 or 6 bits, then the huffman
code for "in" had better compress to 10 or fewer bits or it's a net
loss!
* I checked though! "i" is 5 bits, "n" is 6 bits (lucky guess)
but the bigram 0x83 also just 6 bits, so this one is a win of
5 bits for every "it" minus overhead. Yay, this round goes to team
compression.
* On the other hand, the least frequent bigram 0x9d " n" is 10 bits
long and its constituent code points are 4+6 bits so there's no
savings, but there is the cost of the table entry.
* and somehow 0x9f 'an' is never used at all!
With or without accounting for overlaps, there is some optimum number
of bigrams. Adding one more bigram uses at least 2 bytes (for the
entry in the bigram table; 4 bytes if code points >255 are in the
source text) and also needs a slot in the Huffman dictionary, so
adding bigrams beyond the optimim number makes compression worse again.
If it's an improvement, the fact that it's not guaranteed optimal
doesn't seem to matter too much. It just leaves a little more fruit
for the next sweep to pick up. Perhaps try adding the most frequent
bigram not yet present, until it doesn't improve compression overall.
Right now, de_DE is again the "fullest" build on trinket_m0. (It's
reclaimed that spot from the ja translation somehow) This change saves
104 bytes there, increasing free space about 6.8%. In the larger
(but not critically full) pyportal build it saves 324 bytes.
The specific number of bigrams used (32) was chosen as it is the max
number that fit within the 0x80..0xbf range. Larger tables would
require the use of 16 bit code points in the de_DE build, losing savings
overall.
(Side note: The most frequent letters in English have been said
to be: ETA OIN SHRDLU; but we have UAC EIL MOPRST in our corpus)
Otherwise, out of range writes would occur in tilegrid_set_tile, causing a safe mode reset.
```
Hardware watchpoint 6: -location *stack_alloc->ptr
Old value = 24652061
New value = 24641565
0x000444f2 in common_hal_displayio_tilegrid_set_tile (self=0x200002c8 <supervisor_terminal_text_grid>, x=1, y=1, tile_index=0 '\000')
at ../../shared-module/displayio/TileGrid.c:236
236 if (!self->partial_change) {
(gdb)
```
.. however, the number of endpoints is only set for SAMD (8).
Other ports need to set the value. Otherwise, the build will show
the message
```
Unable to check whether maximum number of endpoints is respected
```
The font is missing many characters and the build needs the space.
We can optimize font storage when we get a good font.
The serial output will work as usual.
Before this, a background callback that was on the list when
background_callback_reset was called could have ended up in a state
that made it "un-queueable": its "prev" pointer could have been non-NULL.
A background callback must never outlive its related object. By
collecting the head of the linked list of background tasks, this will
not happen.
One hypothetical case where this could happen is if an MP3Decoder is
deleted while its callback to fill its buffer is scheduled.
CALLBACK_CRITICAL_BEGIN is heavyweight, but we can be confident we do
not have work to do as long as callback_head is NULL.
This gives back performance on nRF.
In time, we should transition interrupt driven background tasks out of the
overall run_background_tasks into distinct background callbacks,
so that the number of checks that occur with each tick is reduced.
This restores the ability to remove CDC and/or MSC, at the price of
giving up the new automatic check that USB_DEVICES is correct.
Since devices have to have CDC and MSC to be "CircuitPython",
this is not a facility that is going to be used by any in-tree drivers.
Since Actions passed on the previous commit, where this computed value
was checked against the specified value (if any), this is no net change,
except that we no longer need to specify it for particular boards or
ports.
Few peripherals are actually tested. However, USB, I2C and GPIO seem to work.
Most pins are silkscreened with the "PX00" style, so the board module
only includes the small number that are screened differently.
The default SPI, I2C, and UART are the ones on the EXT2 header. This is
arbitrary, but the I2C on this connector is shared with the on-board I2C
devices and the PCC header, making it the most versatile.
Length was stored as a 16-bit number always. Most translations have
a max length far less. For example, US English translation lengths
always fit in just 8 bits. probably all languages fit in 9 bits.
This also has the side effect of reducing the alignment of
compressed_string_t from 2 bytes to 1.
testing performed: ran in german and english on pyruler, printed messages
looked right.
Firmware size, en_US
Before: 3044 bytes free in flash
After: 3408 bytes free in flash
Firmware size, de_DE (with #2967 merged to restore translations)
Before: 1236 bytes free in flash
After: 1600 bytes free in flash
Check to see if the current exception is a Watchdog exception, if it's
enabled. This ensures we break out of the current sleep() if a watchdog
timeout hits.
Signed-off-by: Sean Cross <sean@xobs.io>
Allow for passing `-DCFG_TUSB_DEBUG=1` or `-DCFG_TUSB_DEBUG=2` on the
command line to enable debugging tinyusb within circuitpython.
Signed-off-by: Sean Cross <sean@xobs.io>
Add a field to allow specifying a timeout when initiating advertising.
As part of this, add a new property to determine if the device is still
advertising.
Additionally, have the `anonymous` property require a timeout, and set
the timeout to the maximum possible value if no timeout is specified.
Signed-off-by: Sean Cross <sean@xobs.io>
Add a new parameter to the `start_advertising()` function to enable
anonymous advertising. This forces a call to `sd_ble_gap_privacy_set()`
with `privacy_mode` set to `BLE_GAP_PRIVACY_MODE_DEVICE_PRIVACY` and
`private_addr_type` set to
`BLE_GAP_ADDR_TYPE_RANDOM_PRIVATE_RESOLVABLE`.
With this, addresses will cycle at a predefined rate (currently once
every 15 minutes).
Signed-off-by: Sean Cross <sean@xobs.io>
This gets all the purely internal references. Some uses of
protomatter/Protomatter/PROTOMATTER remain, as they are references
to symbols in the Protomatter C library itself.
- bump supervisor alloc count by 4 (we actually use 5)
- move reconstruct to after gc heap is reset
- destroy protomatter object entirely if not used by a FramebufferDisplay
- ensure previous supervisor allocations are released
- zero out pointers so GC can collect them
testing performed:
* successfully store and retrieve a 500kB file on the flash
* square wave output on each pin appears on o'scope
* board.SPI(), board.SERIAL(), board.I2C() all construct
Introduces a way to place CircuitPython code and data into
tightly coupled memory (TCM) which is accessible by the CPU in a
single cycle. It also frees up room in the corresponding cache for
intermittent data. Loading from external flash is slow!
The data cache is also now enabled.
Adds support for the iMX RT 1021 chip. Adds three new boards:
* iMX RT 1020 EVK
* iMX RT 1060 EVK
* Teensy 4.0
Related to #2492, #2472 and #2477. Fixes#2475.
It's extremely dubious that we have these handles that we think
are to GC'd memory at a time when the gc pool may not be initialized.
Hopefully, they WERE valid GC memory and are undisturbed by the teardown
of the interpreter that can lead to this state.
In this case, don't try to m_free them, the memory will become free when
the GC heap is reinitialized.
Closes: #2338 (together with previous commit)
By treating each unicode code-point as a single entity for huffman
compression, the overall compression rate can be somewhat improved
without changing the algorithm. On the decompression side, when
compressed values above 127 are encountered, they need to be
converted from a 16-bit Unicode code point into a UTF-8 byte
sequence.
Doing this returns approximately 1.5kB of flash storage with the
zh_Latn_pinyin translation. (292 -> 1768 bytes remaining in my build
of trinket_m0)
Other "more ASCII" translations benefit less, and in fact
zh_Latn_pinyin is no longer the most constrained translation!
(de_DE 1156 -> 1384 bytes free in flash, I didn't check others
before pushing for CI)
English is slightly pessimized, 2840 -> 2788 bytes, probably mostly
because the "values" array was changed from uint8_t to uint16_t,
which is strictly not required for an all-ASCII translation. This
could probably be avoided in this case, but as English is not the
most constrained translation it doesn't really matter.
Testing performed: built for feather nRF52840 express and trinket m0
in English and zh_Latn_pinyin; ran and verified the localized
messages such as
Àn xià rènhé jiàn jìnrù REPL. Shǐyòng CTRL-D chóngxīn jiāzài.
and
Press any key to enter the REPL. Use CTRL-D to reload.
were properly displayed.
This improves performance of running python code by 34%, based
on the "pystone" benchmark on metro m4 express at 5000 passes
(1127.65 -> 1521.6 passes/second).
In addition, by instrumenting the tick function and monitoring on an
oscilloscope, the time actually spent in run_background_tasks() on
the metro m4 decreases from average 43% to 0.5%. (however, there's
some additional overhead that is moved around and not accounted for
in that "0.5%" figure, each time supervisor_run_background_tasks_if_tick
is called but no tick has occurred)
On the CPB, it increases pystone from 633 to 769, a smaller percentage
increase of 21%. I did not measure the time actually spent in
run_background_tasks() on CPB.
Testing performed: on metro m4 and cpb, run pystone adapted from python3.4
(change time.time to time.monotonic for sub-second resolution)
Besides running a 5000 pass test, I also ran a 50-pass test while
scoping how long an output pin was set. Average: 34.59ms or 1445/s on m4,
67.61ms or 739/s on cbp, both matching the other pystone result reasonably
well.
import pystone
import board
import digitalio
import time
d = digitalio.DigitalInOut(board.D13)
d.direction = digitalio.Direction.OUTPUT
while True:
d.value = 0
time.sleep(.01)
d.value = 1
pystone.main(50)
This code is shared by most parts, except where not all the #ifdefs
inside the tick function were present in all ports. This mostly would
have broken gamepad tick support on non-samd ports.
The "ms32" and "ms64" variants of the tick functions are introduced
because there is no 64-bit atomic read. Disabling interrupts avoids
a low probability bug where milliseconds could be off by ~49.5 days
once every ~49.5 days (2^32 ms).
Avoiding disabling interrupts when only the low 32 bits are needed is a minor
optimization.
Testing performed: on metro m4 express, USB still works and
time.monotonic_ns() still counts up
In cases where more than one board is connected to a single computer it can become pretty hard to figure out which board you're actually talking to. For example, if you have several MIDI-compatible boards they all show up as "CircuitPython MIDI". This change allows boards to replace the "CircuitPython" part of their USB descriptors with more specific text, for example, "CircuitPython Feather" or just "Feather". This will let folks more easily tell boards apart.
The new option is named `USB_INTERFACE_NAME` and is available in `mkconfigboard.mk`. For example:
```
USB_INTERFACE_NAME = "Feather"
```
This PR refines the _bleio API. It was originally motivated by
the addition of a new CircuitPython service that enables reading
and modifying files on the device. Moving the BLE lifecycle outside
of the VM motivated a number of changes to remove heap allocations
in some APIs.
It also motivated unifying connection initiation to the Adapter class
rather than the Central and Peripheral classes which have been removed.
Adapter now handles the GAP portion of BLE including advertising, which
has moved but is largely unchanged, and scanning, which has been enhanced
to return an iterator of filtered results.
Once a connection is created (either by us (aka Central) or a remote
device (aka Peripheral)) it is represented by a new Connection class.
This class knows the current connection state and can discover and
instantiate remote Services along with their Characteristics and
Descriptors.
Relates to #586
The GitHub Actions build for this PR is failing for reasons that make no sense. Make a tiny commit change to see if it will refresh things and work again.
This also improves Palette so it stores the original RGB888 colors.
Lastly, it adds I2CDisplay as a display bus to talk over I2C. Particularly
useful for the SSD1306.
Fixes#1828. Fixes#1956
This fixes the bug that bitmap changes do not cause screen updates
and optimizes the refresh when the bitmap is simply shown on the
screen. If the bitmap is used in tiles, then changing it will
cause all TileGrids using it to do a full refresh.
Fixes#1981
Different operations to the display tree have different costs. Be
aware of these costs when optimizing your code.
* Changing tiles indices in a TileGrid will update an area
covering them all.
* Changing a palette will refresh every object that references it.
* Moving a TileGrid will update both where it was and where it moved to.
* Adding something to a Group will refresh each individual area it
covers.
* Removing things from a Group will refresh one area that covers all
previous locations. (Not separate areas like add.)
* Setting a new top level Group will refresh the entire display.
Only TileGrid moves are optimized for overlap. All other overlaps
cause sending of duplicate pixels.
This also adds flip_x, flip_y and transpose_xy to TileGrid. They
change the direction of the pixels but not the location.
Fixes#1169. Fixes#1705. Fixes#1923.
This changes the displayio pixel computation from per-pixel to
per-area. This is precursor work to updating portions of the screen
(#1169). It should provide mild speedups because bounds checks are
done once per area rather than once per pixel. Filling by area also
allows TileGrid to maintain a row-associative fill pattern even when
the display's refresh is orthogonal to it.
If one of the default pins was already in use it would crash.
The internal API has been refined to allow us to get the value
without causing an init of the singleton.
Fixes#1753
Without this, a double free can occur when a display (and terminal)
is released and then a crash occurs. Upon a second release,
different memory is released (sometimes the heap). When this is
followed by an allocation for the flash cache, the cache can
overwrite the active heap causing crashes.
Fixes#1667
This changes a number of things in displayio:
* Introduces BuiltinFont and Glyph so the built in font can be used by libraries. For boards with
a font it is available as board.TERMINAL_FONT. Fixes#1172
* Remove _load_row from Bitmap in favor of bitmap[] access. Index can be x/y tuple or overall index. Fixes#1191
* Add width and height properties to Bitmap.
* Add insert and [] access to Group. Fixes#1518
* Add index param to pop on Group.
* Terminal no longer takes unicode character info. It takes a BuiltinFont instead.
* Fix Terminal's handling of [###D vt100 commands used when up arrowing into repl history.
* Add x and y positions to Group plus scale as well.
* Add bitmap accessor for BuiltinFont
* Fix Hallowing.
* Fix builds without displayio.
* Fix y bounds that appears as untrollable row of pixels.
* Add scrolling to TileGrid.
* Remove Sprite to save space. TileGrid is a drop in replacement.
This creates a common safe mode mechanic that ports can share.
As a result, the nRF52 now has safe mode support as well.
The common safe mode adds a 700ms delay at startup where a reset
during that window will cause a reset into safe mode. This window
is designated by a yellow status pixel and flashing the single led
three times.
A couple NeoPixel fixes are included for the nRF52 as well.
Fixes#1034. Fixes#990. Fixes#615.
The backtrace cannot be given because it relies on the validity
of the qstr data structures on the heap which may have been
corrupted.
In fact, it still can crash hard when the bytecode itself is
overwritten. To fix, we'd need a way to skip gathering the
backtrace completely.
This also increases the default stack size on M4s so it can
accomodate the stack needed by ASF4s nvm API.
We were writing with quad page program including the address (0x38)
which is unsupported by the GD25Q16C but it is supported by the
flash on the DK. So, we use the single address, quad data command
(0x32).
This started while adding USB MIDI support (and descriptor support is
in this change.) When seeing that I'd have to implement the MIDI class
logic twice, once for atmel-samd and once for nrf, I decided to refactor
the USB stack so its shared across ports. This has led to a number of
changes that remove items from the ports folder and move them into
supervisor.
Furthermore, we had external SPI flash support for nrf pending so I
factored out the connection between the usb stack and the flash API as
well. This PR also includes the QSPI support for nRF.
This saves code space in builds which use link-time optimization.
The optimization drops the untranslated strings and replaces them
with a compressed_string_t struct. It can then be decompressed to
a c string.
Builds without LTO work as well but include both untranslated
strings and compressed strings.
This work could be expanded to include QSTRs and loaded strings if
a compress method is added to C. Its tracked in #531.
Its slimmed down by removing the qstr and bit packing TCC info.
The trinket m0 build actually grows by 20 bytes. The arduino zero
build shrinks by 188 bytes.
This allows for the heap to fill all space but the stack. It also
allows us to designate space for memory outside the runtime for
things such as USB descriptors, flash cache and main filename.
Fixes#754
* Introduce a python script to generate the USB descriptor instead of
a bunch of C macros. In the future, we can use this dynamically in
CircuitPython.
* Add support for detecting read-only mass storage mounts.
Fixes#377
* atmel-samd: Add support for internal filesystems.
This allows us to re-enable `os`. `random` is also enabled because
it solely depends on `os`.
Fixes#266. Its also a pre-requisite for #260.
* atmel-samd: Update SAMD51 linker script comments and MICROPY_MAX_STACK_USAGE enabling.
* atmel-samd: Support patching after updating ASF4.
This makes it possible to automatically fix newer code.
* atmel-samd: Update ASF4 to include flash APIs for SAMD51.
This is the first automatic update that has caused a few deletions
where code was previously copied instead of moved.
This is a prerequisite for #260.
* atmel-samd: Remove ASF3. This will break builds.
* atmel-samd: Add ASF4 for the SAMD21 and SAMD51.
* Introduce the supervisor concept to facilitate porting.
The supervisor is the code which runs individual MicroPython VMs. By
splitting it out we make it more consistent and easier to find.
This also adds very basic SAMD21 and SAMD51 support using the
supervisor. Only the REPL currently works.
This begins the work for #178.
