Instead of counting words in make, which is slightly awful, notice that
possible_devices is local to external_flash.c, so we can declare the array
with an automatic bound, and then get the count as the element-count
(MP_ARRAY_SIZE) of the array.
Since EXTERNAL_FLASH_DEVICE_COUNT is no longer a global macro, switch
a few sites to using EXTERNAL_FLASH_DEVICES in `#if` checks instead.
This adds some additional code in mkfs which doesn't seem necessary, and
Disabling it saves 172 bytes flash.
Testing performed: Using a Feather M0 Adalogger, checked that
* an sdcard could still be mounted (using adafruit_sdcard)
* os.listdir() of "/" and "/sd" worked
* CIRCUITPY still mounted
This is a first go at it, done by naive replacing of all array
operations with corresponding operations on the list. Note that
there is a lot of unnecessary type conversions, here. Also, list_pop
has been copied, because it's decalerd STATIC in py/objlist.h
Since we want to expose the list of group's children to the user,
we should only have the original objects in it, without any other
additional data, and compute the native object as needed.
This saves about 60 bytes (Feather M4 went from 45040 -> 45100 bytes free)
66 bytes of data eliminated, but 6 bytes paid back to initialize the length
field.
The RP2040 is new microcontroller from Raspberry Pi that features
two Cortex M0s and eight PIO state machines that are good for
crunching lots of data. It has 264k RAM and a built in UF2
bootloader too.
Datasheet: https://pico.raspberrypi.org/files/rp2040_datasheet.pdf
Some ports need an extra operation to ensure that the main task is
awoken so that a queued background task will execute during an ongoing
light sleep.
This removes the need to enable supervisor ticks while I2SOut is operating.
Closes: #3952
This changes lots of files to unify `board.h` across ports. It adds
`board_deinit` when CIRCUITPY_ALARM is set. `main.c` uses it to
deinit the board before deep sleeping (even when pretending.)
Deep sleep is now a two step process for the port. First, the
port should prepare to deep sleep based on the given alarms. It
should set alarms for both deep and pretend sleep. In particular,
the pretend versions should be set immediately so that we don't
miss an alarm as we shutdown. These alarms should also wake from
`port_idle_until_interrupt` which is used when pretending to deep
sleep.
Second, when real deep sleeping, `alarm_enter_deep_sleep` is called.
The port should set any alarms it didn't during prepare based on
data it saved internally during prepare.
ESP32-S2 sleep is a bit reorganized to locate more logic with
TimeAlarm. This will help it scale to more alarm types.
Fixes#3786
This allows calls to `allocate_memory()` while the VM is running, it will then allocate from the GC heap (unless there is a suitable hole among the supervisor allocations), and when the VM exits and the GC heap is freed, the allocation will be moved to the bottom of the former GC heap and transformed into a proper supervisor allocation. Existing movable allocations will also be moved to defragment the supervisor heap and ensure that the next VM run gets as much memory as possible for the GC heap.
By itself this breaks terminalio because it violates the assumption that supervisor_display_move_memory() still has access to an undisturbed heap to copy the tilegrid from. It will work in many cases, but if you're unlucky you will get garbled terminal contents after exiting from the vm run that created the display. This will be fixed in the following commit, which is separate to simplify review.
This makes a more useful display on the portrait magtag, allowing 21
characters across instead of just 18. There are 20 full rows of text,
instead of 21. The total number of characters increases slightly from 378
to 420.
For comparison, the Commodore VIC 20 had 22 rows of 23 characters for a
total of 506 characters. :-P
* No weak link for modules. It only impacts _os and _time and is
already disabled for non-full builds.
* Turn off PA00 and PA01 because they are the crystal on the Metro
M0 Express.
* Change ejected default to false to move it to BSS. It is set on
USB connection anyway.
* Set sinc_filter to const. Doesn't help flash but keeps it out of
RAM.
This unifies the flash config to the settings used by the Boot ROM.
This makes the config unique per board which allows for changing
quad enable and status bit differences per flash device. It also
allows for timing differences due to the board layout.
This change also tweaks linker layout to leave more ram space for
the CircuitPython heap.
This requires recovering the pointer of the allocation, which could be done by adding up neighbor lengths, but the simpler way is to stop NULLing it out in the first place and instead mark an allocation as freed by the client by setting the lowest bit of the length (which is always zero in a valid length).
When allocations were freed in a different order from the reverse of how they were allocated (leaving holes), the heap would get into an inconsistent state, eventually resulting in crashes.
free_memory() relies on having allocations in order, but allocate_memory() did not guarantee that: It reused the first allocation with a NULL ptr without ensuring that it was between low_address and high_address. When it belongs to a hole in the allocated memory, such an allocation is not really free for reuse, because free_memory() still needs its length.
Instead, explicitly mark allocations available for reuse with a special (invalid) value in the length field. Only allocations that lie between low_address and high_address are marked that way.
I have a function where it should be impossible to reach the end, so I put in a safe-mode reset at the bottom:
```
int find_unused_slot(void) {
// precondition: you already verified that a slot was available
for (int i=0; i<NUM_SLOTS; i++) {
if( slot_free(i)) {
return i;
}
}
safe_mode_reset(MICROPY_FATAL_ERROR);
}
```
However, the compiler still gave a diagnostic, because safe_mode_reset was not declared NORETURN.
So I started by teaching the compiler that reset_into_safe_mode never returned. This leads at least one level deeper due to reset_cpu needing to be a NORETURN function. Each port is a little different in this area. I also marked reset_to_bootloader as NORETURN.
Additional notes:
* stm32's reset_to_bootloader was not implemented, but now does a bare reset. Most stm32s are not fitted with uf2 bootloaders anyway.
* ditto cxd56
* esp32s2 did not implement reset_cpu at all. I used esp_restart(). (not tested)
* litex did not implement reset_cpu at all. I used reboot_ctrl_write. But notably this is what reset_to_bootloader already did, so one or the other must be incorrect (not tested). reboot_ctrl_write cannot be declared NORETURN, as it returns unless the special value 0xac is written), so a new unreachable forever-loop is added.
* cxd56's reset is via a boardctl() call which can't generically be declared NORETURN, so a new unreacahble "for(;;)" forever-loop is added.
* In several places, NVIC_SystemReset is redeclared with NORETURN applied. This is accepted just fine by gcc. I chose this as preferable to editing the multiple copies of CMSIS headers where it is normally declared.
* the stub safe_mode reset simply aborts. This is used in mpy-cross.
These changes remove the caveat from supervisor.runtime.serial_connected.
It appears that _tud_cdc_connected() only tracks explicit changes to the
"DTR" bit, which leads to disconnects not being registered.
Instead:
* when line state is changed explicitly, track the dtr value in
_serial_connected
* when the USB bus is suspended, set _serial_connected to False
Testing performed (using sam e54 xplained): Run a program to show
the state of `serial_connected` on the LED:
```
import digitalio
import supervisor
import board
led = digitalio.DigitalInOut(board.LED)
while True:
led.switch_to_output(not supervisor.runtime.serial_connected)
```
Try all the following:
* open, close serial terminal program
- LED status tracks whether terminal is open
* turn on/off data lines using the switchable charge-only cable
- LED turns off when switch is in "charger" position
- LED turns back on when switch is in Data position and terminal is
opened (but doesn't turn back on just because switch position is
changed)
Massive savings. Thanks so much @ciscorn for providing the initial
code for choosing the dictionary.
This adds a bit of time to the build, both to find the dictionary
but also because (for reasons I don't fully understand), the binary
search in the compress() function no longer worked and had to be
replaced with a linear search.
I think this is because the intended invariant is that for codebook
entries that encode to the same number of bits, the entries are ordered
in ascending value. However, I mis-placed the transition from "words"
to "byte/char values" so the codebook entries for words are in word-order
rather than their code order.
Because this price is only paid at build time, I didn't care to determine
exactly where the correct fix was.
I also commented out a line to produce the "estimated total memory size"
-- at least on the unix build with TRANSLATION=ja, this led to a build
time KeyError trying to compute the codebook size for all the strings.
I think this occurs because some single unicode code point ('ァ') is
no longer present as itself in the compressed strings, due to always
being replaced by a word.
As promised, this seems to save hundreds of bytes in the German translation
on the trinket m0.
Testing performed:
- built trinket_m0 in several languages
- built and ran unix port in several languages (en, de_DE, ja) and ran
simple error-producing codes like ./micropython -c '1/0'
Two problems: The lead byte for 3-byte sequences was wrong, and one
mid-byte was not even filled in due to a missing "++"!
Apparently this was broken ever since the first "Compress as unicode,
not bytes" commit, but I believed I'd "tested" it by running on the
Pinyin translation.
This rendered at least the Korean and Japanese translations completely
illegible, affecting 5.0 and all later releases.
Compress common unicode bigrams by making code points in the range
0x80 - 0xbf (inclusive) represent them. Then, they can be greedily
encoded and the substituted code points handled by the existing Huffman
compression. Normally code points in the range 0x80-0xbf are not used
in Unicode, so we stake our own claim. Using the more arguably correct
"Private Use Area" (PUA) would mean that for scripts that only use
code points under 256 we would use more memory for the "values" table.
bigram means "two letters", and is also sometimes called a "digram".
It's nothing to do with "big RAM". For our purposes, a bigram represents
two successive unicode code points, so for instance in our build on
trinket m0 for english the most frequent are:
['t ', 'e ', 'in', 'd ', ...].
The bigrams are selected based on frequency in the corpus, but the
selection is not necessarily optimal, for these reasons I can think of:
* Suppose the corpus was just "tea" repeated 100 times. The
top bigrams would be "te", and "ea". However,
overlap, "te" could never be used. Thus, some bigrams might actually
waste space
* I _assume_ this has to be why e.g., bigram 0x86 "s " is more
frequent than bigram 0x85 " a" in English for Trinket M0, because
sequences like "can't add" would get the "t " digram and then
be unable to use the " a" digram.
* And generally, if a bigram is frequent then so are its constituents.
Say that "i" and "n" both encode to just 5 or 6 bits, then the huffman
code for "in" had better compress to 10 or fewer bits or it's a net
loss!
* I checked though! "i" is 5 bits, "n" is 6 bits (lucky guess)
but the bigram 0x83 also just 6 bits, so this one is a win of
5 bits for every "it" minus overhead. Yay, this round goes to team
compression.
* On the other hand, the least frequent bigram 0x9d " n" is 10 bits
long and its constituent code points are 4+6 bits so there's no
savings, but there is the cost of the table entry.
* and somehow 0x9f 'an' is never used at all!
With or without accounting for overlaps, there is some optimum number
of bigrams. Adding one more bigram uses at least 2 bytes (for the
entry in the bigram table; 4 bytes if code points >255 are in the
source text) and also needs a slot in the Huffman dictionary, so
adding bigrams beyond the optimim number makes compression worse again.
If it's an improvement, the fact that it's not guaranteed optimal
doesn't seem to matter too much. It just leaves a little more fruit
for the next sweep to pick up. Perhaps try adding the most frequent
bigram not yet present, until it doesn't improve compression overall.
Right now, de_DE is again the "fullest" build on trinket_m0. (It's
reclaimed that spot from the ja translation somehow) This change saves
104 bytes there, increasing free space about 6.8%. In the larger
(but not critically full) pyportal build it saves 324 bytes.
The specific number of bigrams used (32) was chosen as it is the max
number that fit within the 0x80..0xbf range. Larger tables would
require the use of 16 bit code points in the de_DE build, losing savings
overall.
(Side note: The most frequent letters in English have been said
to be: ETA OIN SHRDLU; but we have UAC EIL MOPRST in our corpus)
Otherwise, out of range writes would occur in tilegrid_set_tile, causing a safe mode reset.
```
Hardware watchpoint 6: -location *stack_alloc->ptr
Old value = 24652061
New value = 24641565
0x000444f2 in common_hal_displayio_tilegrid_set_tile (self=0x200002c8 <supervisor_terminal_text_grid>, x=1, y=1, tile_index=0 '\000')
at ../../shared-module/displayio/TileGrid.c:236
236 if (!self->partial_change) {
(gdb)
```
The font is missing many characters and the build needs the space.
We can optimize font storage when we get a good font.
The serial output will work as usual.
Before this, a background callback that was on the list when
background_callback_reset was called could have ended up in a state
that made it "un-queueable": its "prev" pointer could have been non-NULL.
A background callback must never outlive its related object. By
collecting the head of the linked list of background tasks, this will
not happen.
One hypothetical case where this could happen is if an MP3Decoder is
deleted while its callback to fill its buffer is scheduled.
CALLBACK_CRITICAL_BEGIN is heavyweight, but we can be confident we do
not have work to do as long as callback_head is NULL.
This gives back performance on nRF.
In time, we should transition interrupt driven background tasks out of the
overall run_background_tasks into distinct background callbacks,
so that the number of checks that occur with each tick is reduced.
Few peripherals are actually tested. However, USB, I2C and GPIO seem to work.
Most pins are silkscreened with the "PX00" style, so the board module
only includes the small number that are screened differently.
The default SPI, I2C, and UART are the ones on the EXT2 header. This is
arbitrary, but the I2C on this connector is shared with the on-board I2C
devices and the PCC header, making it the most versatile.
Length was stored as a 16-bit number always. Most translations have
a max length far less. For example, US English translation lengths
always fit in just 8 bits. probably all languages fit in 9 bits.
This also has the side effect of reducing the alignment of
compressed_string_t from 2 bytes to 1.
testing performed: ran in german and english on pyruler, printed messages
looked right.
Firmware size, en_US
Before: 3044 bytes free in flash
After: 3408 bytes free in flash
Firmware size, de_DE (with #2967 merged to restore translations)
Before: 1236 bytes free in flash
After: 1600 bytes free in flash
Check to see if the current exception is a Watchdog exception, if it's
enabled. This ensures we break out of the current sleep() if a watchdog
timeout hits.
Signed-off-by: Sean Cross <sean@xobs.io>
Allow for passing `-DCFG_TUSB_DEBUG=1` or `-DCFG_TUSB_DEBUG=2` on the
command line to enable debugging tinyusb within circuitpython.
Signed-off-by: Sean Cross <sean@xobs.io>
Add a field to allow specifying a timeout when initiating advertising.
As part of this, add a new property to determine if the device is still
advertising.
Additionally, have the `anonymous` property require a timeout, and set
the timeout to the maximum possible value if no timeout is specified.
Signed-off-by: Sean Cross <sean@xobs.io>
Add a new parameter to the `start_advertising()` function to enable
anonymous advertising. This forces a call to `sd_ble_gap_privacy_set()`
with `privacy_mode` set to `BLE_GAP_PRIVACY_MODE_DEVICE_PRIVACY` and
`private_addr_type` set to
`BLE_GAP_ADDR_TYPE_RANDOM_PRIVATE_RESOLVABLE`.
With this, addresses will cycle at a predefined rate (currently once
every 15 minutes).
Signed-off-by: Sean Cross <sean@xobs.io>
This gets all the purely internal references. Some uses of
protomatter/Protomatter/PROTOMATTER remain, as they are references
to symbols in the Protomatter C library itself.
- bump supervisor alloc count by 4 (we actually use 5)
- move reconstruct to after gc heap is reset
- destroy protomatter object entirely if not used by a FramebufferDisplay
- ensure previous supervisor allocations are released
- zero out pointers so GC can collect them
testing performed:
* successfully store and retrieve a 500kB file on the flash
* square wave output on each pin appears on o'scope
* board.SPI(), board.SERIAL(), board.I2C() all construct
Introduces a way to place CircuitPython code and data into
tightly coupled memory (TCM) which is accessible by the CPU in a
single cycle. It also frees up room in the corresponding cache for
intermittent data. Loading from external flash is slow!
The data cache is also now enabled.
Adds support for the iMX RT 1021 chip. Adds three new boards:
* iMX RT 1020 EVK
* iMX RT 1060 EVK
* Teensy 4.0
Related to #2492, #2472 and #2477. Fixes#2475.
It's extremely dubious that we have these handles that we think
are to GC'd memory at a time when the gc pool may not be initialized.
Hopefully, they WERE valid GC memory and are undisturbed by the teardown
of the interpreter that can lead to this state.
In this case, don't try to m_free them, the memory will become free when
the GC heap is reinitialized.
Closes: #2338 (together with previous commit)
By treating each unicode code-point as a single entity for huffman
compression, the overall compression rate can be somewhat improved
without changing the algorithm. On the decompression side, when
compressed values above 127 are encountered, they need to be
converted from a 16-bit Unicode code point into a UTF-8 byte
sequence.
Doing this returns approximately 1.5kB of flash storage with the
zh_Latn_pinyin translation. (292 -> 1768 bytes remaining in my build
of trinket_m0)
Other "more ASCII" translations benefit less, and in fact
zh_Latn_pinyin is no longer the most constrained translation!
(de_DE 1156 -> 1384 bytes free in flash, I didn't check others
before pushing for CI)
English is slightly pessimized, 2840 -> 2788 bytes, probably mostly
because the "values" array was changed from uint8_t to uint16_t,
which is strictly not required for an all-ASCII translation. This
could probably be avoided in this case, but as English is not the
most constrained translation it doesn't really matter.
Testing performed: built for feather nRF52840 express and trinket m0
in English and zh_Latn_pinyin; ran and verified the localized
messages such as
Àn xià rènhé jiàn jìnrù REPL. Shǐyòng CTRL-D chóngxīn jiāzài.
and
Press any key to enter the REPL. Use CTRL-D to reload.
were properly displayed.
This improves performance of running python code by 34%, based
on the "pystone" benchmark on metro m4 express at 5000 passes
(1127.65 -> 1521.6 passes/second).
In addition, by instrumenting the tick function and monitoring on an
oscilloscope, the time actually spent in run_background_tasks() on
the metro m4 decreases from average 43% to 0.5%. (however, there's
some additional overhead that is moved around and not accounted for
in that "0.5%" figure, each time supervisor_run_background_tasks_if_tick
is called but no tick has occurred)
On the CPB, it increases pystone from 633 to 769, a smaller percentage
increase of 21%. I did not measure the time actually spent in
run_background_tasks() on CPB.
Testing performed: on metro m4 and cpb, run pystone adapted from python3.4
(change time.time to time.monotonic for sub-second resolution)
Besides running a 5000 pass test, I also ran a 50-pass test while
scoping how long an output pin was set. Average: 34.59ms or 1445/s on m4,
67.61ms or 739/s on cbp, both matching the other pystone result reasonably
well.
import pystone
import board
import digitalio
import time
d = digitalio.DigitalInOut(board.D13)
d.direction = digitalio.Direction.OUTPUT
while True:
d.value = 0
time.sleep(.01)
d.value = 1
pystone.main(50)
This code is shared by most parts, except where not all the #ifdefs
inside the tick function were present in all ports. This mostly would
have broken gamepad tick support on non-samd ports.
The "ms32" and "ms64" variants of the tick functions are introduced
because there is no 64-bit atomic read. Disabling interrupts avoids
a low probability bug where milliseconds could be off by ~49.5 days
once every ~49.5 days (2^32 ms).
Avoiding disabling interrupts when only the low 32 bits are needed is a minor
optimization.
Testing performed: on metro m4 express, USB still works and
time.monotonic_ns() still counts up
This PR refines the _bleio API. It was originally motivated by
the addition of a new CircuitPython service that enables reading
and modifying files on the device. Moving the BLE lifecycle outside
of the VM motivated a number of changes to remove heap allocations
in some APIs.
It also motivated unifying connection initiation to the Adapter class
rather than the Central and Peripheral classes which have been removed.
Adapter now handles the GAP portion of BLE including advertising, which
has moved but is largely unchanged, and scanning, which has been enhanced
to return an iterator of filtered results.
Once a connection is created (either by us (aka Central) or a remote
device (aka Peripheral)) it is represented by a new Connection class.
This class knows the current connection state and can discover and
instantiate remote Services along with their Characteristics and
Descriptors.
Relates to #586
The GitHub Actions build for this PR is failing for reasons that make no sense. Make a tiny commit change to see if it will refresh things and work again.
This also improves Palette so it stores the original RGB888 colors.
Lastly, it adds I2CDisplay as a display bus to talk over I2C. Particularly
useful for the SSD1306.
Fixes#1828. Fixes#1956
This fixes the bug that bitmap changes do not cause screen updates
and optimizes the refresh when the bitmap is simply shown on the
screen. If the bitmap is used in tiles, then changing it will
cause all TileGrids using it to do a full refresh.
Fixes#1981
Different operations to the display tree have different costs. Be
aware of these costs when optimizing your code.
* Changing tiles indices in a TileGrid will update an area
covering them all.
* Changing a palette will refresh every object that references it.
* Moving a TileGrid will update both where it was and where it moved to.
* Adding something to a Group will refresh each individual area it
covers.
* Removing things from a Group will refresh one area that covers all
previous locations. (Not separate areas like add.)
* Setting a new top level Group will refresh the entire display.
Only TileGrid moves are optimized for overlap. All other overlaps
cause sending of duplicate pixels.
This also adds flip_x, flip_y and transpose_xy to TileGrid. They
change the direction of the pixels but not the location.
Fixes#1169. Fixes#1705. Fixes#1923.
This changes the displayio pixel computation from per-pixel to
per-area. This is precursor work to updating portions of the screen
(#1169). It should provide mild speedups because bounds checks are
done once per area rather than once per pixel. Filling by area also
allows TileGrid to maintain a row-associative fill pattern even when
the display's refresh is orthogonal to it.
If one of the default pins was already in use it would crash.
The internal API has been refined to allow us to get the value
without causing an init of the singleton.
Fixes#1753
Without this, a double free can occur when a display (and terminal)
is released and then a crash occurs. Upon a second release,
different memory is released (sometimes the heap). When this is
followed by an allocation for the flash cache, the cache can
overwrite the active heap causing crashes.
Fixes#1667
This changes a number of things in displayio:
* Introduces BuiltinFont and Glyph so the built in font can be used by libraries. For boards with
a font it is available as board.TERMINAL_FONT. Fixes#1172
* Remove _load_row from Bitmap in favor of bitmap[] access. Index can be x/y tuple or overall index. Fixes#1191
* Add width and height properties to Bitmap.
* Add insert and [] access to Group. Fixes#1518
* Add index param to pop on Group.
* Terminal no longer takes unicode character info. It takes a BuiltinFont instead.
* Fix Terminal's handling of [###D vt100 commands used when up arrowing into repl history.
* Add x and y positions to Group plus scale as well.
* Add bitmap accessor for BuiltinFont
* Fix Hallowing.
* Fix builds without displayio.
* Fix y bounds that appears as untrollable row of pixels.
* Add scrolling to TileGrid.
* Remove Sprite to save space. TileGrid is a drop in replacement.
This creates a common safe mode mechanic that ports can share.
As a result, the nRF52 now has safe mode support as well.
The common safe mode adds a 700ms delay at startup where a reset
during that window will cause a reset into safe mode. This window
is designated by a yellow status pixel and flashing the single led
three times.
A couple NeoPixel fixes are included for the nRF52 as well.
Fixes#1034. Fixes#990. Fixes#615.
The backtrace cannot be given because it relies on the validity
of the qstr data structures on the heap which may have been
corrupted.
In fact, it still can crash hard when the bytecode itself is
overwritten. To fix, we'd need a way to skip gathering the
backtrace completely.
This also increases the default stack size on M4s so it can
accomodate the stack needed by ASF4s nvm API.
We were writing with quad page program including the address (0x38)
which is unsupported by the GD25Q16C but it is supported by the
flash on the DK. So, we use the single address, quad data command
(0x32).
This started while adding USB MIDI support (and descriptor support is
in this change.) When seeing that I'd have to implement the MIDI class
logic twice, once for atmel-samd and once for nrf, I decided to refactor
the USB stack so its shared across ports. This has led to a number of
changes that remove items from the ports folder and move them into
supervisor.
Furthermore, we had external SPI flash support for nrf pending so I
factored out the connection between the usb stack and the flash API as
well. This PR also includes the QSPI support for nRF.
This saves code space in builds which use link-time optimization.
The optimization drops the untranslated strings and replaces them
with a compressed_string_t struct. It can then be decompressed to
a c string.
Builds without LTO work as well but include both untranslated
strings and compressed strings.
This work could be expanded to include QSTRs and loaded strings if
a compress method is added to C. Its tracked in #531.
Its slimmed down by removing the qstr and bit packing TCC info.
The trinket m0 build actually grows by 20 bytes. The arduino zero
build shrinks by 188 bytes.
* atmel-samd: Remove ASF3. This will break builds.
* atmel-samd: Add ASF4 for the SAMD21 and SAMD51.
* Introduce the supervisor concept to facilitate porting.
The supervisor is the code which runs individual MicroPython VMs. By
splitting it out we make it more consistent and easier to find.
This also adds very basic SAMD21 and SAMD51 support using the
supervisor. Only the REPL currently works.
This begins the work for #178.
