If kw_args is NULL then memcpy() gets a NULL source argument.
This is undefined behavior under the C standard, even if 0 bytes
are being copied.
This problem was found using clang 7's scan-build static analyzer.
If a native displayio object is accessed before it's super().__init__()
has been called, then a placeholder is given that will cause a crash if
accessed. This is tricky to get right so we detect this case and raise
a NotInplementedError instead of crashing.
Fixes#1881
* Update pybadge pins and flash for rev D
* TileGrid now validates the type of the pixel_shader.
* Display actually handles incoming subclass objects.
* MicroPython will inspect native parents to see if special
accessors are used.
This changes a number of things in displayio:
* Introduces BuiltinFont and Glyph so the built in font can be used by libraries. For boards with
a font it is available as board.TERMINAL_FONT. Fixes#1172
* Remove _load_row from Bitmap in favor of bitmap[] access. Index can be x/y tuple or overall index. Fixes#1191
* Add width and height properties to Bitmap.
* Add insert and [] access to Group. Fixes#1518
* Add index param to pop on Group.
* Terminal no longer takes unicode character info. It takes a BuiltinFont instead.
* Fix Terminal's handling of [###D vt100 commands used when up arrowing into repl history.
* Add x and y positions to Group plus scale as well.
* Add bitmap accessor for BuiltinFont
This patch is a code optimisation, trading text bytes for speed. On
pyboard it's an increase of 0.06% in code size for a gain (in pystone
performance) of roughly 6.5%.
The patch optimises load/store/delete of attributes in user defined classes
by not looking up special accessors (@property, __get__, __delete__,
__set__, __setattr__ and __getattr_) if they are guaranteed not to exist in
the class.
Currently, if you do my_obj.foo() then the runtime has to do a few checks
to see if foo is a property or has __get__, and if so delegate the call.
And for stores things like my_obj.foo = 1 has to first check if foo is a
property or has __set__ defined on it.
Doing all those checks each and every time the attribute is accessed has a
performance penalty. This patch eliminates all those checks for cases when
it's guaranteed that the checks will always fail, ie no attributes are
properties nor have any special accessor methods defined on them.
To make this guarantee it checks all attributes of a user-defined class
when it is first created. If any of the attributes of the user class are
properties or have special accessors, or any of the base classes of the
user class have them, then it sets a flag in the class to indicate that
special accessors must be checked for. Then in the load/store/delete code
it checks this flag to see if it can take the shortcut and optimise the
lookup.
It's an optimisation that's pretty widely applicable because it improves
lookup performance for all methods of user defined classes, and stores of
attributes, at least for those that don't have special accessors. And, it
allows to enable descriptors with minimal additional runtime overhead if
they are not used for a particular user class.
There is one restriction on dynamic class creation that has been introduced
by this patch: a user-defined class cannot go from zero special accessors
to one special accessor (or more) after that class has been subclassed. If
the script attempts this an AttributeError is raised (see addition to
tests/misc/non_compliant.py for an example of this case).
The cost in code space bytes for the optimisation in this patch is:
unix x64: +528
unix nanbox: +508
stm32: +192
cc3200: +200
esp8266: +332
esp32: +244
Performance tests that were done:
- on unix x86-64, pystone improved by about 5%
- on pyboard, pystone improved by about 6.5%, from 1683 up to 1794
- on pyboard, bm_chaos (from CPython benchmark suite) improved by about 5%
- on esp32, pystone improved by about 30% (but there are caching effects)
- on esp32, bm_chaos improved by about 11%
micropython: ../../py/objtype.c:1100: super_attr: Assertion `MP_OBJ_IS_TYPE(self->type, &mp_type_type)' failed.
e.g., when making calls like
super(1, 1).x
Fixes the following assertion failures when the arguments to type()
were not of valid types:
micropython: ../../py/objtype.c:984: mp_obj_new_type: Assertion `MP_OBJ_IS_TYPE(bases_tuple, &mp_type_tuple)' failed.
micropython: ../../py/objtype.c:994: mp_obj_new_type: Assertion `MP_OBJ_IS_TYPE(items[i], &mp_type_type)' failed.
e.g., when making calls like
type("", (), 3)
type("", 3, {})
Note that the check for elem!=NULL is removed for the
MP_MAP_LOOKUP_ADD_IF_NOT_FOUND case because mp_map_lookup will always
return non-NULL for such a case.
This adapts the allocation process to start from either end of the heap
when searching for free space. The default behavior is identical to the
existing behavior where it starts with the lowest block and looks higher.
Now it can also look from the highest block and lower depending on the
long_lived parameter to gc_alloc. As the heap fills, the two sections may
overlap. When they overlap, a collect may be triggered in order to keep
the long lived section compact. However, free space is always eligable
for each type of allocation.
By starting from either of the end of the heap we have ability to separate
short lived objects from long lived ones. This separation reduces heap
fragmentation because long lived objects are easy to densely pack.
Most objects are short lived initially but may be made long lived when
they are referenced by a type or module. This involves copying the
memory and then letting the collect phase free the old portion.
QSTR pools and chunks are always long lived because they are never freed.
The reallocation, collection and free processes are largely unchanged. They
simply also maintain an index to the highest free block as well as the lowest.
These indices are used to speed up the allocation search until the next collect.
In practice, this change may slightly slow down import statements with the
benefit that memory is much less fragmented afterwards. For example, a test
import into a 20k heap that leaves ~6k free previously had the largest
continuous free space of ~400 bytes. After this change, the largest continuous
free space is over 3400 bytes.
Before this patch, if a user defined the __new__() function for a class
then two instances of that class would be created: once before __new__ is
called and once during the __new__ call (assuming the user creates some
instance, eg using super().__new__, which is most of the time). The first
one was then discarded. This refactor makes it so that a new instance is
only created if the user __new__ function doesn't exist.
This patch cleans up and generalises part of the code which handles
overriding and calling a native base-class's __init__ method. It defers
the call to the native make_new() function until after the user (Python)
__init__() method has run. That user method now has the chance to call the
native __init__/make_new and pass it different arguments. If the user
doesn't call the super().__init__ method then it will be called
automatically after the user code finishes, to finalise construction of the
instance.
Before this patch MP_BINARY_OP_IN had two meanings: coming from bytecode it
meant that the args needed to be swapped, but coming from within the
runtime meant that the args were already in the correct order. This lead
to some confusion in the code and comments stating how args were reversed.
It also lead to 2 bugs: 1) containment for a subclass of a native type
didn't work; 2) the expression "{True} in True" would illegally succeed and
return True. In both of these cases it was because the args to
MP_BINARY_OP_IN ended up being reversed twice.
To fix these things this patch introduces MP_BINARY_OP_CONTAINS which
corresponds exactly to the __contains__ special method, and this is the
operator that built-in types should implement. MP_BINARY_OP_IN is now only
emitted by the compiler and is converted to MP_BINARY_OP_CONTAINS by
swapping the arguments.
This patch introduces a new compile-time config option to disable multiple
inheritance at the Python level: MICROPY_MULTIPLE_INHERITANCE. It is
enabled by default.
Disabling multiple inheritance eliminates a lot of recursion in the call
graph (which is important for some embedded systems), and can be used to
reduce code size for ports that are really constrained (by around 200 bytes
for Thumb2 archs).
With multiple inheritance disabled all tests in the test-suite pass except
those that explicitly test for multiple inheritance.
Introduction of ports subdirectory where all ports are moved to
The main change in this release is the introduction of a "ports/"
subdirectory at the top-level of the repository, and all of the ports are
moved here. In the process the "stmhal" port is renamed to "stm32" to
better reflect the MCU that it targets. In addition, the STM32 CMSIS and
HAL sources are moved to a new submodule called "stm32lib".
The bytecode has changed in this release, compared to the previous release,
and as a consequence the .mpy version number has increased to version 3.
This means that scripts compiled with the previous mpy-cross must be
recompiled to work with this new version.
There have also been various enhancements and optimisations, such as:
check for valid UTF-8 when creating str objects, support for reverse
special binary operations like __radd__, full domain checking in the math
module, support for floor-division and modulo in the viper emitter,
and addition of stack overflow checking when executing a regex.
The stm32 port sees improved support for F7 MCUs, addition of a new board
B_L475E_IOT01A based on the STM32L475, and support for the Wiznet W5500
chipset along with improved socket behaviour.
A detailed list of changes follows.
py core:
- objstr: startswith, endswith: check arg to be a string
- nlrx86,x64: replace #define of defined() with portable macro usage
- objtype: handle NotImplemented return from binary special methods
- objtype: mp_obj_class_lookup: improve debug logging
- map: remove unused new/free functions
- make m_malloc_fail() have void return type, since it doesn't return
- modstruct: in struct.pack, stop converting if there are no args left
- modstruct: check and prevent buffer-read overflow in struct unpacking
- modstruct: check and prevent buffer-write overflow in struct packing
- nlrthumb: get working again on standard Thumb arch (ie not Thumb2)
- objfloat: fix binary ops with incompatible objects
- obj: fix comparison of float/complex NaN with itself
- objtype: implement fallback for instance inplace special methods
- objtuple: properly implement comparison with incompatible types
- objstr: add check for valid UTF-8 when making a str from bytes
- objlist: properly implement comparison with incompatible types
- runtime0.h: move relational ops to the beginning of mp_binary_op_t
- runtime0.h: move MP_BINARY_OP_DIVMOD to the end of mp_binary_op_t
- objtype: make sure mp_binary_op_method_name has full size again
- runtime0.h: put inplace arith ops in front of normal operations
- builtinhelp: simplify code slightly by extracting object type
- runtime: implement dispatch for "reverse op" special methods
- nlrx86: fix building for Android/x86
- builtinhelp: change signature of help text var from pointer to array
- runtime.h: change empty mp_warning macro so var-args are non empty
- modbuiltins: implement abs() by dispatching to MP_UNARY_OP_ABS
- {objfloat,objcomplex}: optimise MP_UNARY_OP_ABS by reusing variables
- mpconfig.h: add note that using computed gotos in VM is not C99
- objstr: strip: don't strip "\0" by default
- objexcept: prevent infinite recursion when allocating exceptions
- stream: remove unnecessary checks for NULL return from vstr_add_len
- vstr: raise a RuntimeError if fixed vstr buffer overflows
- vm: use lowercase letter at start of exception message
- persistentcode: define mp_raw_code_save_file() for any unix target
- add config option to print warnings/errors to stderr
- objfloat: support raising a negative number to a fractional power
- objset: simplify set and frozenset by separating their locals dicts
- objset: check that RHS of a binary op is a set/frozenset
- objset: include the failed key in a KeyError raised from set.remove
- objtype: change type of enum-to-qstr table to uint16_t to save space
- objstr: make empty bytes object have a null-terminating byte
- mpprint: only check for null string printing when NDEBUG not defined
- objtype: clean up unary- and binary-op enum-to-qstr mapping tables
- persistentcode: bump .mpy version number to version 3
- bc: update opcode_format_table to match the bytecode
- modmath: add full checks for math domain errors
- modmath: convert log2 macro into a function
- formatfloat: don't print the negative sign of a NaN value
- formatfloat: use standard isinf, isnan funcs instead of custom ones
- modbuiltins: use existing utf8_get_char helper in builtin ord func
- emitnative: implement floor-division and modulo for viper emitter
- objtype: use CPython compatible method name for sizeof
- objtype: fit qstrs for special methods in byte type
- objtype: define all special methods if requested
- objtype: introduce MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS
extmod:
- modubinascii: only include uzlib/tinf.h when it's really needed
- modussl_mbedtls: allow to compile with MBEDTLS_DEBUG_C disabled
- machine_pinbase: put PinBase singleton in ROM
- re1.5: upgrade to v0.8.2, adds hook for stack overflow checking
- modure: add stack overflow checking when executing a regex
- uos_dupterm: update uos.dupterm() and helper funcs to have index
- uos_dupterm: swallow any errors from dupterm closing the stream
- vfs: replace VLA in proxy func with small, static sized array
- modussl: add finaliser support for ussl objects
- modussl_mbedtls: allow to compile with unix coverage build
lib:
- add new submodule, stm32lib containing STM32 CMSIS and HAL source
- embed/abort_: use mp_raise_msg helper function
- libm: fix tanhf so that it correctly handles +/- infinity args
- libm: remove implementation of log2f, use MP_NEED_LOG2 instead
- axtls: update, support for SSL_EAGAIN return code
- berkeley-db-1.xx: update, allow to override MINCACHE, DEFPSIZE
drivers:
- memory/spiflash: change from hard-coded soft SPI to generic SPI
- display/ssd1306.py: improve performance of graphics methods
- nrf24l01: make nRF24L01 test script more portable
- display/ssd1306: implement SSD1306_I2C poweron method
- display/ssd1306: make poweron() work the same with SSD1306_SPI
- wiznet5k: improve the performance of socket ops with threading
- wiznet5k: get low-level W5500 driver working
tools:
- upip: upgrade to 1.2.2
- pyboard: use repr() when quoting data in error messages
- pyboard: update docstring for additional device support
tests:
- object_new: better messages, check user __new__() method
- class_new: add checks for __init__ being called and other improvements
- class_new: add another testcase for __new__/__init__ interaction
- class_inplace_op: test for inplace op fallback to normal one
- run-bench-tests: update locations of executables, now in ports/
- class_reverse_op: test for reverse arith ops special methods
- run-tests: skip class_inplace_op for minimal profile
- run-tests: fix copy-paste mistake in var name
- cpydiff: add cases for locals() discrepancies
- extmod: add test for ure regexes leading to infinite recursion
- extmod: add test for '-' in character class in regex
- run-tests: close device under test using "finally"
- net_inet: update tls test to work with CPython and incl new site
unix port:
- rename modsocket.c to modusocket.c
- modusocket: remove #if MICROPY_SOCKET_EXTRA code blocks
- enable MICROPY_PY_REVERSE_SPECIAL_METHODS
stm32 port:
- modmachine: make machine.bootloader() work when MPU is enabled
- modmachine: improve support for sleep/deepsleep on F7 MCUs
- compute PLL freq table during build instead of at run time
- modmachine: for F7 MCU, save power by reducing internal volt reg
- boards/pllvalues.py: make script work with both Python 2 and 3
- Makefile: use lib/stm32lib instead of local cmsis and hal files
- remove cmsis and hal files, they are now a submodule
- Makefile: automatically fetch stm32lib submodule if needed
- update to new STM Cube HAL library
- fix clock initialisation of L4 MCUs
- rename stmhal port directory to stm32
- remove unused usbd_msc.c file
- boards: change remaining stm32f4xx_hal_conf.h to unix line ending
- boards: change linker scripts to use "K" instead of hex byte size
- boards: fix I2C1 pin mapping on NUCLEO_F401RE/F411RE boards
- i2c: when scanning for I2C devices only do 1 probe per address
- modnwwiznet5k: release the GIL on blocking network operations
- boards: add new board B_L475E_IOT01A based on STM32L475
- make-stmconst.py: make sure mpz const data lives in ROM
- timer: make pyb.Timer() instances persistent
- mpconfigport.h: add configuration for max periphs on L4 series
- usbdev: make the USBD callback struct const so it can go in ROM
- usbdev: change static function variable to non-static
- usbdev: put all CDC state in a struct
- usbdev: put all HID state in a struct
- usbdev: simplify CDC tx/rx buffer passing
- usbdev: simplify HID tx/rx buffer passing
- usbdev/core: add state parameter to all callback functions
- usbdev: put all state for the USB device driver in a struct
- usbdev: simplify pointers to MSC state and block dev operations
- usbdev: merge all global USB device state into a single struct
- usbdev: make device descriptor callbacks take a state pointer
- usbdev: move all the USB device descriptor state into its struct
- timer: enable ARPE so that timer freq can be changed smoothly
- modnwwiznet5k: get the IP address of an established socket
- boards: fix typos in stm32f767_af.csv table
- usbd_cdc_interface: don't reset CDC output buf on initialisation
- modnwwiznet5k: implement WIZNET5K.isconnected() method
- modusocket: make getaddrinfo() work when passed an IP address
- modusocket: return OSError(-2) if getaddrinfo fails
- mpconfigport.h: add MICROPY_THREAD_YIELD() macro
- modnwwiznet5k: add support for W5500 Ethernet chip
- modnwwiznet5k: increase SPI bus speed to 42MHz
- modnwwiznet5k: implement stream ioctl for the Wiznet driver
- mphalport: improve efficiency of mp_hal_stdout_tx_strn_cooked
- make uos.dupterm() conform to specs by using extmod version
cc3200 port:
- enable micropython.kbd_intr() method
- use standard implementation of keyboard interrupt
esp8266 port:
- rename axtls_helpers.c to posix_helpers.c
- posix_helpers: set ENOMEM on memory alloc failure
- set DEFPSIZE=1024, MINCACHE=3 for "btree" module
- esp_mphal: send data in chunks to mp_uos_dupterm_tx_strn
- modnetwork: add "bssid" keyword arg to WLAN.connect() method
- modules/webrepl_setup: add info about allowed password length
zephyr port:
- Makefile: revamp "test" target after ports were moved to ports/
- use CONFIG_NET_APP_SETTINGS to setup initial network addresses
- switch to interrupt-driven pull-style console
pic16bit port:
- add definition of SEEK_SET to unistd.h
docs:
- pyboard/tutorial: add "timeout=0" to UART in pass-through example
- more xrefs to "MicroPython port" in glossary
- library/network: fix ref to "socket" module (should be "usocket")
- machine.Signal: improve style/grammar and add usage example
- library: add description of "index" parameter to uos.dupterm()
- library/micropython: fix typo in RST formatting
- library/framebuf.rst: generalise constructor to all colour formats
- btree: describe page caching policy of the underlying implementation
- esp8266/tutorial: update neopixel with example of using 4 bbp
- library/network: clarify usage of "bssid" arg in connect() method
- pyboard/quickref: add info for Switch, RTC, CAN, Accel classes
- pyboard/tutorial: update now that yellow LED also supports PWM
- esp8266/quickref: add quickref info for RTC class
- library: add missing cross-ref links for classes in pyb module
- library/network: update docs to state that W5500 is supported
- uselect: document one-shot polling mode
- usocket: elaborate descriptions
- usocket: document inet_ntop(), inet_pton()
- library/network: add dhcp_hostname parameter
- reference/isr_rules: minor typo correction
- ussl: fix module name refs and use "MicroPython port" term
- esp8266/general: add section on TLS limitations
- usocket: document that settimeout() isn't supported by all ports
- ure: add "|" (alternative) to the list of supported operators
- reference/isr_rules.rst: add tutorial on use of micropython.schedule()
travis:
- use --upgrade when pip is installing cpp-coveralls
- update build command now that stm32 Wiznet config has changed
examples:
- hwconfig_console: add .on()/.off() methods
all:
- convert mp_uint_t to mp_unary_op_t/mp_binary_op_t where appropriate
- convert remaining "mp_uint_t n_args" to "size_t n_args"
- make new ports/ sub-directory and move all ports there
- update Makefiles and others to build with new ports/ dir layout
- remove inclusion of internal py header files
- use NULL instead of "" when calling mp_raise exception helpers
README:
- update "Dependencies" section
- add explicit section on contributing
- add gcc and arm-none-eabi-newlib to list of required components
.gitattributes:
- remove obsolete entries for stmhal/hal, stmhal/cmsis
- add entries for files that will move to ports/ dir
This allows to configure support for inplace special methods separately,
similar to "normal" and reverse special methods. This is useful, because
inplace methods are "the most optional" ones, for example, if inplace
methods aren't defined, the operation will be executed using normal
methods instead.
As a caveat, __iadd__ and __isub__ are implemented even if
MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS isn't defined. This is similar
to the state of affairs before binary operations refactor, and allows
to run existing tests even if MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS
isn't defined.
If MICROPY_PY_ALL_SPECIAL_METHODS is defined, actually define all special
methods (still subject to gating by e.g. MICROPY_PY_REVERSE_SPECIAL_METHODS).
This adds quite a number of qstr's, so should be used sparingly.
Update makeqstrdata.py to sort strings starting with "__" to the beginning
of qstr list, so they get low qstr id's, guaranteedly fitting in 8 bits.
Then use this property to further compact op_id => qstr mapping arrays.
Per https://docs.python.org/3/library/sys.html#sys.getsizeof:
getsizeof() calls the object’s __sizeof__ method. Previously, "getsizeof"
was used mostly to save on new qstr, as we don't really support calling
this method on arbitrary objects (so it was used only for reporting).
However, normalize it all now.
Header files that are considered internal to the py core and should not
normally be included directly are:
py/nlr.h - internal nlr configuration and declarations
py/bc0.h - contains bytecode macro definitions
py/runtime0.h - contains basic runtime enums
Instead, the top-level header files to include are one of:
py/obj.h - includes runtime0.h and defines everything to use the
mp_obj_t type
py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
and defines everything to use the general runtime support functions
Additional, specific headers (eg py/objlist.h) can be included if needed.
Qstr values fit in 16-bits (and this fact is used elsewhere in the code) so
no need to use more than that for the large lookup tables. The compiler
will anyway give a warning if the qstr values don't fit in 16 bits. Saves
around 80 bytes of code space for Thumb2 archs.
If, for class X, X.__add__(Y) doesn't exist (or returns NotImplemented),
try Y.__radd__(X) instead.
This patch could be simpler, but requires undoing operand swap and
operation switch to get non-confusing error message in case __radd__
doesn't exist.
