Commit Graph

3537 Commits

Author SHA1 Message Date
Scott Shawcroft 72d993d60c
py changes for supporting superclass constructors that take kwargs 2019-01-14 17:29:19 -08:00
Dan Halbert 1dc3957e72 LocalPeripheral is now Peripheral; more work on basic GATTS support; UART not working yet 2018-12-30 22:33:49 -05:00
Dan Halbert 4167bf5b24 wip: advertising works, but not connection 2018-12-27 00:04:04 -05:00
Dan Halbert bfa66861ef Merge remote-tracking branch 'adafruit/master' into bleio-rev 2018-12-13 16:33:15 -05:00
Scott Shawcroft 6ef8639971
Rework safe mode and have heap overwrite trigger it.
This creates a common safe mode mechanic that ports can share.
As a result, the nRF52 now has safe mode support as well.

The common safe mode adds a 700ms delay at startup where a reset
during that window will cause a reset into safe mode. This window
is designated by a yellow status pixel and flashing the single led
three times.

A couple NeoPixel fixes are included for the nRF52 as well.

Fixes #1034. Fixes #990. Fixes #615.
2018-12-06 14:24:20 -08:00
Dan Halbert 125901e4e2 Merge remote-tracking branch 'adafruit/master' into bleio-rev 2018-12-06 12:41:38 -05:00
Scott Shawcroft 7ad2e6ace3
Add stack validity check and raise an error when it happens.
The backtrace cannot be given because it relies on the validity
of the qstr data structures on the heap which may have been
corrupted.

In fact, it still can crash hard when the bytecode itself is
overwritten. To fix, we'd need a way to skip gathering the
backtrace completely.

This also increases the default stack size on M4s so it can
accomodate the stack needed by ASF4s nvm API.
2018-12-04 23:26:04 -08:00
Dan Halbert 80db2cec99 UART changes: timeout in secs, write bytes, etc. 2018-12-03 12:04:32 -05:00
Scott Shawcroft 9d07e95351
Add support for adding release info into adafruit/circuitpython-org
This also changes the build script to python with better output.
2018-11-30 00:30:57 -08:00
Dan Halbert 1763ffe245 More UUID work; use mp_raise for exceptions 2018-11-20 23:04:58 -05:00
Noralf Trønnes c5aa2e9300 Support OSError attributes
This adds support for the OSError attributes : errno, strerror, filename and filename2.
CPython only sets errno if 2 arguments has been passed in. This has not been implemented here.

CPython OSError.args is capped at 2 items for backward compatibility reasons. This has not been
implemented here.

MICROPY_CPYTHON_COMPAT has to be enabled to get these attributes.

mp_common_errno_to_str() has been extended to check mp_errno_to_str() as well. This is done to ease
reuse for the strerror argument.
2018-11-13 22:04:44 +01:00
Scott Shawcroft 355abc835e
Fix output overflow and make help translatable 2018-11-09 16:41:08 -08:00
Scott Shawcroft 168e23e466
Build refinement to handle warnings and quiet output 2018-11-09 00:11:43 -08:00
Scott Shawcroft dc9d338612
Merge pull request #1167 from notro/cpython_stdlib
Support CPython standard library
2018-10-24 12:57:41 -07:00
Scott Shawcroft cb0126131a
Use python3 for mpy-tool 2018-10-18 10:37:42 -07:00
Scott Shawcroft 4f9c8b7361
Add debug info to the generated frozen_mpy.c
It adds size info and uses macros for byte code to make it more
readable.
2018-10-17 17:45:47 -07:00
Noralf Trønnes db4a8f5d1a modsys: exc_info: Add traceback
Add traceback chain to sys.exec_info()[2].
No actual frame info is added, but just enough to recreate the printed
exception traceback.

Used by the unittest module which collects errors and failures and prints
them at the end.
2018-10-07 19:37:49 +02:00
Noralf Trønnes b897603cfa py/objboundmeth: Support __func__ property as in CPython
This gives access to the function underlying the bound method.

Used in the converted CPython stdlib logging.Formatter class to handle
overrriding a default converter method bound to a class variable.
The method becomes bound when accessed from an instance of that class.

I didn't investigate why CircuitPython turns it into a bound method.
2018-10-07 19:36:24 +02:00
Noralf Trønnes a63555abc1 py/builtinimport: Set __file__ on MPY modules
This sets the __file__ property on MPY modules like how it's done on pure python modules.
2018-10-07 19:36:12 +02:00
Uri Shaked cea079a022
fix compilation errors in emitinlinethumb.c 2018-10-07 02:09:44 +03:00
Noralf Trønnes 2d1fecc804 Revert "py/vm: Improve performance of opcode dispatch when using switch stmt."
This reverts commit 869024dd6e.

Ctrl-C stopped producing KeyboardInterrupt with this change on CircuitPython.

The Unix and stm32 ports handles Ctrl-C differently with a handler which is
probably why they where not affected.

Fixes #1092
2018-08-25 13:14:58 +02:00
Scott Shawcroft 4a4d84ba42
Merge pull request #1064 from notro/i2cslave
Add busio.I2CSlave
2018-08-17 16:51:45 -07:00
Scott Shawcroft c3c5951d44
Fix test that broke due to decompressing to too small of a scope. 2018-08-16 17:41:53 -07:00
Scott Shawcroft 2cd166b573
Fix esp and samd 2018-08-16 17:41:35 -07:00
Scott Shawcroft 137a30ad75
fix mpy-cross 2018-08-16 17:40:57 -07:00
Scott Shawcroft de5a9d72dc
Compress all translated strings with Huffman coding.
This saves code space in builds which use link-time optimization.
The optimization drops the untranslated strings and replaces them
with a compressed_string_t struct. It can then be decompressed to
a c string.

Builds without LTO work as well but include both untranslated
strings and compressed strings.

This work could be expanded to include QSTRs and loaded strings if
a compress method is added to C. Its tracked in #531.
2018-08-16 17:40:57 -07:00
Noralf Trønnes 1ed0e45fc2 Add i2cslave.I2CSlave bindings 2018-08-16 17:11:20 +02:00
Scott Shawcroft 4513bd6ea3
Fix translation newlines
Escape table was incorrect
2018-08-10 16:17:03 -07:00
Scott Shawcroft 24e53ad591
Rework escaping and fix ESP build. 2018-08-09 15:58:45 -07:00
Scott Shawcroft 1835f1ab14
Update filter and handle nested quotes 2018-08-09 14:16:28 -07:00
Scott Shawcroft 96ebf5bc3f
Two fixes and translate more strings.
* Fix finding translations with escaped characters.
* Add back \r to translations since its needed by screen.
2018-08-09 13:29:30 -07:00
Scott Shawcroft 933add6cd8
Support internationalisation. 2018-08-07 14:58:57 -07:00
Scott Shawcroft b50f46d4be
Rename to SRC_QSTR_PREPROCESSOR for clarity. 2018-08-02 11:07:22 -07:00
Scott Shawcroft f6d1b63ecd
Handle emitnative.c which is #included into other .c files. 2018-08-02 00:51:53 -07:00
Scott Shawcroft 8cf03d2d00
Speed up QSTR creation by pre-filtering files before pre-processing. 2018-08-02 00:51:52 -07:00
Dan Halbert 2e260a0eab Merge remote-tracking branch 'adafruit/master' into micropython-25ae98f-merge 2018-07-28 14:13:54 -04:00
Dan Halbert f48b70050e merge finished 2018-07-28 13:29:47 -04:00
Scott Shawcroft 1ffb0a714f
Merge remote-tracking branch 'adafruit/3.x' into import_merge 2018-07-25 11:49:18 -07:00
Dan Halbert bc760dd341 WIP: complete manual inspection of all significant changes 2018-07-23 21:34:25 -04:00
Dan Halbert be1d882a8b Prevent repetitive recursive scanning of dicts when making them long-lived 2018-07-19 15:19:21 -04:00
Dan Halbert 2809b4f9dd Merge branch 'master' into micropython-25ae98f-merge 2018-07-16 19:36:40 -04:00
Dan Halbert 6dfa527f79 do not call property if referenced by class, not object 2018-07-14 22:17:25 -04:00
Dan Halbert e2e01efa84 compiles and runs; hangs on import storage;storage.VfsFat.<tab> 2018-07-13 22:51:10 -04:00
Dan Halbert 0d27f4d9a6 continued WIP: almost compiling 2018-07-12 14:13:51 -04:00
Dan Halbert 7c219600a2 WIP: after merge; before testing 2018-07-11 16:45:30 -04:00
Scott Shawcroft cced51cbd2
Limit qstr pool size to reduce memory waste. 2018-07-03 05:45:51 -07:00
Scott Shawcroft 9de611c056
Always collect after an import
Imports generate a lot of garbage so cleaning it up immediately
reduces the likelihood longer lived data structures don't end up in
the middle of the heap.

Fixes #856
2018-07-03 05:45:50 -07:00
Scott Shawcroft 252aacdddf
Analysis fixes and long lived tweaks. 2018-07-03 05:45:50 -07:00
Damien George 25ae98f07c py/compile: Combine expr, xor_expr and and_expr into one function.
This and the previous 4 commits combined have change in code size of:

   bare-arm:   -92
minimal x86:  -544
   unix x64:  -544
unix nanbox:  -712
      stm32:  -116
     cc3200:  -128
    esp8266:  -348
      esp32:  -232
2018-06-22 17:00:29 +10:00
Damien George 36e474e83f py/compile: Combine or_test and and_test compile functions. 2018-06-22 17:00:29 +10:00
Damien George 1a7109d65a py/compile: Combine global and nonlocal statement compile functions. 2018-06-22 17:00:29 +10:00
Damien George d23bec3fc8 py/compile: Combine subscript_2 and subscript_3 into one function. 2018-06-22 17:00:29 +10:00
Damien George c149197928 py/compile: Combine break and continue compile functions. 2018-06-22 17:00:29 +10:00
Damien George 34344a413f py/stream: Remove stray empty line at start of file.
This was accidentally added in 6abede2ca9
2018-06-20 16:26:12 +10:00
Damien George 582b190764 py: Add checks for stream objects in print() and sys.print_exception(). 2018-06-20 15:57:10 +10:00
Damien George 2c8d130f70 py/stream: Update comment for mp_stream_write_adaptor. 2018-06-20 15:56:32 +10:00
Damien George c00ee200ac py/objarray: Replace 0x80 with new MP_OBJ_ARRAY_TYPECODE_FLAG_RW macro. 2018-06-18 13:40:53 +10:00
Damien George 6abede2ca9 py/stream: Introduce and use efficient mp_get_stream to access stream_p.
The existing mp_get_stream_raise() helper does explicit checks that the
input object is a real pointer object, has a non-NULL stream protocol, and
has the desired stream C method (read/write/ioctl).  In most cases it is
not necessary to do these checks because it is guaranteed that the input
object has the stream protocol and desired C methods.  For example, native
objects that use the stream wrappers (eg mp_stream_readinto_obj) in their
locals dict always have the stream protocol (or else they shouldn't have
these wrappers in their locals dict).

This patch introduces an efficient mp_get_stream() which doesn't do any
checks and just extracts the stream protocol struct.  This should be used
in all cases where the argument object is known to be a stream.  The
existing mp_get_stream_raise() should be used primarily to verify that an
object does have the correct stream protocol methods.

All uses of mp_get_stream_raise() in py/stream.c have been converted to use
mp_get_stream() because the argument is guaranteed to be a proper stream
object.

This patch improves efficiency of stream operations and reduces code size.
2018-06-18 12:35:56 +10:00
Damien George 035906419d extmod/uos_dupterm: Use native C stream methods on dupterm object.
This patch changes dupterm to call the native C stream methods on the
connected stream objects, instead of calling the Python readinto/write
methods.  This is much more efficient for native stream objects like UART
and webrepl and doesn't require allocating a special dupterm array.

This change is a minor breaking change from the user's perspective because
dupterm no longer accepts pure user stream objects to duplicate on.  But
with the recent addition of uio.IOBase it is possible to still create such
classes just by inheriting from uio.IOBase, for example:

    import uio, uos

    class MyStream(uio.IOBase):
        def write(self, buf):
            # existing write implementation
        def readinto(self, buf):
            # existing readinto implementation

    uos.dupterm(MyStream())
2018-06-12 15:06:11 +10:00
Damien George 7ad04d17da py/mkrules.mk: Regenerate all qstrs when config files change.
A port can define QSTR_GLOBAL_DEPENDENCIES to add extra files.
2018-06-12 13:53:43 +10:00
Yonatan Goldschmidt 6630354ffe extmod/moduhashlib: Allow to disable the sha256 class.
Via the config value MICROPY_PY_UHASHLIB_SHA256.  Default to enabled to
keep backwards compatibility.

Also add default value for the sha1 class, to at least document its
existence.
2018-06-12 13:50:11 +10:00
Damien George af0932a779 py/modio: Add uio.IOBase class to allow to define user streams.
A user class derived from IOBase and implementing readinto/write/ioctl can
now be used anywhere a native stream object is accepted.

The mapping from C to Python is:

    stream_p->read  --> readinto(buf)
    stream_p->write --> write(buf)
    stream_p->ioctl --> ioctl(request, arg)

Among other things it allows the user to:

- create an object which can be passed as the file argument to print:
  print(..., file=myobj), and then print will pass all the data to the
  object via the objects write method (same as CPython)
- pass a user object to uio.BufferedWriter to buffer the writes (same as
  CPython)
- use select.select on a user object
- register user objects with select.poll, in particular so user objects can
  be used with uasyncio
- create user files that can be returned from user filesystems, and import
  can import scripts from these user files

For example:

    class MyOut(io.IOBase):
        def write(self, buf):
            print('write', repr(buf))
            return len(buf)

    print('hello', file=MyOut())

The feature is enabled via MICROPY_PY_IO_IOBASE which is disabled by
default.
2018-06-12 12:29:26 +10:00
Damien George 6a445b60fa py/lexer: Add support for underscores in numeric literals.
This is a very convenient feature introduced in Python 3.6 by PEP 515.
2018-06-12 12:17:43 +10:00
Damien George 522ea80f06 py/gc: Add gc_sweep_all() function to run all remaining finalisers.
This patch adds the gc_sweep_all() function which does a garbage collection
without tracing any root pointers, so frees all the memory, and most
importantly runs any remaining finalisers.

This helps primarily for soft reset: it will close any open files, any open
sockets, and help to get the system back to a clean state upon soft reset.
2018-06-12 11:55:29 +10:00
Damien George 36c1052183 py/objtype: Optimise instance get/set/del by skipping special accessors.
This patch is a code optimisation, trading text bytes for speed.  On
pyboard it's an increase of 0.06% in code size for a gain (in pystone
performance) of roughly 6.5%.

The patch optimises load/store/delete of attributes in user defined classes
by not looking up special accessors (@property, __get__, __delete__,
__set__, __setattr__ and __getattr_) if they are guaranteed not to exist in
the class.

Currently, if you do my_obj.foo() then the runtime has to do a few checks
to see if foo is a property or has __get__, and if so delegate the call.
And for stores things like my_obj.foo = 1 has to first check if foo is a
property or has __set__ defined on it.

Doing all those checks each and every time the attribute is accessed has a
performance penalty.  This patch eliminates all those checks for cases when
it's guaranteed that the checks will always fail, ie no attributes are
properties nor have any special accessor methods defined on them.

To make this guarantee it checks all attributes of a user-defined class
when it is first created.  If any of the attributes of the user class are
properties or have special accessors, or any of the base classes of the
user class have them, then it sets a flag in the class to indicate that
special accessors must be checked for.  Then in the load/store/delete code
it checks this flag to see if it can take the shortcut and optimise the
lookup.

It's an optimisation that's pretty widely applicable because it improves
lookup performance for all methods of user defined classes, and stores of
attributes, at least for those that don't have special accessors.  And, it
allows to enable descriptors with minimal additional runtime overhead if
they are not used for a particular user class.

There is one restriction on dynamic class creation that has been introduced
by this patch: a user-defined class cannot go from zero special accessors
to one special accessor (or more) after that class has been subclassed.  If
the script attempts this an AttributeError is raised (see addition to
tests/misc/non_compliant.py for an example of this case).

The cost in code space bytes for the optimisation in this patch is:

   unix x64:  +528
unix nanbox:  +508
      stm32:  +192
     cc3200:  +200
    esp8266:  +332
      esp32:  +244

Performance tests that were done:

- on unix x86-64, pystone improved by about 5%
- on pyboard, pystone improved by about 6.5%, from 1683 up to 1794
- on pyboard, bm_chaos (from CPython benchmark suite) improved by about 5%
- on esp32, pystone improved by about 30% (but there are caching effects)
- on esp32, bm_chaos improved by about 11%
2018-06-08 12:12:08 +10:00
Damien George bace1a16d0 py/objtype: Don't expose mp_obj_instance_attr().
mp_obj_is_instance_type() can be used instead to check for instance types.
2018-06-08 11:48:25 +10:00
Damien George db5d8c97f1 py/obj.h: Introduce a "flags" entry in mp_obj_type_t. 2018-06-08 11:48:25 +10:00
Damien George a8b9e71ac1 py/mpconfig.h: Add default MICROPY_VFS_FAT config value.
At least to document it's existence.
2018-06-06 14:33:42 +10:00
Damien George a93144cb65 py/reader: Allow MICROPY_VFS_POSIX to work with MICROPY_READER_POSIX. 2018-06-06 14:28:23 +10:00
Damien George 8d82b0edbd extmod: Add VfsPosix filesystem component.
This VFS component allows to mount a host POSIX filesystem within the uPy
VFS sub-system.  All traditional POSIX file access then goes through the
VFS, allowing to sandbox a uPy process to a certain sub-dir of the host
system, as well as mount other filesystem types alongside the host
filesystem.
2018-06-06 14:28:23 +10:00
Damien George 1427f8f593 py/stream: Move definition of mp_stream_p_t from obj.h to stream.h.
Since a long time now, mp_obj_type_t no longer refers explicitly to
mp_stream_p_t but rather to an abstract "const void *protocol".  So there's
no longer any need to define mp_stream_p_t in obj.h and it can go with all
its associated definitions in stream.h.  Pretty much all users of this type
will already include the stream header.
2018-06-04 16:53:17 +10:00
Matt Wozniski 1c35dbfb5d Provide mp_errno_to_str even without uerrno
We can provide a basic version of mp_errno_to_str even if the uerrno
module won't be provided.  Rather than looking errno names up in the
uerrno module's globals dict, we'll just rely on a simple mapping in the
function itself.
2018-06-01 19:03:51 -04:00
Jeff Epler c60589c02b py/objtype: Fix assertion failures in super_attr by checking type.
Fixes assertion failures and segmentation faults when making calls like:

    super(1, 1).x
2018-05-30 11:14:07 +10:00
Jeff Epler 05b13fd292 py/objtype: Fix assertion failures in mp_obj_new_type by checking types.
Fixes assertion failures when the arguments to type() were not of valid
types, e.g., when making calls like:

    type("", (), 3)
    type("", 3, {})
2018-05-30 11:11:24 +10:00
Damien George dfeaea1441 py/objtype: Remove TODO comment about needing to check for property.
Instance members are always treated as values, even if they are properties.
A test is added to show this is the case.
2018-05-25 10:59:40 +10:00
Damien George 18e6358480 py/emit: Combine setup with/except/finally into one emit function.
This patch reduces code size by:

   bare-arm:   -16
minimal x86:  -156
   unix x64:  -288
unix nanbox:  -184
      stm32:   -48
     cc3200:   -16
    esp8266:   -96
      esp32:   -16

The last 10 patches combined reduce code size by:

   bare-arm:  -164
minimal x86: -1260
   unix x64: -3416
unix nanbox: -1616
      stm32:  -676
     cc3200:  -232
    esp8266: -1144
      esp32:  -268
2018-05-23 00:35:16 +10:00
Damien George 436e0d4c54 py/emit: Merge build set/slice into existing build emit function.
Reduces code size by:

   bare-arm:    +0
minimal x86:    +0
   unix x64:  -368
unix nanbox:  -248
      stm32:  -128
     cc3200:   -48
    esp8266:  -184
      esp32:   -40
2018-05-23 00:23:36 +10:00
Damien George d97906ca9a py/emit: Combine import from/name/star into one emit function.
Change in code size is:

   bare-arm:    +4
minimal x86:   -88
   unix x64:  -456
unix nanbox:   -88
      stm32:   -44
     cc3200:    +0
    esp8266:  -104
      esp32:    +8
2018-05-23 00:23:08 +10:00
Damien George 8a513da5a5 py/emit: Combine break_loop and continue_loop into one emit function.
Reduces code size by:

   bare-arm:    +0
minimal x86:    +0
   unix x64:   -80
unix nanbox:    +0
      stm32:   -12
     cc3200:    +0
    esp8266:   -28
      esp32:    +0
2018-05-23 00:23:04 +10:00
Damien George 6211d979ee py/emit: Combine load/store/delete attr into one emit function.
Reduces code size by:

   bare-arm:   -20
minimal x86:  -140
   unix x64:  -408
unix nanbox:  -140
      stm32:   -68
     cc3200:   -16
    esp8266:   -80
      esp32:   -32
2018-05-23 00:22:59 +10:00
Damien George a4941a8ba4 py/emit: Combine load/store/delete subscr into one emit function.
Reduces code size by:

   bare-arm:    -8
minimal x86:  -104
   unix x64:  -312
unix nanbox:  -120
      stm32:   -60
     cc3200:   -16
    esp8266:   -92
      esp32:   -24
2018-05-23 00:22:55 +10:00
Damien George d298013939 py/emit: Combine name and global into one func for load/store/delete.
Reduces code size by:

   bare-arm:   -56
minimal x86:  -300
   unix x64:  -576
unix nanbox:  -300
      stm32:  -164
     cc3200:   -56
    esp8266:  -236
      esp32:   -76
2018-05-23 00:22:47 +10:00
Damien George 26b5754092 py/emit: Combine build tuple/list/map emit funcs into one.
Reduces code size by:

   bare-arm:   -24
minimal x86:  -192
   unix x64:  -288
unix nanbox:  -184
      stm32:   -72
     cc3200:   -16
    esp8266:  -148
      esp32:   -32
2018-05-23 00:22:44 +10:00
Damien George e686c94052 py/emit: Combine yield value and yield-from emit funcs into one.
Reduces code size by:

   bare-arm:   -24
minimal x86:   -72
   unix x64:  -200
unix nanbox:   -72
      stm32:   -52
     cc3200:   -32
    esp8266:   -84
      esp32:   -24
2018-05-23 00:22:35 +10:00
Damien George 0a25fff956 py/emit: Combine fast and deref into one function for load/store/delete.
Reduces code size by:

   bare-arm:   -16
minimal x86:  -208
   unix x64:  -408
unix nanbox:  -248
      stm32:   -12
     cc3200:   -24
    esp8266:   -96
      esp32:   -44
2018-05-23 00:22:20 +10:00
Damien George 400273a799 py/objgenerator: Protect against reentering a generator.
Generators that are already executing cannot be reexecuted.  This patch
puts in a check for such a case.

Thanks to @jepler for finding the bug.
2018-05-22 16:54:03 +10:00
Damien George 771cb359af py/objgenerator: Save state in old_globals instead of local variable.
The code_state.old_globals variable is there to save the globals state so
should be used for this purpose, to avoid the need for additional local
variables on the C stack.
2018-05-22 16:39:19 +10:00
Jan Klusacek b318ebf101 py/modbuiltins: Add support for rounding integers.
As per CPython semantics.  This feature is controlled by
MICROPY_PY_BUILTINS_ROUND_INT which is disabled by default.
2018-05-22 14:18:16 +10:00
Damien George f2ec792554 py/parsenum: Adjust braces so they are balanced. 2018-05-22 13:20:00 +10:00
Damien George 6bd78741c1 py/gc: When GC threshold is hit don't unnecessarily collect twice.
Without this, if GC threshold is hit and there is not enough memory left to
satisfy the request, gc_collect() will run a second time and the search for
memory will happen again and will fail again.

Thanks to @adritium for pointing out this issue, see #3786.
2018-05-21 13:36:21 +10:00
Jeff Epler 95e43efc99 py/objfloat: Fix undefined integer behavior hashing negative zero.
Under ubsan, when evaluating hash(-0.) the following diagnostic occurs:

    ../../py/objfloat.c:102:15: runtime error: negation of
    -9223372036854775808 cannot be represented in type 'mp_int_t' (aka
    'long'); cast to an unsigned type to negate this value to itself

So do just that, to tell the compiler that we want to perform this
operation using modulo arithmetic rules.
2018-05-21 12:49:56 +10:00
Jeff Epler c4dafcef4f py/mpz: Avoid undefined behavior at integer overflow in mpz_hash.
Before this, ubsan would detect a problem when executing
hash(006699999999999999999999999999999999999999999999999999999999999999999999)

    ../../py/mpz.c:1539:20: runtime error: left shift of 1067371580458 by
    32 places cannot be represented in type 'mp_int_t' (aka 'long')

When the overflow does occur it now happens as defined by the rules of
unsigned arithmetic.
2018-05-21 12:48:26 +10:00
Jeff Epler 60eb5305f6 py/objfloat: Fix undefined shifting behavior in high-quality float hash.
When computing e.g. hash(0.4e3) with ubsan enabled, a diagnostic like the
following would occur:

    ../../py/objfloat.c:91:30: runtime error: shift exponent 44 is too
    large for 32-bit type 'int'

By casting constant "1" to the right type the intended value is preserved.
2018-05-21 12:42:22 +10:00
Jeff Epler 4f71a2a75a py/parsenum: Avoid undefined behavior parsing floats with large exponents.
Fuzz testing combined with the undefined behavior sanitizer found that
parsing unreasonable float literals like 1e+9999999999999 resulted in
undefined behavior due to overflow in signed integer arithmetic, and a
wrong result being returned.
2018-05-21 12:37:57 +10:00
Damien George 5efc575067 py/parsenum: Use int instead of mp_int_t for parsing float exponent.
There is no need to use the mp_int_t type which may be 64-bits wide, there
is enough bit-width in a normal int to parse reasonable exponents.  Using
int helps to reduce code size for 64-bit ports, especially nan-boxing
builds.  (Similarly for the "dig" variable which is now an unsigned int.)
2018-05-21 12:27:38 +10:00
Jeff Epler bc6c0b28bf py/emitbc: Avoid undefined behavior calling memset() with NULL 1st arg.
Calling memset(NULL, value, 0) is not standards compliant so we must add an
explicit check that emit->label_offsets is indeed not NULL before calling
memset (this pointer will be NULL on the first pass of the parse tree and
it's more logical / safer to check this pointer rather than check that the
pass is not the first one).

Code sanitizers will warn if NULL is passed as the first value to memset,
and compilers may optimise the code based on the knowledge that any pointer
passed to memset is guaranteed not to be NULL.
2018-05-21 12:04:20 +10:00
Damien George 828ce16dc8 py/compile: Change comment about ITER_BUF_NSLOTS to a static assertion. 2018-05-18 23:31:00 +10:00
Damien George 43d08d6dd6 py/misc.h: Add MP_STATIC_ASSERT macro to do static assertions. 2018-05-18 23:31:00 +10:00
Li Weiwei 3e6ab82179 py/repl: Fix handling of unmatched brackets and unfinished quotes.
Before this patch:

    >>> print(')
    ... ')
    Traceback (most recent call last):
      File "<stdin>", line 1
    SyntaxError: invalid syntax

After this patch:

    >>> print(')
    Traceback (most recent call last):
      File "<stdin>", line 1
    SyntaxError: invalid syntax

This matches CPython and prevents getting stuck in REPL continuation when a
1-quote is unmatched.
2018-05-18 15:23:02 +10:00
Damien George 869024dd6e py/vm: Improve performance of opcode dispatch when using switch stmt.
Before this patch, when using the switch statement for dispatch in the VM
(not computed goto) a pending exception check was done after each opcode.
This is not necessary and this patch makes the pending exception check only
happen when explicitly requested by certain opcodes, like jump.  This
improves performance of the VM by about 2.5% when using the switch.
2018-05-18 11:47:03 +10:00
Damien George 46ce395130 py/vm: Use enum names instead of magic numbers in multi-opcode dispatch. 2018-05-18 11:44:26 +10:00
Tom Collins a883fe12d9 py/objfun: Fix variable name in DECODE_CODESTATE_SIZE() macro.
This patch fixes the macro so you can pass any name in, and the macro will
make more sense if you're reading it on its own.  It worked previously
because n_state is always passed in as n_state_out_var.
2018-05-17 11:20:06 +10:00
Scott Shawcroft 6a8db03ade
Merge pull request #838 from rhooper/master
add supervisor.reload() to soft reboot from code
2018-05-16 17:52:16 -04:00
Scott Shawcroft 99e34e38eb
Merge pull request #829 from jerryneedell/jerryn_i2c
add keyword timeout to I2C -- only used for bitbangioi
2018-05-16 14:52:11 -04:00
Damien George 1b7487e519 py/vm: Adjust #if logic for gil_divisor so braces are balanced.
Having balanced braces { and } makes it easier to navigate the function.
2018-05-16 12:33:39 +10:00
Roy Hooper 90a09dba59 Merge branch 'master' of github.com:adafruit/circuitpython 2018-05-15 14:54:54 -04:00
Dan Halbert 54293397c5
Merge pull request #837 from godlygeek/human_readable_oserror
Human readable OSError messages
2018-05-15 10:12:09 -04:00
Damien George c97607db5c py/nlrx86: Use naked attribute on nlr_push for gcc 8.0 and higher.
gcc 8.0 supports the naked attribute for x86 systems so it can now be used
here.  And in fact it is necessary to use this for nlr_push because gcc 8.0
no longer generates a prelude for this function (even without the naked
attribute).
2018-05-15 11:17:28 +10:00
Roy Hooper 92b1cb5743 move reload exception to reload.c 2018-05-14 17:41:17 -04:00
Roy Hooper 567f3e30a7 Initial implementation of supervisor.reload() 2018-05-14 16:57:50 -04:00
Matt Wozniski 5833cbe6de Display human readable strings for common OSErrors
Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
2018-05-14 15:12:40 -04:00
bildzeitung a8cc4a1a62 Updated library source: ucollections -> collections 2018-05-14 14:33:50 -04:00
Jerry Needell d6c26942a5 add timeout keyword to I2C - for bitbangio - ignored for busio 2018-05-13 21:54:44 -04:00
Damien George 749b16174b py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs.
This patch moves the start of the root pointer section in mp_state_ctx_t
so that it skips entries that are not pointers and don't need scanning.

Previously, the start of the root pointer section was at the very beginning
of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t).
This was the original assembler version of the NLR code was hard-coded to
have the nlr_top pointer at the start of this state structure.  But now
that the NLR code is partially written in C there is no longer this
restriction on the location of nlr_top (and a comment to this effect has
been removed in this patch).

So now the root pointer section starts part way through the
mp_state_thread_t structure, after the entries which are not root pointers.

This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER
outside the root pointer section.

Moving non-pointer entries out of the root pointer section helps to make
the GC more precise and should help to prevent some cases of collectable
garbage being kept.

This patch also has a measurable improvement in performance of the
pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of
roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-13 22:53:28 +10:00
Damien George 9630376dbc py/mpconfig.h: Be stricter when autodetecting machine endianness.
This patch changes 2 things in the endianness detection:

1. Don't assume that __BYTE_ORDER__ not being __ORDER_LITTLE_ENDIAN__ means
   that the machine is big endian, so add an explicit check that this macro
   is indeed __ORDER_BIG_ENDIAN__ (same with __BYTE_ORDER, __LITTLE_ENDIAN
   and __BIG_ENDIAN).  A machine could have PDP endianness.

2. Remove the checks which base their autodetection decision on whether any
   little or big endian macros are defined (eg __LITTLE_ENDIAN__ or
   __BIG_ENDIAN__).  Just because a system defines these does not mean it
   has that endianness.

See issue #3760.
2018-05-11 21:51:34 +10:00
Damien George 6046e68fe1 py/repl: Initialise q_last variable to prevent compiler warnings.
Some older compilers cannot deduce that q_last is always written to before
being read.
2018-05-11 13:48:47 +10:00
Damien George 095d397017 py/objdeque: Fix sign extension bug when computing len of deque object.
For cases where size_t is smaller than mp_int_t (eg nan-boxing builds) the
difference between two size_t's is not sign extended into mp_int_t and so
the result is never negative.  This patch fixes this bug by using ssize_t
for the type of the result.
2018-05-11 13:44:50 +10:00
Damien George 3678a6bdc6 py/modbuiltins: Make built-in dir support the __dir__ special method.
If MICROPY_PY_ALL_SPECIAL_METHODS is enabled then dir() will now delegate
to the special method __dir__ if the object it is listing has this method.
2018-05-10 23:14:23 +10:00
Damien George 29d28c2574 py/modbuiltins: In built-in dir make use of mp_load_method_protected.
This gives dir() better behaviour when listing the attributes of a user
type that defines __getattr__: it will now not list those attributes for
which __getattr__ raises AttributeError (meaning the attribute is not
supported by the object).
2018-05-10 23:07:19 +10:00
Damien George 7241d90272 py/repl: Use mp_load_method_protected to prevent leaking of exceptions.
This patch fixes the possibility of a crash of the REPL when tab-completing
an object which raises an exception when its attributes are accessed.

See issue #3729.
2018-05-10 23:05:43 +10:00
Damien George 529860643b py/modbuiltins: Make built-in hasattr work properly for user types.
It now allows __getattr__ in a user type to raise AttributeError when the
attribute does not exist.
2018-05-10 23:03:30 +10:00
Damien George bc87b862fd py/runtime: Add mp_load_method_protected helper which catches exceptions
This new helper function acts like mp_load_method_maybe but is wrapped in
an NLR handler so it can catch exceptions.  It prevents AttributeError from
propagating out, and optionally all other exceptions.  This helper can be
used to fully implement hasattr (see follow-up commit), and also for cases
where mp_load_method_maybe is used but it must now raise an exception.
2018-05-10 23:00:04 +10:00
Damien George eb88803ac8 py/{modbuiltins,repl}: Start qstr probing from after empty qstr.
The list of qstrs starts with MP_QSTR_NULL followed by MP_QSTR_, and these
should never appear in dir() or REPL tab completion, so skip them.
2018-05-09 16:15:02 +10:00
Jeff Epler 11a97bdffe uhashlib: masquerade as hashlib for python3 compatibility 2018-05-06 12:31:54 -05:00
Damien George 3cf02be4e0 py/emitnx86: Fix 32-bit x86 native emitter build by including header. 2018-05-04 20:39:16 +10:00
Scott Shawcroft 3d36f63f9e
Merge pull request #788 from jepler/binascii-py3
(u)binascii: better python3 compatibility
2018-05-02 10:15:31 -07:00
Lars Kellogg-Stedman 4e977aafe4 correct path to conf.py in makeversionhdr.py
the py/makeversionhdr.py script was looking for `conf.py` in the
`docs/` directory, but this was relocated in 46e7f8e.  This is used by
the fallback `get_version_info_from_docs_conf` method, which is only
consulted if `git` is unavailable in the build environment.

Closes #791.
2018-05-02 08:50:17 -04:00
Damien George 60db80920a py/builtinhelp: Change occurrence of mp_uint_t to size_t. 2018-05-02 16:50:28 +10:00
Jeff Epler c2f0c57a26 ubinascii: masquerade as binascii for python3 compatibility 2018-05-01 17:10:53 -05:00
Damien George 6b4b6d388b py/obj.h: Fix math.e constant for nan-boxing builds.
Due to a typo, math.e was too small by around 6e-11.
2018-05-01 23:25:18 +10:00
Ayke van Laethem d43c737756 py/stream: Use uPy errno instead of system's for non-blocking check.
This is a more consistent use of errno codes.  For example, it may be that
a stream returns MP_EAGAIN but the mp_is_nonblocking_error() macro doesn't
catch this value because it checks for EAGAIN instead (which may have a
different value than MP_EAGAIN when MICROPY_USE_INTERNAL_ERRNO is enabled).
2018-05-01 15:54:50 +10:00
Damien George 96740be357 py/mperrno: Define MP_EWOULDBLOCK as EWOULDBLOCK, not EAGAIN.
Most modern systems have EWOULDBLOCK aliased to EAGAIN, ie they have the
same value.  But some systems use different values for these errnos and if
a uPy port is using the system errno values (ie not the internal uPy
values) then it's important to be able to distinguish EWOULDBLOCK from
EAGAIN.  Eg if a system call returned EWOULDBLOCK it must be possible to
check for this return value, and this patch makes this now possible.
2018-05-01 15:53:25 +10:00
Ayke van Laethem deaa46aa66 py/nlrthumb: Fix Clang support wrt use of "return 0".
Clang defines __GNUC__ so we have to check for it specifically.
2018-04-27 15:10:42 +10:00
ladyada 1418445b5e fix for spaces in path 2018-04-12 23:12:36 -04:00
Damien George ef12a4bd05 py: Refactor how native emitter code is compiled with a file per arch.
Instead of emitnative.c having configuration code for each supported
architecture, and then compiling this file multiple times with different
macros defined, this patch adds a file per architecture with the necessary
code to configure the native emitter.  These files then #include the
emitnative.c file.

This simplifies emitnative.c (which is already very large), and simplifies
the build system because emitnative.c no longer needs special handling for
compilation and qstr extraction.
2018-04-10 15:06:47 +10:00
Jeff Epler cbf981f330 py/objgenerator: Check stack before resuming a generator.
This turns a hard crash in a recursive generator into a 'maximum recursion
depth exceeded' exception.
2018-04-10 14:06:26 +10:00
Damien George cf31d384f1 py/stream: Switch stream close operation from method to ioctl.
This patch moves the implementation of stream closure from a dedicated
method to the ioctl of the stream protocol, for each type that implements
closing.  The benefits of this are:

1. Rounds out the stream ioctl function, which already includes flush,
   seek and poll (among other things).

2. Makes calling mp_stream_close() on an object slightly more efficient
   because it now no longer needs to lookup the close method and call it,
   rather it just delegates straight to the ioctl function (if it exists).

3. Reduces code size and allows future types that implement the stream
   protocol to be smaller because they don't need a dedicated close method.

Code size reduction is around 200 bytes smaller for x86 archs and around
30 bytes smaller for the bare-metal archs.
2018-04-10 13:41:32 +10:00
Jeff Epler d6cf5c6749 py/objstr: In find/rfind, don't crash when end < start. 2018-04-05 16:14:17 +10:00
Damien George f1df86a017 py/objint: Simplify LHS arg type checking in int binary op functions.
The LHS passed to mp_obj_int_binary_op() will always be an integer, either
a small int or a big int, so the test for this type doesn't need to include
an "other, unsupported type" case.
2018-04-05 01:11:26 +10:00
Damien George 3f420c0c27 py: Don't include mp_optimise_value or opt_level() if compiler disabled.
Without the compiler enabled the mp_optimise_value is unused, and the
micropython.opt_level() function is not useful, so exclude these from the
build to save RAM and code size.
2018-04-04 14:24:03 +10:00
Damien George 323b5f7270 py/modsys: Don't compile getsizeof function if feature is disabled. 2018-04-04 14:23:25 +10:00
Damien George bc36521386 py/vm: Optimise handling of stackless mode when pystack is enabled.
When pystack is enabled mp_obj_fun_bc_prepare_codestate() will always
return a valid pointer, and if there is no more pystack available then it
will raise an exception (a RuntimeError).  So having pystack enabled with
stackless enabled automatically gives strict stackless mode.  There is
therefore no need to have code for strict stackless mode when pystack is
enabled, and this patch optimises the VM for such a case.
2018-04-04 00:51:10 +10:00
Damien George c7f880eda3 py/vm: Don't do unnecessary updates of ip and sp variables.
Neither the ip nor sp variables are used again after the execution of the
RAISE_VARARGS opcode, so they don't need to be updated.
2018-04-04 00:46:31 +10:00
Dan Halbert 435e894fa0 Merge branch 'master' into 3.0_hid 2018-04-02 19:19:43 -04:00
Scott Shawcroft e4ae1e3d59
Merge pull request #734 from jepler/str-find-backwards-circuitpython
py/objstr: Don't crash when end < start
2018-04-01 22:41:59 -07:00
Jeff Epler a909007fef py/objgenerator: Check stack before resuming a generator
This turns a hard crash in a recursive generator into
a 'maximum recursion depth exceeded' exception.
2018-04-01 16:40:15 -05:00
Jeff Epler 0041df0c6b py/objstr: Don't crash when end < start
.. and add testcases for the same.

(crash found by afl-fuzz)
2018-03-31 22:17:11 -05:00
Scott Shawcroft 3215b85568
Merge pull request #728 from jepler/double-splat-crash-circuitpython
py/bc: Turn assertion error into exception
2018-03-31 09:57:25 -07:00
Jeff Epler 853f7ac4d0 py/bc: Turn assertion error into exception 2018-03-31 08:44:29 -05:00
Dan Halbert df91878d2e WIP: works with just keyboard but not complex report descriptor 2018-03-30 23:24:00 -04:00
Damien George f50b64cab5 py/runtime: Be sure that non-intercepted thrown object is an exception.
The VM expects that, if mp_resume() returns MP_VM_RETURN_EXCEPTION, then
the returned value is an exception instance (eg to add a traceback to it).
It's possible that a value passed to a generator's throw() is not an
exception so must be explicitly checked for if the thrown value is not
intercepted by the generator.

Thanks to @jepler for finding the bug.
2018-03-30 12:43:38 +11:00
Damien George 3280788195 py/runtime: Check that keys in dicts passed as ** args are strings.
Prior to this patch the code would crash if a key in a ** dict was anything
other than a str or qstr.  This is because mp_setup_code_state() assumes
that keys in kwargs are qstrs (for efficiency).

Thanks to @jepler for finding the bug.
2018-03-30 11:13:32 +11:00
Jeff Epler ff06a45599 Fix assertion failures in super_attr
micropython: ../../py/objtype.c:1100: super_attr: Assertion `MP_OBJ_IS_TYPE(self->type, &mp_type_type)' failed.

e.g., when making calls like
    super(1, 1).x
2018-03-29 06:42:10 -05:00
Jeff Epler 6da8d7c465 Fix assertion failures in mp_obj_new_type
Fixes the following assertion failures when the arguments to type()
were not of valid types:
micropython: ../../py/objtype.c:984: mp_obj_new_type: Assertion `MP_OBJ_IS_TYPE(bases_tuple, &mp_type_tuple)' failed.
micropython: ../../py/objtype.c:994: mp_obj_new_type: Assertion `MP_OBJ_IS_TYPE(items[i], &mp_type_type)' failed.

e.g., when making calls like
    type("", (), 3)
    type("", 3, {})
2018-03-29 06:42:10 -05:00
Jeff Epler a55988a547 Fix assertion failure in mpz_divmod_inpl
.. turning this from an assertion failure into an exception:
    pow(1,1,0)
2018-03-29 06:42:10 -05:00
Scott Shawcroft 4517ab8ba4
Merge pull request #709 from jepler/core-class-superproperty
Make test core_class_superproperty.py succeed
2018-03-26 21:59:05 -07:00
Jeff Epler 355bf8b553 Conditionally compile out nonstandard array/struct typecodes
.. defaulting to off for circuitpython-supported boards, on for others.

.. fixing up the tests that fail when it is turned off, so that they skip
instead of failing
2018-03-26 18:13:49 -05:00
Jeff Epler 74fefe45a4 super(): Do the same lookup tasks as regular getattr
.. in the presence of properties and descriptors
2018-03-25 15:11:40 -05:00
Jeff Epler 922b7c3131 Don't assume the type of the prop->proxy objects
This fixes a crash running the cpydiff/core_class_superproperty.py
test, but it does not fix the difference to cpython3.

Closes: #705
2018-03-25 15:11:30 -05:00
Jeff Epler cdb83b18ec Implement * and *= for array.array 2018-03-23 07:37:07 -05:00
Damien George 5edce4539b py/objexcept: Make MP_DEFINE_EXCEPTION public so ports can define excs. 2018-03-17 00:31:40 +11:00
Damien George f6a1f18603 py/makeqstrdefs.py: Optimise by using compiled re's so it runs faster.
By using pre-compiled regexs, using startswith(), and explicitly checking
for empty lines (of which around 30% of the input lines are), automatic
qstr extraction is speed up by about 10%.
2018-03-16 23:54:06 +11:00
Damien George e0bc438e4b py/obj.h: Move declaration of mp_obj_list_init to objlist.h.
If this function is used then objlist.h is already included to get the
definition of mp_obj_list_t.
2018-03-13 14:03:15 +11:00
Damien George 9f811e9096 py/obj.h: Clean up by removing commented-out inline versions of macros. 2018-03-13 14:01:55 +11:00
Damien George d4b55eff44 py/misc.h: Remove unused count_lead_ones() inline function.
This function was never used for unicode/utf8 handling code, or anything
else, so remove it to keep things clean.
2018-03-13 13:23:30 +11:00
Damien George 9884a2c712 py/objint: Remove unreachable code checking for int type in format func.
All callers of mp_obj_int_formatted() are expected to pass in a valid int
object, and they do:

- mp_obj_int_print() should always pass through an int object because it is
  the print special method for int instances.

- mp_print_mp_int() checks that the argument is an int, and if not converts
  it to a small int.

This patch saves around 20-50 bytes of code space.
2018-03-02 11:01:24 +11:00
Damien George 955ee6477f py/formatfloat: Fix case where floats could render with negative digits.
Prior to this patch, some architectures (eg unix x86) could render floats
with "negative" digits, like ")".  For example, '%.23e' % 1e-80 would come
out as "1.0000000000000000/)/(,*0e-80".  This patch fixes the known cases.
2018-03-01 17:00:02 +11:00
Damien George 7b050fa76c py/formatfloat: Fix case where floats could render with a ":" character.
Prior to this patch, some architectures (eg unix x86) could render floats
with a ":" character in them, eg 1e+39 would come out as ":e+38" (":" is
just after "9" in ASCII so this is like 10e+38).  This patch fixes some of
these cases.
2018-03-01 16:02:59 +11:00
Damien George bc12eca461 py/formatfloat: Fix rounding of %f format with edge-case FP values.
Prior to this patch the %f formatting of some FP values could be off by up
to 1, eg '%.0f' % 123 would return "122" (unix x64).  Depending on the FP
precision (single vs double) certain numbers would format correctly, but
others wolud not.  This patch should fix all cases of rounding for %f.
2018-03-01 15:51:03 +11:00
Damien George a9f6d49218 py/vm: Simplify handling of special-case STOP_ITERATION in yield from.
There's no need to have MP_OBJ_NULL a special case, the code can re-use
the MP_OBJ_STOP_ITERATION value to signal the special case and the VM can
detect this with only one check (for MP_OBJ_STOP_ITERATION).
2018-02-27 15:48:09 +11:00
Damien George 22ade2f5c4 py/vm: Fix case of handling raised StopIteration within yield from.
This patch concerns the handling of an NLR-raised StopIteration, raised
during a call to mp_resume() which is handling the yield from opcode.

Previously, commit 6738c1dded introduced code
to handle this case, along with a test.  It seems that it was lucky that
the test worked because the code did not correctly handle the stack pointer
(sp).

Furthermore, commit 79d996a57b improved the
way mp_resume() propagated certain exceptions: it changed raising an NLR
value to returning MP_VM_RETURN_EXCEPTION.  This change meant that the
test introduced in gen_yield_from_ducktype.py was no longer hitting the
code introduced in 6738c1dded.

The patch here does two things:

1. Fixes the handling of sp in the VM for the case that yield from is
   interrupted by a StopIteration raised via NLR.

2. Introduces a new test to check this handling of sp and re-covers the
   code in the VM.
2018-02-27 15:39:31 +11:00
Damien George 9d8347a9aa py/mpstate.h: Add repl_line state for MICROPY_REPL_EVENT_DRIVEN. 2018-02-26 16:08:58 +11:00
Damien George f75c7ad1a9 py/mpz: In mpz_clone, remove unused check for NULL dig.
This path for src->deg==NULL is never used because mpz_clone() is always
called with an argument that has a non-zero integer value, and hence has
some digits allocated to it (mpz_clone() is a static function private to
mpz.c all callers of this function first check if the integer value is zero
and if so take a special-case path, bypassing the call to mpz_clone()).

There is some unused and commented-out functions that may actually pass a
zero-valued mpz to mpz_clone(), so some TODOs are added to these function
in case they are needed in the future.
2018-02-25 22:59:19 +11:00
Damien George c0bcf00ed1 py/asm*.c: Remove unnecessary check for num_locals<0 in asm entry func.
All callers of the asm entry function guarantee that num_locals>=0, so no
need to add an explicit check for it.  Use an assertion instead.

Also, the signature of asm_x86_entry is changed to match the other asm
entry functions.
2018-02-24 23:10:20 +11:00
Damien George 7dfa56e40e py/compile: Adjust c_assign_atom_expr() to use return instead of goto.
Makes the flow of the function a little more obvious, and allows to reach
100% coverage of compile.c when using gcov.
2018-02-24 23:03:17 +11:00
Damien George 638b860066 extmod/vfs_fat: Merge remaining vfs_fat_misc.c code into vfs_fat.c.
The only function left in vfs_fat_misc.c is fat_vfs_import_stat() which
can logically go into vfs_fat.c, allowing to remove vfs_fat_misc.c.
2018-02-23 17:24:57 +11:00
Damien George 6af4515969 py: Use "GEN" consistently for describing files generated in the build. 2018-02-22 12:48:51 +11:00
Damien George 65ef59a9b5 py/py.mk: Remove .. path component from list of extmod files.
This just makes it a bit cleaner in the output of the build process:
instead of "CC ../../py/../extmod/" there is now "CC ../../extmod/".
2018-02-22 12:48:51 +11:00
Damien George 8ca469cae2 py/py.mk: Split list of uPy sources into core and extmod files.
If a port only needs the core files then it can now use the $(PY_CORE_O)
variable instead of $(PY_O).  $(PY_EXTMOD_O) contains the list of extmod
files (including some files from lib/). $(PY_O) retains its original
definition as the list of all object file (including those for frozen code)
and is a convenience variable for ports that want everything.
2018-02-22 12:48:15 +11:00
Damien George 6e675c1baa py/objdeque: Use m_new0 when allocating items to avoid need to clear.
Saves a few bytes of code space, and is more efficient because with
MICROPY_GC_CONSERVATIVE_CLEAR enabled by default all memory is already
cleared when allocated.
2018-02-21 23:36:46 +11:00
Damien George 160d670868 py/objdeque: Protect against negative maxlen in deque constructor.
Otherwise passing -1 as maxlen will lead to a zero allocation and
subsequent unbound buffer overflow in deque.append() because i_put is
allowed to grow without bound.
2018-02-21 23:34:17 +11:00
Damien George 6c3faf6c17 py/objdeque: Allow to compile without warnings by disabling deque_clear. 2018-02-21 22:52:58 +11:00
Paul Sokolovsky 970eedce8f py/objdeque: Implement ucollections.deque type with fixed size.
So far, implements just append() and popleft() methods, required for
a normal queue. Constructor doesn't accept an arbitarry sequence to
initialize from (am empty deque is always created), so an empty tuple
must be passed as such. Only fixed-size deques are supported, so 2nd
argument (size) is required.

There's also an extension to CPython - if True is passed as 3rd argument,
append(), instead of silently overwriting the oldest item on queue
overflow, will throw IndexError. This behavior is desired in many
cases, where queues should store information reliably, instead of
silently losing some items.
2018-02-21 22:39:25 +11:00
Damien George fe3e17b026 py/objint: Use MP_OBJ_IS_STR_OR_BYTES macro instead of 2 separate ones. 2018-02-21 00:20:46 +11:00
Damien George 8769049e93 py/objstr: Remove unnecessary check for positive splits variable.
At this point in the code the variable "splits" is guaranteed to be
positive due to the check for "splits == 0" above it.
2018-02-20 19:19:02 +11:00
Damien George 7e2a48858c py/modmicropython: Allow to have stack_use() func without mem_info().
The micropython.stack_use() function is useful to query the current C stack
usage, and it's inclusion in the micropython module doesn't need to be tied
to the inclusion of mem_info()/qstr_info() because it doesn't rely on any
of the code from these functions.  So this patch introduces the config
option MICROPY_PY_MICROPYTHON_STACK_USE which can be used to independently
control the inclusion of stack_use().  By default it is enabled if
MICROPY_PY_MICROPYTHON_MEM_INFO is enabled (thus not changing any of the
existing ports).
2018-02-20 18:30:22 +11:00
Damien George 209936880d py/builtinimport: Add compile-time option to disable external imports.
The new option is MICROPY_ENABLE_EXTERNAL_IMPORT and is enabled by default
so that the default behaviour is the same as before.  With it disabled
import is only supported for built-in modules, not for external files nor
frozen modules.  This allows to support targets that have no filesystem of
any kind and that only have access to pre-supplied built-in modules
implemented natively.
2018-02-20 18:00:44 +11:00
Damien George 6e7819ee2e py/objmodule: Factor common code for calling __init__ on builtin module. 2018-02-20 17:56:58 +11:00
Damien George 4e469085c1 py/objstr: Protect against creating bytes(n) with n negative.
Prior to this patch uPy (on a 32-bit arch) would have severe issues when
calling bytes(-1): such a call would call vstr_init_len(vstr, -1) which
would then +1 on the len and call vstr_init(vstr, 0), which would then
round this up and allocate a small amount of memory for the vstr.  The
bytes constructor would then attempt to zero out all this memory, thinking
it had allocated 2^32-1 bytes.
2018-02-19 16:25:30 +11:00
Damien George 165aab12a3 py/repl: Generalise REPL autocomplete to use qstr probing.
This patch changes the way REPL autocomplete finds matches.  It now probes
the target object for all qstrs via mp_load_method_maybe to look for a
match with the given input string.  Similar to how the builtin dir()
function works, this new algorithm now find all methods and instances of
user-defined classes including attributes of their parent classes.  This
helps a lot at the REPL prompt for user-discovery and to autocomplete names
even for classes that are derived.

The downside is that this new algorithm is slower than the previous one,
and in particular will be slower the more qstrs there are in the system.
But because REPL autocomplete is primarily used in an interactive way it is
not that important to make it fast, as long as it is "fast enough" compared
to human reaction.

On a slow microcontroller (CPU running at 16MHz) the autocomplete time for
a list of 35 names in the outer namespace (pressing tab at a bare prompt)
takes about 160ms with this algorithm, compared to about 40ms for the
previous implementation (this time includes the actual printing of the
names as well).  This time of 160ms is very reasonable especially given the
new functionality of listing all the names.

This patch also decreases code size by:

   bare-arm:    +0
minimal x86:  -128
   unix x64:  -128
unix nanbox:  -224
      stm32:   -88
     cc3200:   -80
    esp8266:   -92
      esp32:   -84
2018-02-19 16:12:44 +11:00
Damien George 98647e83c7 py/modbuiltins: Simplify and generalise dir() by probing qstrs.
This patch improves the builtin dir() function by probing the target object
with all possible qstrs via mp_load_method_maybe.  This is very simple (in
terms of implementation), doesn't require recursion, and allows to list all
methods of user-defined classes (without duplicates) even if they have
multiple inheritance with a common parent.  The downside is that it can be
slow because it has to iterate through all the qstrs in the system, but
the "dir()" function is anyway mostly used for testing frameworks and user
introspection of types, so speed is not considered a priority.

In addition to providing a more complete implementation of dir(), this
patch is simpler than the previous implementation and saves some code
space:

   bare-arm:   -80
minimal x86:   -80
   unix x64:   -56
unix nanbox:   -48
      stm32:   -80
     cc3200:   -80
    esp8266:  -104
      esp32:   -64
2018-02-19 16:12:44 +11:00
Damien George a8775aaeb0 py/qstr: Add QSTR_TOTAL() macro to get number of qstrs. 2018-02-19 16:12:44 +11:00
Damien George 2a0cbc0d38 py/gc: Update comment now that gc_drain_stack is called gc_mark_subtree. 2018-02-19 16:08:20 +11:00
Ayke van Laethem 736faef223 py/gc: Make GC stack pointer a local variable.
This saves a bit in code size, and saves some precious .bss RAM:

                 .text  .bss
minimal CROSS=1: -28    -4
unix (64-bit):   -64    -8
2018-02-19 16:05:46 +11:00
Ayke van Laethem 5c9e5618e0 py/gc: Rename gc_drain_stack to gc_mark_subtree and pass it first block.
This saves a bit in code size:

minimal CROSS=1: -44
unix:            -96
2018-02-19 16:00:59 +11:00
Ayke van Laethem ea7cf2b738 py/gc: Reduce code size by specialising VERIFY_MARK_AND_PUSH macro.
This macro is written out explicitly in the two locations that it is used
and then the code is optimised, opening possibilities for further
optimisations and reducing code size:

unix:            -48
minimal CROSS=1: -32
stm32:           -32
2018-02-19 15:58:49 +11:00
Mike Wadsten a3e01d3642 py/objdict: Disallow possible modifications to fixed dicts. 2018-02-18 21:51:04 -06:00
Damien George 7b2a9b059a py/pystack: Use "pystack exhausted" as error msg for out of pystack mem.
Using the message "maximum recursion depth exceeded" for when the pystack
runs out of memory can be misleading because the pystack can run out for
reasons other than deep recursion (although in most cases pystack
exhaustion is probably indirectly related to deep recursion).  And it's
important to give the user more precise feedback as to the reason for the
error: if they know precisely that the pystack was exhausted then they have
a chance to increase the amount of memory available to the pystack (as
opposed to not knowing if it was the C stack or pystack that ran out).

Also, C stack exhaustion is more serious than pystack exhaustion because it
could have been that the C stack overflowed and overwrote/corrupted some
data and so the system must be restarted.  The pystack can never corrupt
data in this way so pystack exhaustion does not require a system restart.
Knowing the difference between these two cases is therefore important.

The actual exception type for pystack exhaustion remains as RuntimeError so
that programatically it behaves the same as a C stack exhaustion.
2018-02-19 00:26:14 +11:00
Ayke van Laethem 5591bd237a py/nlrthumb: Do not mark nlr_push as not returning anything.
By adding __builtin_unreachable() at the end of nlr_push, we're
essentially telling the compiler that this function will never return.
When GCC LTO is in use, this means that any time nlr_push() is called
(which is often), the compiler thinks this function will never return
and thus eliminates all code following the call.

Note: I've added a 'return 0' for older GCC versions like 4.6 which
complain about not returning anything (which doesn't make sense in a
naked function). Newer GCC versions (tested 4.8, 5.4 and some others)
don't complain about this.
2018-02-18 01:35:27 +01:00
Damien George 73d1d20b46 py/objexcept: Remove long-obsolete mp_const_MemoryError_obj.
This constant exception instance was once used by m_malloc_fail() to raise
a MemoryError without allocating memory, but it was made obsolete long ago
by 3556e45711.  The functionality is now
replaced by the use of mp_emergency_exception_obj which lives in the global
uPy state, and which can handle any exception type, not just MemoryError.
2018-02-15 16:50:02 +11:00