Commit Graph

25 Commits

Author SHA1 Message Date
Petr Viktorin
25a9bccdee py/compile: Disallow 'import *' outside module level.
This check follows CPython's behaviour, because 'import *' always populates
the globals with the imported names, not locals.

Since it's safe to do this (doesn't lead to a crash or undefined behaviour)
the check is only enabled for MICROPY_CPYTHON_COMPAT.

Fixes issue #5121.
2019-10-04 16:46:47 +10:00
Damien George
c8c0fd4ca3 py: Rework and compress second part of bytecode prelude.
This patch compresses the second part of the bytecode prelude which
contains the source file name, function name, source-line-number mapping
and cell closure information.  This part of the prelude now begins with a
single varible length unsigned integer which encodes 2 numbers, being the
byte-size of the following 2 sections in the header: the "source info
section" and the "closure section".  After decoding this variable unsigned
integer it's possible to skip over one or both of these sections very
easily.

This scheme saves about 2 bytes for most functions compared to the original
format: one in the case that there are no closure cells, and one because
padding was eliminated.
2019-10-01 12:26:22 +10:00
Damien George
b5ebfadbd6 py: Compress first part of bytecode prelude.
The start of the bytecode prelude contains 6 numbers telling the amount of
stack needed for the Python values and exceptions, and the signature of the
function.  Prior to this patch these numbers were all encoded one after the
other (2x variable unsigned integers, then 4x bytes), but using so many
bytes is unnecessary.

An entropy analysis of around 150,000 bytecode functions from the CPython
standard library showed that the optimal Shannon coding would need about
7.1 bits on average to encode these 6 numbers, compared to the existing 48
bits.

This patch attempts to get close to this optimal value by packing the 6
numbers into a single, varible-length unsigned integer via bit-wise
interleaving.  The interleaving scheme is chosen to minimise the average
number of bytes needed, and at the same time keep the scheme simple enough
so it can be implemented without too much overhead in code size or speed.
The scheme requires about 10.5 bits on average to store the 6 numbers.

As a result most functions which originally took 6 bytes to encode these 6
numbers now need only 1 byte (in 80% of cases).
2019-10-01 12:26:22 +10:00
Damien George
5716c5cf65 py/persistentcode: Bump .mpy version to 5.
The bytecode opcodes have changed (there are more, and they have been
reordered).
2019-09-26 16:39:37 +10:00
Damien George
67fdfebe64 tests: Update tests for changes to opcode ordering. 2019-09-26 15:27:11 +10:00
Damien George
48f43b77aa tests: Add tests for overriding builtins.__import__. 2019-07-31 22:37:44 +10:00
Damien George
1e23a29c8a tests/import: Add test for importing x64 native code. 2019-03-08 17:20:17 +11:00
Damien George
5996eeb48f py/persistentcode: Add a qstr window to save mpy files more efficiently.
This is an implementation of a sliding qstr window used to reduce the
number of qstrs stored in a .mpy file.  The window size is configured to 32
entries which takes a fixed 64 bytes (16-bits each) on the C stack when
loading/saving a .mpy file.  It allows to remember the most recent 32 qstrs
so they don't need to be stored again in the .mpy file.  The qstr window
uses a simple least-recently-used mechanism to discard the least recently
used qstr when the window overflows (similar to dictionary compression).
This scheme only needs a single pass to save/load the .mpy file.

Reduces mpy file size by about 25% with a window size of 32.
2019-03-05 16:25:07 +11:00
Paul Sokolovsky
d94aa577a6 tests/import_long_dyn: Test for "import *" of a long dynamic name.
Such names aren't stored as qstr in module dict, and there was a bug in
"import *" handling which assumed any name in a module dict is a qstr.
2018-11-01 13:33:16 +11:00
Damien George
27ca9ab8b2 tests/import: Add .exp file for module_getattr.py to not require Py 3.7. 2018-10-23 11:56:58 +11:00
Paul m. p. P
454cca6016 py/objmodule: Implement PEP 562's __getattr__ for modules.
Configurable via MICROPY_MODULE_GETATTR, disabled by default.  Among other
things __getattr__ for modules can help to build lazy loading / code
unloading at runtime.
2018-10-23 11:22:50 +11:00
Damien George
6d8816fe84 tests/import: Add test for importing invalid .mpy file. 2018-06-18 17:50:34 +10:00
Damien George
409fc8f9c1 tests/import: Update comment now that uPy raises correct exception. 2017-06-28 12:21:29 +10:00
Damien George
3a9445c6b3 tests/import: Add a test for the builtin __import__ function. 2017-06-28 12:21:29 +10:00
Damien George
63e291de70 py/builtinimport: Raise ValueError for bad relative import, per CPython. 2017-01-16 16:21:04 +11:00
Rami Ali
50e14ca619 tests/import: Improve builtinimport.c test coverage. 2017-01-16 15:59:33 +11:00
Damien George
67f3edc10a tests/import: Add a test which uses ... in from-import statement. 2016-12-21 11:25:53 +11:00
Damien George
5e22afce41 tests: Improve test coverage of py/compile.c. 2016-10-11 12:30:32 +11:00
Damien George
3c582bc7cb tests/import: Add test for compiling "import a.b as c". 2016-09-30 14:48:22 +10:00
Paul Sokolovsky
9896314f5b tests: Add test for relative import without package context. 2015-06-27 00:40:22 +03:00
Paul Sokolovsky
ee831cafa9 tests: Add another testcase for relative imports. 2015-02-16 12:11:41 +02:00
Paul Sokolovsky
d0f5e61ab5 py: Implement __file__ attribute for modules. 2014-07-28 21:21:59 +03:00
Damien George
539681fffd tests: Rename test scripts, changing - to _ for consistency.
From now on, all new tests must use underscore.

Addresses issue #727.
2014-07-05 06:14:29 +01:00
Damien George
559d8239ca tests: Move gen_context to import tests, because it relies on import. 2014-04-17 23:21:52 +01:00
Damien George
5cd0b2227f tests: Split out those tests requiring float and import.
Tests in basics (which should probably be renamed to core) should not
rely on float, or import any non-built-in files.  This way these tests
can be run when those features are not available.

All test in basics now pass on the pyboard using stmhal port, except for
string-repr which has some issues with character hex printing.
2014-04-17 16:21:43 +01:00