Commit Graph

143 Commits

Author SHA1 Message Date
Paul Sokolovsky fb742cdc12 py/{builtinimport,frozenmod}: Rework frozen modules support to support packages.
Now frozen modules is treated just as a kind of VFS, and all operations
performed on it correspond to operations on normal filesystem. This allows
to support packages properly, and potentially also data files.

This change also have changes to rework frozen bytecode modules support to
use the same framework, but it's not finished (and actually may not work,
as older adhox handling of any type of frozen modules is removed).
2016-05-21 21:38:50 +03:00
Damien George 0a2e9650f5 py: Add ability to have frozen persistent bytecode from .mpy files.
The config variable MICROPY_MODULE_FROZEN is now made of two separate
parts: MICROPY_MODULE_FROZEN_STR and MICROPY_MODULE_FROZEN_MPY.  This
allows to have none, either or both of frozen strings and frozen mpy
files (aka frozen bytecode).
2016-04-13 16:07:47 +01:00
Damien George 4b72b3a133 py: Change type signature of builtin funs that take variable or kw args.
With this patch the n_args parameter is changed type from mp_uint_t to
size_t.
2016-01-11 00:49:27 +00:00
Damien George dd5353a405 py: Add MICROPY_ENABLE_COMPILER and MICROPY_PY_BUILTINS_EVAL_EXEC opts.
MICROPY_ENABLE_COMPILER can be used to enable/disable the entire compiler,
which is useful when only loading of pre-compiled bytecode is supported.
It is enabled by default.

MICROPY_PY_BUILTINS_EVAL_EXEC controls support of eval and exec builtin
functions.  By default they are only included if MICROPY_ENABLE_COMPILER
is enabled.

Disabling both options saves about 40k of code size on 32-bit x86.
2015-12-18 12:35:44 +00:00
Damien George 999cedb90f py: Wrap all obj-ptr conversions in MP_OBJ_TO_PTR/MP_OBJ_FROM_PTR.
This allows the mp_obj_t type to be configured to something other than a
pointer-sized primitive type.

This patch also includes additional changes to allow the code to compile
when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of
mp_uint_t, and various casts.
2015-11-29 14:25:35 +00:00
Damien George cbf7674025 py: Add MP_ROM_* macros and mp_rom_* types and use them. 2015-11-29 14:25:04 +00:00
Damien George 432e8275a9 py: Allow to import compiled bytecode files. 2015-11-13 12:49:18 +00:00
Paul Sokolovsky 9780e55274 builtinimport: Fix running package submodule with -m.
When "micropython -m pkg.mod" command was used, relative imports in pkg.mod
didn't work, because pkg.mod.__name__ was set to __main__, and the fact that
it's a package submodule was missed. This is an original workaround to this
issue. TODO: investigate and compare how CPython deals with this issue.
2015-06-29 00:26:45 +03:00
Paul Sokolovsky c4045f57e3 builtinimport: Catch case when relative import happens without active package.
CPython raises SystemError in this case, but we don't have that enabled, so
raise ImportError.
2015-06-27 00:40:21 +03:00
Paul Sokolovsky 07408cbd1f unix: Make micropython -m <module> work for frozen modules.
This requires some special handling, which was previosuly applied only to
the main code path.
2015-06-06 00:10:58 +03:00
Damien George 044c473de2 py: Add %q format support to mp_[v]printf, and use it. 2015-04-16 14:30:16 +00:00
Paul Sokolovsky 9e6c82960e builtinimport: Revamp&refactor handling of relative imports.
Relative imports are based of a package, so we're currently at a module
within a package, we should get to package first.

Also, factor out path travsering operation, but this broke testing for
boundary errors with relative imports. TODO: reintroduce them, together
with proper tests.
2015-02-16 12:11:34 +02:00
Paul Sokolovsky 078172dcab builtinimport: Improve debugging output. 2015-02-16 12:11:03 +02:00
Damien George 827b0f747b py: Change vstr_null_terminate -> vstr_null_terminated_str, returns str. 2015-01-29 13:57:23 +00:00
Damien George 0d3cb6726d py: Change vstr so that it doesn't null terminate buffer by default.
This cleans up vstr so that it's a pure "variable buffer", and the user
can decide whether they need to add a terminating null byte.  In most
places where vstr is used, the vstr did not need to be null terminated
and so this patch saves code size, a tiny bit of RAM, and makes vstr
usage more efficient.  When null termination is needed it must be
done explicitly using vstr_null_terminate.
2015-01-28 23:43:01 +00:00
Paul Sokolovsky 8064892c9b builtinimport: Make sure that qstr is used properly to load frozen modules. 2015-01-21 23:14:46 +02:00
Paul Sokolovsky 640e0b221e py: Implement very simple frozen modules support.
Only modules (not packages) supported now. Source modules can be converted
to frozen module structures using tools/make-frozen.py script.
2015-01-20 11:52:12 +02:00
Damien George 963a5a3e82 py, unix: Allow to compile with -Wsign-compare.
See issue #699.
2015-01-16 17:47:07 +00:00
Damien George 51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Paul Sokolovsky 8a8c1fc82f py: Add basic framework for issuing compile/runtime warnings. 2015-01-01 22:09:18 +02:00
Damien George 78d702c300 py: Allow builtins to be overridden.
This patch adds a configuration option (MICROPY_CAN_OVERRIDE_BUILTINS)
which, when enabled, allows to override all names within the builtins
module.  A builtins override dict is created the first time the user
assigns to a name in the builtins model, and then that dict is searched
first on subsequent lookups.  Note that this implementation doesn't
allow deleting of names.

This patch also does some refactoring of builtins code, creating the
modbuiltins.c file.

Addresses issue #959.
2014-12-09 16:19:48 +00:00
Damien George a4c52c5a3d py: Optimise lexer by exposing lexer type.
mp_lexer_t type is exposed, mp_token_t type is removed, and simple lexer
functions (like checking current token kind) are now inlined.

This saves 784 bytes ROM on 32-bit unix, 348 bytes on stmhal, and 460
bytes on bare-arm.  It also saves a tiny bit of RAM since mp_lexer_t
is a bit smaller.  Also will run a bit more efficiently.
2014-12-05 19:35:18 +00:00
Damien George 1e9a92f84f py: Use shorter, static error msgs when ERROR_REPORTING_TERSE enabled.
Going from MICROPY_ERROR_REPORTING_NORMAL to
MICROPY_ERROR_REPORTING_TERSE now saves 2020 bytes ROM for ARM Thumb2,
and 2200 bytes ROM for 32-bit x86.

This is about a 2.5% code size reduction for bare-arm.
2014-11-06 17:36:16 +00:00
Sven Wegener 238ab5013b py: Deactivate more code without MICROPY_PY_SYS
When compiler optimization has been turned on, gcc knows that this code
block is not going to be executed. But with -O0 it complains about
path_items being used uninitialized.

Signed-off-by: Sven Wegener <sven.wegener@stealer.net>
2014-11-05 22:52:51 +00:00
Damien George 7860c2a68a py: Fix some macros defines; cleanup some includes. 2014-11-05 21:16:41 +00:00
Paul Sokolovsky e503512f83 unix: Implement -m option (execute module from stdlib).
Support for packages as argument not implemented, but otherwise error and
exit handling should be correct. This for example will allow to do:

pip-micropython install micropython-test.pystone
micropython -m test.pystone
2014-10-26 15:54:19 +00:00
Paul Sokolovsky e5a3759ff5 py: Factor out mp_obj_is_package() function. 2014-10-25 22:31:33 +03:00
Paul Sokolovsky 8becca7c82 py: mp_builtin___import__(): Add const to arg type. 2014-10-25 22:31:26 +03:00
Damien George c14a81662c py: Add module weak link support.
With this patch a port can enable module weak link support and provide
a dict of qstr->module mapping.  This mapping is looked up only if an
import fails to find the requested module in the filesystem.

This allows to have the builtin module named, eg, usocket, and provide
a weak link of "socket" to the same module, but this weak link can be
overridden if a file by the name "socket.py" is found in the import
path.
2014-10-12 20:18:40 +01:00
Damien George c4d0868df1 py: Implement proper context save/restore for eval/exec; factor code.
This has benefits all round: code factoring for parse/compile/execute,
proper context save/restore for exec, allow to sepcify globals/locals
for eval, and reduced ROM usage by >100 bytes on stmhal and unix.

Also, the call to mp_parse_compile_execute is tail call optimised for
the import code, so it doesn't increase stack memory usage.
2014-10-05 20:13:34 +01:00
Damien George a91ac2011f py: Make compiler return a proper exception on SyntaxError. 2014-10-05 19:01:34 +01:00
Damien George 42f3de924b py: Convert [u]int to mp_[u]int_t where appropriate.
Addressing issue #50.
2014-10-03 17:44:14 +00:00
Damien George 52b5d76a6b py: Free non-interned strings in the parser when not needed.
mp_parse_node_free now frees the memory associated with non-interned
strings.  And the parser calls mp_parse_node_free when discarding a
non-used node (such as a doc string).

Also, the compiler now frees the parse tree explicitly just before it
exits (as opposed to relying on the caller to do this).

Addresses issue #708 as best we can.
2014-09-23 15:31:56 +00:00
Damien George eaaebf3291 stmhal: Initialise stack pointer correctly.
Stack is full descending and must be 8-byte aligned.  It must start off
pointing to just above the last byte of RAM.

Previously, stack started pointed to last byte of RAM (eg 0x2001ffff)
and so was not 8-byte aligned.  This caused a bug in combination with
alloca.

This patch also updates some debug printing code.

Addresses issue #872 (among many other undiscovered issues).
2014-09-23 10:59:05 +01:00
Damien George 4abff7500f py: Change uint to mp_uint_t in runtime.h, stackctrl.h, binary.h.
Part of code cleanup, working towards resolving issue #50.
2014-08-30 14:59:21 +01:00
Damien George d182b98a37 py: Change all uint to mp_uint_t in obj.h.
Part of code cleanup, working towards resolving issue #50.
2014-08-30 14:19:41 +01:00
Damien George 9c4cbe2ac0 py: Make tuple and list use mp_int_t/mp_uint_t.
Part of code cleanup, to resolve issue #50.
2014-08-30 14:04:14 +01:00
Paul Sokolovsky d0f5e61ab5 py: Implement __file__ attribute for modules. 2014-07-28 21:21:59 +03:00
Damien George 8a0801ad24 py: Make 3 functions static. 2014-06-11 19:55:46 +01:00
Marcus von Appen 0c90eb1658 - FreeBSD provides alloca() via stdlib.h, in contrast to Linux and Windows
- Move the includes for alloca() intp mpconfigport.h
2014-06-07 09:36:04 +02:00
Chris Angelico daf973ae00 Change comments (mainly URLs) to no longer specifically say Python 3.3 2014-06-06 03:51:03 +10:00
Paul Sokolovsky 75ce9256b2 objstr: Implement "%(key)s" % {} formatting for strings and dicts.
Also, make sure that args to "*" format specifiers are bounds-checked
properly and don't lead for segfaults in case of mismatch.
2014-06-05 20:06:15 +03:00
Damien George 2617eebf2f Change const byte* to const char* where sensible.
This removes need for some casts (at least, more than it adds need
for new casts!).
2014-05-25 22:27:57 +01:00
Damien George ee3fd46f13 Rename configuration variables controling Python features.
Now of the form MICROPY_PY_*.  See issue #35.
2014-05-24 23:03:12 +01:00
Damien George 58ebde4664 Tidy up some configuration options.
MP_ALLOC_* -> MICROPY_ALLOC_*
MICROPY_PATH_MAX -> MICROPY_ALLOC_PATH_MAX
MICROPY_ENABLE_REPL_HELPERS -> MICROPY_HELPER_REPL
MICROPY_ENABLE_LEXER_UNIX -> MICROPY_HELPER_LEXER_UNIX
MICROPY_EXTRA_* -> MICROPY_PORT_*

See issue #35.
2014-05-21 20:32:59 +01:00
Paul Sokolovsky da9f0924ef py, unix: Add copyright for modules I worked closely on. 2014-05-13 18:41:25 +03:00
Paul Sokolovsky ad6178bb08 builtinimport: Fix broken namespace imports due to dup vstr_cut_tail_bytes(). 2014-05-10 19:00:03 +03:00
Paul Sokolovsky f9589d2f23 builtinimport: Fix comment orphaned by one of previous commits. 2014-05-10 18:46:02 +03:00
Damien George 04b9147e15 Add license header to (almost) all files.
Blanket wide to all .c and .h files.  Some files originating from ST are
difficult to deal with (license wise) so it was left out of those.

Also merged modpyb.h, modos.h, modstm.h and modtime.h in stmhal/.
2014-05-03 23:27:38 +01:00
stijn 98e2ee0ec1 py: Use <alloca.h> for alloca()
alloca() is declared in alloca.h which als happens to be included by stdlib.h.
On mingw however it resides in malloc.h only.
So if we include alloca.h directly, and add an alloca.h for mingw in it's port
directory we can get rid of the mingw-specific define to include malloc.h
and the other ports are happy as well.
2014-05-03 10:14:53 +02:00
Paul Sokolovsky f54bcbf099 py, unix: Make "mpconfig.h" be first included, as other headers depend on it.
Specifically, nlr.h does.
2014-05-02 17:48:40 +03:00
Paul Sokolovsky 20e9ed3916 builtinimport: If there was error compiling imported module, raise exception. 2014-04-22 02:53:20 +03:00
Paul Sokolovsky 353b0289f3 py: Add win32-specific header for alloca(). 2014-04-20 22:24:44 +03:00
Paul Sokolovsky a5854d2bc5 builtinimport: Add basic support for namespace packages.
That was easy - just avoid erroring out on seeing candidate dir for namespace
package. That's far from being complete though - namespace packages should
support importing portions of package from different sys.path entries, here
we require first matching entry to contain all namespace package's portions.

And yet, that's a way to put parts of the same Python package into multiple
installable package - something we really need for *Micro*Python.
2014-04-15 01:30:25 +03:00
Damien George 49f20b841d py: Add more #if's for configurable MOD_SYS. 2014-04-13 13:05:16 +01:00
Damien George 640e7e4779 Merge pull request #476 from pfalcon/static-sys
Convert sys module to static allocation
2014-04-13 12:52:39 +01:00
Damien George 4b01de44ba py: Add traceback info to syntax errors.
Should fix issue #463.
2014-04-13 11:56:02 +01:00
Paul Sokolovsky 5500cdeec7 py, unix: Convert sys module to static representation. 2014-04-13 07:02:56 +03:00
Paul Sokolovsky a5afc9009f builtinimport: Implement relative imports. 2014-04-12 18:25:17 +03:00
Paul Sokolovsky 2ff3d9d0b2 builtinimport: Set __path__ attribute ASAP as it's clear we have a package.
This helps with handling "recursive" imports in sane manner, for example
when foo/__init__.py has something like "from foo import submod".
2014-04-12 02:55:18 +03:00
Paul Sokolovsky 13d52df4c5 builtinimport: Set __path__ attribute on packages.
Per https://docs.python.org/3.3/reference/import.html , this is the way to
tell module from package: "Specifically, any module that contains a __path__
attribute is considered a package." And it for sure will be needed to
implement relative imports.
2014-04-12 00:39:55 +03:00
Paul Sokolovsky e081329098 builtinimport: Elaborate debug output support. 2014-04-12 00:39:55 +03:00
Andrew Scheller f78cfaf8b5 Remove exception name from inside the exception message
This prevents micropython printing exception messages like
ImportError: ImportError: No module named 'foo'
2014-04-09 19:56:38 +01:00
Damien George 65cad12d38 py: Add option to compiler to specify default code emitter.
Also add command line option to unix port to select emitter.
2014-04-06 11:48:15 +01:00
Damien George 7efc5b3f34 py: Make globals and locals proper dictionary objects.
Finishes addressing issue #424.

In the end this was a very neat refactor that now makes things a lot
more consistent across the py code base.  It allowed some
simplifications in certain places, now that everything is a dict object.

Also converted builtins tables to dictionaries.  This will be useful
when we need to turn builtins into a proper module.
2014-04-05 22:36:42 +01:00
Damien George 8b0535e23f py: Change module globals from mp_map_t* to mp_obj_dict_t*.
Towards addressing issue #424.

Had a small increase to ROM usage (order 60 bytes).
2014-04-05 21:53:54 +01:00
Damien George ea13f407a3 py: Change nlr_jump to nlr_raise, to aid in debugging.
This does not affect code size or performance when debugging turned off.

To address issue #420.
2014-04-05 18:32:08 +01:00
Damien George df6567e634 Merge map.h into obj.h.
Pretty much everyone needs to include map.h, since it's such an integral
part of the Micro Python object implementation.  Thus, the definitions
are now in obj.h instead.  map.h is removed.
2014-03-30 13:54:02 +01:00
Damien George d17926db71 Rename rt_* to mp_*.
Mostly just a global search and replace.  Except rt_is_true which
becomes mp_obj_is_true.

Still would like to tidy up some of the names, but this will do for now.
2014-03-30 13:35:08 +01:00
Damien George caac542b23 Proper support for registering builtin modules in ROM.
Comes with some refactoring of code and renaming of files.  All modules
are now named mod*.[ch].
2014-03-25 14:18:18 +00:00
Damien George 24ff063e80 py: Remove obsolete declarations; make mp_obj_get_array consistent. 2014-03-24 10:47:13 +00:00
xbe efe3422394 py: Clean up includes.
Remove unnecessary includes. Add includes that improve portability.
2014-03-17 02:43:40 -07:00
Damien George 280e7208d8 Add vstr_ins and vstr_cut_out; improve stmhal readline. 2014-03-15 14:33:09 +00:00
Paul Sokolovsky feacaa12ac __import__: Catch relative import attempts and throw NotImplementedError. 2014-02-21 01:15:20 +02:00
Paul Sokolovsky fb7f94392d import: Implement "from pkg.mod import sym" syntax properly.
http://docs.python.org/3.3/library/functions.html#__import__ :
"When the name variable is of the form package.module, normally, the
top-level package (the name up till the first dot) is returned, not
the module named by name. However, when a non-empty fromlist argument
is given, the module named by name is returned."
2014-02-20 00:37:12 +02:00
Paul Sokolovsky 91ba7a54c5 builtinimport: Get the basic (and only basic) package imports work. 2014-02-16 02:55:46 +02:00
Paul Sokolovsky d3783570a4 builtinimport.c: Recognize "namespace package" and error out as unsupported.
See http://www.python.org/dev/peps/pep-0420/#specification for spec. See
https://github.com/micropython/micropython/issues/298 for the discussion
of the implemented behavior.
2014-02-16 02:55:46 +02:00
Damien George c5966128c7 Implement proper exception type hierarchy.
Each built-in exception is now a type, with base type BaseException.
C exceptions are created by passing a pointer to the exception type to
make an instance of.  When raising an exception from the VM, an
instance is created automatically if an exception type is raised (as
opposed to an exception instance).

Exception matching (RT_BINARY_OP_EXCEPTION_MATCH) is now proper.

Handling of parse error changed to match new exceptions.

mp_const_type renamed to mp_type_type for consistency.
2014-02-15 16:10:44 +00:00
Damien George 698ec21e46 Make mp_obj_str_get_data return char* instead of byte*.
Can't decide which is better for string type, char or byte pointer.
Changing to char removes a few casts.  Really need to do proper unicode.
2014-02-08 18:17:23 +00:00
Damien George 354d15a964 Implement fixed buffer vstrs; use them for import path. 2014-02-06 21:11:19 +00:00
Damien George e09ffa1400 Search paths properly on import and execute __init__.py if it exists. 2014-02-05 23:57:48 +00:00
Paul Sokolovsky e11b17c25f Implement support for sys.path when loading modules.
sys.path is not initialized by rt_init(), that's left for platform-specific
startup code. (For example, bare metal port may have some hardcoded defaults,
and let user change sys.path directly; while port for OS with environment
feature can take path from environment). If it's not explicitly initialized,
modules will be imported only from a current directory.
2014-02-05 01:40:41 +02:00
Paul Sokolovsky 1d938c9503 Expose __import__() function. 2014-02-04 00:47:06 +02:00
Damien George b829b5caec Implement mp_parse_node_free; print properly repr(string). 2014-01-25 13:51:19 +00:00
Damien George 5fa93b6755 Second stage of qstr revamp: uPy str object can be qstr or not. 2014-01-22 14:35:10 +00:00
Damien George 55baff4c9b Revamp qstrs: they now include length and hash.
Can now have null bytes in strings.  Can define ROM qstrs per port using
qstrdefsport.h
2014-01-21 21:40:13 +00:00
Paul Sokolovsky d720ab5236 Implement modules as singletons Python semantics.
In Python, importing module several times returns same underlying module
object. This also fixes import statement handling for builtin modules.

There're still issues:
1. CPython exposes set of loaded modules as sys.modules, we may want to
do that either.
2. Builtin modules are implicitly imported, which is not really correct.
We should separate registering a (builtin) module and importing a module.
CPython keeps builtin module names in sys.builtin_module_names .
2014-01-20 00:38:39 +02:00
Damien George 08335004cf Add source file name and line number to error messages.
Byte code has a map from byte-code offset to source-code line number,
used to give better error messages.
2014-01-18 23:24:36 +00:00
Damien George 9528cd66d7 Convert parse errors to exceptions.
Parser no longer prints an error, but instead returns an exception ID
and message.
2014-01-15 21:23:31 +00:00
Damien George 2300537c79 Cleanup built-ins, and fix some compiler warnings/errors. 2014-01-13 19:39:01 +00:00
Damien George eb7bfcb286 Split qstr into pools, and put initial pool in ROM.
Qstr's are now split into a linked-list of qstr pools.  This has 2
benefits: the first pool can be in ROM (huge benefit, since we no longer
use RAM for the core qstrs), and subsequent pools use m_new for the next
pool instead of m_renew (thus avoiding a huge single table for all the
qstrs).

Still would be better to use a hash table, but this scheme takes us part
of the way (eventually convert the pools to hash tables).

Also fixed bug with import.

Also improved the way the module code is referenced (not magic number 1
anymore).
2014-01-04 15:57:35 +00:00
Damien George 1fb031744f Change mp_compile so that it returns a function object for the module. 2014-01-03 14:22:03 +00:00
Damien George 66028ab6dc Basic implementation of import.
import works for simple cases.  Still work to do on finding the right
script, and setting globals/locals correctly when running an imported
function.
2014-01-03 14:03:48 +00:00