Commit Graph

22 Commits

Author SHA1 Message Date
Paul Sokolovsky 93e353e384 py/gc: Implement GC running by allocation threshold.
Currently, MicroPython runs GC when it could not allocate a block of memory,
which happens when heap is exhausted. However, that policy can't work well
with "inifinity" heaps, e.g. backed by a virtual memory - there will be a
lot of swap thrashing long before VM will be exhausted. Instead, in such
cases "allocation threshold" policy is used: a GC is run after some number of
allocations have been made. Details vary, for example, number or total amount
of allocations can be used, threshold may be self-adjusting based on GC
outcome, etc.

This change implements a simple variant of such policy for MicroPython. Amount
of allocated memory so far is used for threshold, to make it useful to typical
finite-size, and small, heaps as used with MicroPython ports. And such GC policy
is indeed useful for such types of heaps too, as it allows to better control
fragmentation. For example, if a threshold is set to half size of heap, then
for an application which usually makes big number of small allocations, that
will (try to) keep half of heap memory in a nice defragmented state for an
occasional large allocation.

For an application which doesn't exhibit such behavior, there won't be any
visible effects, except for GC running more frequently, which however may
affect performance. To address this, the GC threshold is configurable, and
by default is off so far. It's configured with gc.threshold(amount_in_bytes)
call (can be queries without an argument).
2016-07-21 00:37:30 +03:00
Paul Sokolovsky 426112ce88 extmod/moduos_dupterm: Reserve buffer bytearray object for dupterm.
Allocating it for each read/write operation is a memory fragmentation
hazard.
2016-07-04 13:32:30 +03:00
Damien George 4cec63a9db py: Implement a simple global interpreter lock.
This makes the VM/runtime thread safe, at the cost of not being able to
run code in parallel.
2016-06-28 11:28:50 +01:00
Damien George 1f54ad2aed py: Make interning of qstrs thread safe. 2016-06-28 11:28:50 +01:00
Damien George c93d9caa8b py/gc: Make memory manager and garbage collector thread safe.
By using a single, global mutex, all memory-related functions (alloc,
free, realloc, collect, etc) are made thread safe.  This means that only
one thread can be in such a function at any one time.
2016-06-28 11:28:49 +01:00
Damien George 27cc07721b py: Add basic _thread module, with ability to start a new thread. 2016-06-28 11:28:48 +01:00
Damien George 330165a2cc py: Add MP_STATE_THREAD to hold state specific to a given thread. 2016-06-28 11:09:31 +01:00
Damien George ea23520403 py: Add MICROPY_DYNAMIC_COMPILER option to config compiler at runtime.
This new compile-time option allows to make the bytecode compiler
configurable at runtime by setting the fields in the mp_dynamic_compiler
structure.  By using this feature, the compiler can generate bytecode
that targets any MicroPython runtime/VM, regardless of the host and
target compile-time settings.

Options so far that fall under this dynamic setting are:
- maximum number of bits that a small int can hold;
- whether caching of lookups is used in the bytecode;
- whether to use unicode strings or not (lexer behaviour differs, and
  therefore generated string constants differ).
2016-02-25 10:05:46 +00:00
Paul Sokolovsky 72085a669b py/mpstate.h: fs_user_mount is now standard, reusable uPy functionality. 2016-02-15 00:02:03 +02:00
Paul Sokolovsky 00ee84e1e1 py: Clean up instantiation of dupterm object.
To comply with already established scheme for extmod's.
2016-01-01 14:22:57 +02:00
Damien George 257848587f py/qstr: Use size_t instead of mp_uint_t when counting allocated bytes. 2015-12-17 12:41:40 +00:00
Damien George d977d268e8 py/gc: Use size_t instead of mp_uint_t to count things related to heap.
size_t is the correct type to use to count things related to the size of
the address space.  Using size_t (instead of mp_uint_t) is important for
the efficiency of ports that configure mp_uint_t to larger than the
machine word size.
2015-12-16 20:09:11 -05:00
Paul Sokolovsky 1a1d11fa32 py/modsys: Implement sys.modules.
This for example will allow people to reload modules which didn't load
successfully (e.g. due to syntax error).
2015-12-05 00:13:29 +02:00
Damien George 999cedb90f py: Wrap all obj-ptr conversions in MP_OBJ_TO_PTR/MP_OBJ_FROM_PTR.
This allows the mp_obj_t type to be configured to something other than a
pointer-sized primitive type.

This patch also includes additional changes to allow the code to compile
when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of
mp_uint_t, and various casts.
2015-11-29 14:25:35 +00:00
Damien George 94fe6e523d py/gc: Move away from using mp_uint_t, instead use uintptr_t and size_t.
The GC works with concrete pointers and so the types should reflect this.
2015-11-29 14:25:04 +00:00
Damien George 994ff738c8 py/mpstate: Make mp_pending_exception volatile.
It can change asynchronously.
2015-11-17 14:27:21 +00:00
Paul Sokolovsky e0d7740a22 extmod/modlwip: slip: Use stream protocol and be port-independent.
Based on the original patch by Galen Hazelwood:
https://github.com/micropython/micropython/pull/1517 .
2015-10-27 00:04:59 +03:00
Damien George ade9a05236 py: Improve allocation policy of qstr data.
Previous to this patch all interned strings lived in their own malloc'd
chunk.  On average this wastes N/2 bytes per interned string, where N is
the number-of-bytes for a quanta of the memory allocator (16 bytes on 32
bit archs).

With this patch interned strings are concatenated into the same malloc'd
chunk when possible.  Such chunks are enlarged inplace when possible,
and shrunk to fit when a new chunk is needed.

RAM savings with this patch are highly varied, but should always show an
improvement (unless only 3 or 4 strings are interned).  New version
typically uses about 70% of previous memory for the qstr data, and can
lead to savings of around 10% of total memory footprint of a running
script.

Costs about 120 bytes code size on Thumb2 archs (depends on how many
calls to gc_realloc are made).
2015-07-14 22:56:32 +01:00
Paul Sokolovsky 8b85d14b92 modsys: Add basic sys.exc_info() implementation.
The implementation is very basic and non-compliant and provided solely for
CPython compatibility. The function itself is bad Python2 heritage, its
usage is discouraged.
2015-04-25 03:49:23 +03:00
Damien George e1e359ff59 py: Put mp_sys_path, mp_sys_argv and gc_collected in mp_state_ctx_t.
Without mp_sys_path and mp_sys_argv in the root pointer section of the
state, their memory was being incorrectly collected by GC.
2015-02-07 17:24:10 +00:00
Damien George 8b77e3dd2f stmhal: Put mod_network_nic_list in global root-pointer state.
It needs to be scanned by GC.  Thanks to Daniel Campora.
2015-01-22 00:16:41 +00:00
Damien George b4b10fd350 py: Put all global state together in state structures.
This patch consolidates all global variables in py/ core into one place,
in a global structure.  Root pointers are all located together to make
GC tracing easier and more efficient.
2015-01-07 20:33:00 +00:00