Commit Graph

172 Commits

Author SHA1 Message Date
Damien George 749b16174b py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs.
This patch moves the start of the root pointer section in mp_state_ctx_t
so that it skips entries that are not pointers and don't need scanning.

Previously, the start of the root pointer section was at the very beginning
of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t).
This was the original assembler version of the NLR code was hard-coded to
have the nlr_top pointer at the start of this state structure.  But now
that the NLR code is partially written in C there is no longer this
restriction on the location of nlr_top (and a comment to this effect has
been removed in this patch).

So now the root pointer section starts part way through the
mp_state_thread_t structure, after the entries which are not root pointers.

This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER
outside the root pointer section.

Moving non-pointer entries out of the root pointer section helps to make
the GC more precise and should help to prevent some cases of collectable
garbage being kept.

This patch also has a measurable improvement in performance of the
pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of
roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-13 22:53:28 +10:00
Damien George 2a0cbc0d38 py/gc: Update comment now that gc_drain_stack is called gc_mark_subtree. 2018-02-19 16:08:20 +11:00
Ayke van Laethem 736faef223 py/gc: Make GC stack pointer a local variable.
This saves a bit in code size, and saves some precious .bss RAM:

                 .text  .bss
minimal CROSS=1: -28    -4
unix (64-bit):   -64    -8
2018-02-19 16:05:46 +11:00
Ayke van Laethem 5c9e5618e0 py/gc: Rename gc_drain_stack to gc_mark_subtree and pass it first block.
This saves a bit in code size:

minimal CROSS=1: -44
unix:            -96
2018-02-19 16:00:59 +11:00
Ayke van Laethem ea7cf2b738 py/gc: Reduce code size by specialising VERIFY_MARK_AND_PUSH macro.
This macro is written out explicitly in the two locations that it is used
and then the code is optimised, opening possibilities for further
optimisations and reducing code size:

unix:            -48
minimal CROSS=1: -32
stm32:           -32
2018-02-19 15:58:49 +11:00
Scott Shawcroft aa0ce98b3e Fix the initial state and polish a couple comments. 2018-01-24 14:13:26 -08:00
Scott Shawcroft 416abe33ed Introduce a long lived section of the heap.
This adapts the allocation process to start from either end of the heap
when searching for free space. The default behavior is identical to the
existing behavior where it starts with the lowest block and looks higher.
Now it can also look from the highest block and lower depending on the
long_lived parameter to gc_alloc. As the heap fills, the two sections may
overlap. When they overlap, a collect may be triggered in order to keep
the long lived section compact. However, free space is always eligable
for each type of allocation.

By starting from either of the end of the heap we have ability to separate
short lived objects from long lived ones. This separation reduces heap
fragmentation because long lived objects are easy to densely pack.

Most objects are short lived initially but may be made long lived when
they are referenced by a type or module. This involves copying the
memory and then letting the collect phase free the old portion.

QSTR pools and chunks are always long lived because they are never freed.

The reallocation, collection and free processes are largely unchanged. They
simply also maintain an index to the highest free block as well as the lowest.
These indices are used to speed up the allocation search until the next collect.

In practice, this change may slightly slow down import statements with the
benefit that memory is much less fragmented afterwards. For example, a test
import into a 20k heap that leaves ~6k free previously had the largest
continuous free space of ~400 bytes. After this change, the largest continuous
free space is over 3400 bytes.
2018-01-24 10:33:46 -08:00
Damien George 02d830c035 py: Introduce a Python stack for scoped allocation.
This patch introduces the MICROPY_ENABLE_PYSTACK option (disabled by
default) which enables a "Python stack" that allows to allocate and free
memory in a scoped, or Last-In-First-Out (LIFO) way, similar to alloca().

A new memory allocation API is introduced along with this Py-stack.  It
includes both "local" and "nonlocal" LIFO allocation.  Local allocation is
intended to be equivalent to using alloca(), whereby the same function must
free the memory.  Nonlocal allocation is where another function may free
the memory, so long as it's still LIFO.

Follow-up patches will convert all uses of alloca() and VLA to the new
scoped allocation API.  The old behaviour (using alloca()) will still be
available, but when MICROPY_ENABLE_PYSTACK is enabled then alloca() is no
longer required or used.

The benefits of enabling this option are (or will be once subsequent
patches are made to convert alloca()/VLA):
- Toolchains without alloca() can use this feature to obtain correct and
  efficient scoped memory allocation (compared to using the heap instead
  of alloca(), which is slower).
- Even if alloca() is available, enabling the Py-stack gives slightly more
  efficient use of stack space when calling nested Python functions, due to
  the way that compilers implement alloca().
- Enabling the Py-stack with the stackless mode allows for even more
  efficient stack usage, as well as retaining high performance (because the
  heap is no longer used to build and destroy stackless code states).
- With Py-stack and stackless enabled, Python-calling-Python is no longer
  recursive in the C mp_execute_bytecode function.

The micropython.pystack_use() function is included to measure usage of the
Python stack.
2017-12-11 13:49:09 +11:00
Paul Sokolovsky dea3fb93c7 py/gc: In sweep debug output, print pointer as a pointer.
Or it will be truncated on a 64-bit platform.
2017-12-09 01:54:01 +02:00
Paul Sokolovsky 5453d88d5d py/gc: Factor out a macro to trace GC mark operations.
To allow easier override it for custom tracing.
2017-12-09 01:48:26 +02:00
Paul Sokolovsky 9ef4be8b41 py/gc: Add CLEAR_ON_SWEEP option to debug mis-traced objects.
Accessing them will crash immediately instead still working for some time,
until overwritten by some other data, leading to much less deterministic
crashes.
2017-12-08 00:10:44 +02:00
Damien George 74fad3536b py/gc: In gc_realloc, convert pointer sanity checks to assertions.
These checks are assumed to be true in all cases where gc_realloc is
called with a valid pointer, so no need to waste code space and time
checking them in a non-debug build.
2017-11-29 17:17:08 +11:00
Scott Shawcroft 73c15dcf8b Merge commit 'f869d6b2e339c04469c6c9ea3fb2fabd7bbb2d8c' into nrf2_merge
This is prep for merging in the NRF5 pull request.
2017-10-24 22:31:16 -07:00
Damien George a3dc1b1957 all: Remove inclusion of internal py header files.
Header files that are considered internal to the py core and should not
normally be included directly are:
    py/nlr.h - internal nlr configuration and declarations
    py/bc0.h - contains bytecode macro definitions
    py/runtime0.h - contains basic runtime enums

Instead, the top-level header files to include are one of:
    py/obj.h - includes runtime0.h and defines everything to use the
        mp_obj_t type
    py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
        and defines everything to use the general runtime support functions

Additional, specific headers (eg py/objlist.h) can be included if needed.
2017-10-04 12:37:50 +11:00
Dan Halbert ef61b5ecb5 Initial merge of micropython v1.9.2 into circuitpython 2.0.0 (in development) master.
cpx build compiles and loads and works in repl; test suite not run yet
esp8266 not tested yet
2017-08-25 22:17:07 -04:00
Stefan Naumann ace9fb5405 py: Add verbose debug compile-time flag MICROPY_DEBUG_VERBOSE.
It enables all the DEBUG_printf outputs in the py/ source code.
2017-08-15 11:53:36 +10:00
Alexander Steffen 55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Damien George 12d4fa9b37 py/gc: Refactor assertions in gc_free function.
gc_free() expects either NULL or a valid pointer into the heap, so the
checks for a valid pointer can be turned into assertions.
2017-07-12 12:17:38 +10:00
Scott Shawcroft 30ee7019ca Merge tag 'v1.9.1'
Fixes for stmhal USB mass storage, lwIP bindings and VFS regressions

This release provides an important fix for the USB mass storage device in
the stmhal port by implementing the SCSI SYNCHRONIZE_CACHE command, which
is now require by some Operating Systems.  There are also fixes for the
lwIP bindings to improve non-blocking sockets and error codes.  The VFS has
some regressions fixed including the ability to statvfs the root.

All changes are listed below.

py core:
- modbuiltins: add core-provided version of input() function
- objstr: catch case of negative "maxsplit" arg to str.rsplit()
- persistentcode: allow to compile with complex numbers disabled
- objstr: allow to compile with obj-repr D, and unicode disabled
- modsys: allow to compile with obj-repr D and PY_ATTRTUPLE disabled
- provide mp_decode_uint_skip() to help reduce stack usage
- makeqstrdefs.py: make script run correctly with Python 2.6
- objstringio: if created from immutable object, follow copy on write policy

extmod:
- modlwip: connect: for non-blocking mode, return EINPROGRESS
- modlwip: fix error codes for duplicate calls to connect()
- modlwip: accept: fix error code for non-blocking mode
- vfs: allow to statvfs the root directory
- vfs: allow "buffering" and "encoding" args to VFS's open()
- modframebuf: fix signed/unsigned comparison pendantic warning

lib:
- libm: use isfinite instead of finitef, for C99 compatibility
- utils/interrupt_char: remove support for KBD_EXCEPTION disabled

tests:
- basics/string_rsplit: add tests for negative "maxsplit" argument
- float: convert "sys.exit()" to "raise SystemExit"
- float/builtin_float_minmax: PEP8 fixes
- basics: convert "sys.exit()" to "raise SystemExit"
- convert remaining "sys.exit()" to "raise SystemExit"

unix port:
- convert to use core-provided version of built-in import()
- Makefile: replace references to make with $(MAKE)

windows port:
- convert to use core-provided version of built-in import()

qemu-arm port:
- Makefile: adjust object-file lists to get correct dependencies
- enable micropython.mem_*() functions to allow more tests

stmhal port:
- boards: enable DAC for NUCLEO_F767ZI board
- add support for NUCLEO_F446RE board
- pass USB handler as parameter to allow more than one USB handler
- usb: use local USB handler variable in Start-of-Frame handler
- usb: make state for USB device private to top-level USB driver
- usbdev: for MSC implement SCSI SYNCHRONIZE_CACHE command
- convert from using stmhal's input() to core provided version

cc3200 port:
- convert from using stmhal's input() to core provided version

teensy port:
- convert from using stmhal's input() to core provided version

esp8266 port:
- Makefile: replace references to make with $(MAKE)
- Makefile: add clean-modules target
- convert from using stmhal's input() to core provided version

zephyr port:
- modusocket: getaddrinfo: Fix mp_obj_len() usage
- define MICROPY_PY_SYS_PLATFORM (to "zephyr")
- machine_pin: use native Zephyr types for Zephyr API calls

docs:
- machine.Pin: remove out_value() method
- machine.Pin: add on() and off() methods
- esp8266: consistently replace Pin.high/low methods with .on/off
- esp8266/quickref: polish Pin.on()/off() examples
- network: move confusingly-named cc3200 Server class to its reference
- uos: deconditionalize, remove minor port-specific details
- uos: move cc3200 port legacy VFS mounting functions to its ref doc
- machine: sort machine classes in logical order, not alphabetically
- network: first step to describe standard network class interface

examples:
- embedding: use core-provided KeyboardInterrupt object
2017-06-20 10:56:05 -07:00
Scott Shawcroft c6d539ace3 atmel-samd: Fix sporadic "syntax errors"
The GC was deleting memory that was in use because its scan of the
stack missed the very top. Switching to _estack fixes this by relying
on the location from the linker.

Fixes #124
2017-05-05 12:04:20 -07:00
Damien George c7e8c6f7de py/gc: Execute finaliser code in a protected environment.
If a finaliser raises an exception then it must not propagate through the
GC sweep function.  This patch protects against such a thing by running
finaliser code via the mp_call_function_1_protected call.

This patch also adds scheduler lock/unlock calls around the finaliser
execution to further protect against any possible reentrancy issues: the
memory manager is already locked when doing a collection, but we also don't
want to allow any scheduled code to run, KeyboardInterrupts to interupt the
code, nor threads to switch.
2017-04-12 13:52:04 +10:00
Scott Shawcroft 05c14dfe78 Revert "py: Add ability to manually mark blocks during collect."
This reverts commit 9321828158.
2016-11-29 14:11:32 -08:00
Scott Shawcroft a8fbad5d8b Add heap analysis scripts based on GDB breakpoint logs. 2016-11-23 17:31:53 -08:00
Scott Shawcroft ccbb5e84f9 This introduces an alternative hardware API called nativeio structured around different functions that are typically accelerated by native hardware. Its not meant to reflect the structure of the hardware.
Docs are here: http://tannewt-micropython.readthedocs.io/en/microcontroller/

It differs from upstream's machine in the following ways:

* Python API is identical across ports due to code structure. (Lives in shared-bindings)
* Focuses on abstracting common functionality (AnalogIn) and not representing structure (ADC).
* Documentation lives with code making it easy to ensure they match.
* Pin is split into references (board.D13 and microcontroller.pin.PA17) and functionality (DigitalInOut).
* All nativeio classes claim underlying hardware resources when inited on construction, support Context Managers (aka with statements) and have deinit methods which release the claimed hardware.
* All constructors take pin references rather than peripheral ids. Its up to the implementation to find hardware or throw and exception.
2016-11-21 14:11:52 -08:00
Scott Shawcroft 9321828158 py: Add ability to manually mark blocks during collect. 2016-11-21 14:11:49 -08:00
Damien George 5ffe1d8dc0 py/gc: Add MICROPY_GC_CONSERVATIVE_CLEAR option to always zero memory.
There can be stray pointers in memory blocks that are not properly zero'd
after allocation.  This patch adds a new config option to always zero all
allocated memory (via gc_alloc and gc_realloc) and hence help to eliminate
stray pointers.

See issue #2195.
2016-08-26 15:35:26 +10:00
Paul Sokolovsky 93e353e384 py/gc: Implement GC running by allocation threshold.
Currently, MicroPython runs GC when it could not allocate a block of memory,
which happens when heap is exhausted. However, that policy can't work well
with "inifinity" heaps, e.g. backed by a virtual memory - there will be a
lot of swap thrashing long before VM will be exhausted. Instead, in such
cases "allocation threshold" policy is used: a GC is run after some number of
allocations have been made. Details vary, for example, number or total amount
of allocations can be used, threshold may be self-adjusting based on GC
outcome, etc.

This change implements a simple variant of such policy for MicroPython. Amount
of allocated memory so far is used for threshold, to make it useful to typical
finite-size, and small, heaps as used with MicroPython ports. And such GC policy
is indeed useful for such types of heaps too, as it allows to better control
fragmentation. For example, if a threshold is set to half size of heap, then
for an application which usually makes big number of small allocations, that
will (try to) keep half of heap memory in a nice defragmented state for an
occasional large allocation.

For an application which doesn't exhibit such behavior, there won't be any
visible effects, except for GC running more frequently, which however may
affect performance. To address this, the GC threshold is configurable, and
by default is off so far. It's configured with gc.threshold(amount_in_bytes)
call (can be queries without an argument).
2016-07-21 00:37:30 +03:00
Paul Sokolovsky 749cbaca7f py/gc: Calculate (and report) maximum contiguous free block size.
Just as maximum allocated block size, it's reported in allocation units
(not bytes).
2016-07-01 00:09:55 +03:00
Paul Sokolovsky 6a6e0b7e05 py/gc: Be sure to count last allocated block at heap end in stats.
Previously, if there was chain of allocated blocks ending with the last
block of heap, it wasn't included in number of 1/2-block or max block
size stats.
2016-06-30 12:56:21 +03:00
Damien George a1c93a62b1 py: Don't use gc or qstr mutex when the GIL is enabled.
There is no need since the GIL already makes gc and qstr operations
atomic.
2016-06-28 11:28:50 +01:00
Damien George 3653f5144a py/gc: Fix GC+thread bug where ptr gets lost because it's not computed.
GC_EXIT() can cause a pending thread (waiting on the mutex) to be
scheduled right away.  This other thread may trigger a garbage
collection.  If the pointer to the newly-allocated block (allocated by
the original thread) is not computed before the switch (so it's just left
as a block number) then the block will be wrongly reclaimed.

This patch makes sure the pointer is computed before allowing any thread
switch to occur.
2016-06-28 11:28:49 +01:00
Damien George e33806aaff py/gc: Fix 2 cases of concurrent access to ATB and FTB. 2016-06-28 11:28:49 +01:00
Damien George c93d9caa8b py/gc: Make memory manager and garbage collector thread safe.
By using a single, global mutex, all memory-related functions (alloc,
free, realloc, collect, etc) are made thread safe.  This means that only
one thread can be in such a function at any one time.
2016-06-28 11:28:49 +01:00
Damien George 330165a2cc py: Add MP_STATE_THREAD to hold state specific to a given thread. 2016-06-28 11:09:31 +01:00
Paul Sokolovsky 68a7a92cec py/gc: gc_dump_alloc_table(): Dump heap offset instead of actual address.
Address printed was truncated anyway and in general confusing to outsider.
A line which dumps it is still left in the source, commented, for peculiar
cases when it may be needed (e.g. when running under debugger).
2016-05-13 00:16:38 +03:00
Paul Sokolovsky 9a8751b006 gc: gc_dump_alloc_table(): Use '=' char for tail blocks.
'=' is pretty natural character for tail, and gives less dense picture
where it's easier to see what object types are actually there.
2016-05-13 00:16:38 +03:00
Paul Sokolovsky bc04dc277e py/gc: Make (byte)array type dumping conditional on these types being enabled. 2016-05-11 19:21:53 +03:00
Paul Sokolovsky 3d7f3f00e0 py/gc: gc_dump_alloc_table(): Show byte/str and (byte)array objects.
These are typical consumers of large chunks of memory, so it's useful to
see at least their number (how much memory isn't clearly shown, as the data
for these objects is allocated elsewhere).
2016-05-11 19:00:15 +03:00
Paul Sokolovsky 3ea03a1188 py/gc: Improve mark/sweep debug output.
Previously, mark operation weren't logged at all, while it's quite useful
to see cascade of marks in case of over-marking (and in other cases too).
Previously, sweep was logged for each block of object in memory, but that
doesn't make much sense and just lead to longer output, harder to parse
by a human. Instead, log sweep only once per object. This is similar to
other memory manager operations, e.g. an object is allocated, then freed.
Or object is allocated, then marked, otherwise swept (one log entry per
operation, with the same memory address in each case).
2015-12-27 20:40:36 +02:00
Damien George acaccb37ec py/gc: When printing info, use %u instead of UINT_FMT for size_t args.
Ideally we'd use %zu for size_t args, but that's unlikely to be supported
by all runtimes, and we would then need to implement it in mp_printf.
So simplest and most portable option is to use %u and cast the argument
to uint(=unsigned int).

Note: reason for the change is that UINT_FMT can be %llu (size suitable
for mp_uint_t) which is wider than size_t and prints incorrect results.
2015-12-18 12:52:45 +00:00
Damien George d977d268e8 py/gc: Use size_t instead of mp_uint_t to count things related to heap.
size_t is the correct type to use to count things related to the size of
the address space.  Using size_t (instead of mp_uint_t) is important for
the efficiency of ports that configure mp_uint_t to larger than the
machine word size.
2015-12-16 20:09:11 -05:00
Damien George f7782f8082 py/gc: For finaliser, interpret a pointer into the heap as concrete obj. 2015-12-16 19:45:42 -05:00
Damien George 969e4bbe6a py/gc: Scan GC blocks as an array of pointers, not an array of objects.
The GC should search for pointers within the heap.  This patch makes a
difference when an object is larger than a pointer (eg 64-bit NaN
boxing).
2015-12-16 19:41:37 -05:00
Paul Sokolovsky 75feece208 py/gc: Make GC block size be configurable. 2015-12-03 01:40:52 +02:00
Damien George 999cedb90f py: Wrap all obj-ptr conversions in MP_OBJ_TO_PTR/MP_OBJ_FROM_PTR.
This allows the mp_obj_t type to be configured to something other than a
pointer-sized primitive type.

This patch also includes additional changes to allow the code to compile
when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of
mp_uint_t, and various casts.
2015-11-29 14:25:35 +00:00
Damien George 94fe6e523d py/gc: Move away from using mp_uint_t, instead use uintptr_t and size_t.
The GC works with concrete pointers and so the types should reflect this.
2015-11-29 14:25:04 +00:00
Dave Hylands 7f3c0d1ea8 py: Clear finalizer flag when calling gc_free.
Currently, the only place that clears the bit is in gc_collect.
So if a block with a finalizer is allocated, and subsequently
freed, and then the block is reallocated with no finalizer then
the bit remains set.

This could also be fixed by having gc_alloc clear the bit, but
I'm pretty sure that free is called way less than alloc, so doing
it in free is more efficient.
2015-11-07 14:26:11 +00:00
Damien George 3a2171e406 py: Eliminate some cases which trigger unused parameter warnings. 2015-09-04 16:53:46 +01:00
Damien George ade9a05236 py: Improve allocation policy of qstr data.
Previous to this patch all interned strings lived in their own malloc'd
chunk.  On average this wastes N/2 bytes per interned string, where N is
the number-of-bytes for a quanta of the memory allocator (16 bytes on 32
bit archs).

With this patch interned strings are concatenated into the same malloc'd
chunk when possible.  Such chunks are enlarged inplace when possible,
and shrunk to fit when a new chunk is needed.

RAM savings with this patch are highly varied, but should always show an
improvement (unless only 3 or 4 strings are interned).  New version
typically uses about 70% of previous memory for the qstr data, and can
lead to savings of around 10% of total memory footprint of a running
script.

Costs about 120 bytes code size on Thumb2 archs (depends on how many
calls to gc_realloc are made).
2015-07-14 22:56:32 +01:00
Damien George e72cda99fd py: Convert occurrences of non-debug printf to mp_printf. 2015-04-16 14:30:16 +00:00
Damien George 12ab9eda8d py: Make heap printing compatible with 16-bit word size. 2015-04-03 14:11:13 +01:00
Damien George e1e359ff59 py: Put mp_sys_path, mp_sys_argv and gc_collected in mp_state_ctx_t.
Without mp_sys_path and mp_sys_argv in the root pointer section of the
state, their memory was being incorrectly collected by GC.
2015-02-07 17:24:10 +00:00
Damien George abc1959e2c py, unix, lib: Allow to compile with -Wold-style-definition. 2015-01-12 22:34:38 +00:00
Damien George ec21405821 py: Add (commented out) code to gc_dump_alloc_table for qstr info. 2015-01-11 14:37:06 +00:00
stijn afd6c8e1d2 Remove obsolete bss-related code/build features
GC for unix/windows builds doesn't make use of the bss section anymore,
so we do not need the (sometimes complicated) build features and code related to it
2015-01-08 15:29:44 +01:00
Damien George b4b10fd350 py: Put all global state together in state structures.
This patch consolidates all global variables in py/ core into one place,
in a global structure.  Root pointers are all located together to make
GC tracing easier and more efficient.
2015-01-07 20:33:00 +00:00
Damien George fd40a9c38e py: Make GC's STACK_SIZE definition a proper MICROPY_ config variable. 2015-01-01 22:04:46 +00:00
Damien George 51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Damien George 7860c2a68a py: Fix some macros defines; cleanup some includes. 2014-11-05 21:16:41 +00:00
Damien George 109c1de015 py: Make gc.enable/disable just control auto-GC; alloc is still allowed.
gc.enable/disable are now the same as CPython: they just control whether
automatic garbage collection is enabled or not.  If disabled, you can
still allocate heap memory, and initiate a manual collection.
2014-10-31 21:30:46 +00:00
Damien George 0b13f3e026 py: Improve memory usage debugging; better GC AT dumping.
In unix port, mem_info(1) now prints pretty GC alloc table.
2014-10-24 23:12:25 +01:00
Damien George e7bb0443cd py: Properly free string parse-node; add assertion to gc_free. 2014-10-23 14:13:05 +01:00
Damien George 37378f8a9d py: Clean up edge cases of malloc/realloc/free. 2014-10-23 12:02:00 +01:00
Damien George c30595eb1b py: Add more debug printing code in gc_dump_alloc_table. 2014-10-17 14:12:57 +00:00
Damien George 37ada236b3 py: Take gc_pool_start out of bss section, to reclaim 1st block of heap. 2014-10-16 21:50:39 +01:00
Damien George 9b0b373e5e py: Fix GC realloc issue, where memory chunks were never shrunk.
Previously, a realloc to a smaller memory chunk size would not free the
unused blocks in the tail of the chunk.
2014-10-15 18:24:47 +00:00
Damien George 516b09efc3 py, gc: Further reduce heap fragmentation with new, faster gc alloc.
The heap allocation is now exactly as it was before the "faster gc
alloc" patch, but it's still nearly as fast.  It is fixed by being
careful to always update the "last free block" pointer whenever the heap
changes (eg free or realloc).

Tested on all tests by enabling EXTENSIVE_HEAP_PROFILING in py/gc.c:
old and new allocator have exactly the same behaviour, just the new one
is much faster.
2014-08-28 23:06:38 +01:00
Damien George b796e3d848 py: Reduce fragmentation of GC heap.
Recent speed up of GC allocation made the GC have a fragmented heap.
This patch restores "original fragmentation behaviour" whilst still
retaining relatively fast allocation.  This patch works because there is
always going to be a single block allocated now and then, which advances
the gc_last_free_atb_index pointer often enough so that the whole heap
doesn't need scanning.

Should address issue #836.
2014-08-28 10:18:40 +01:00
Damien George d5e7f6e37e py: Speed up GC allocation.
This simple patch gives a very significant speed up for memory allocation
with the GC.

Eg, on PYBv1.0:
tests/basics/dict_del.py: 3.55 seconds -> 1.19 seconds
tests/misc/rge_sm.py:     15.3 seconds -> 2.48 seconds
2014-08-22 18:17:02 +01:00
Damien George a1d3ee376c py: Fix bug where GC finaliser table was not completely zeroed out.
This was a nasty bug to track down.  It only had consequences when the
heap size was just the right size to expose the rounding error in the
calculation of the finaliser table size.  And, a script had to allocate
a small (1 or 2 cell) object at the very end of the heap.  And, this
object must not have a finaliser.  And, the initial state of the heap
must have been all bits set to 1.  All these conspire on the pyboard,
but only if your run the script fresh (so unused memory is all 1's),
and if your script allocates a lot of small objects (eg 2-char strings
that are not interned).
2014-08-08 12:33:49 +01:00
Damien George 40f3c02682 Rename machine_(u)int_t to mp_(u)int_t.
See discussion in issue #50.
2014-07-03 13:25:24 +01:00
Dave Hylands 2fe841d2fa Try not to cause a MemoryError when raising an exception during nterrupt handling.
Step 1 fixes #732
2014-06-30 22:49:21 -07:00
Paul Sokolovsky 59c675a64c py: Include mpconfig.h before all other includes.
It defines types used by all other headers.

Fixes #691.
2014-06-21 22:43:22 +03:00
stijn 9acb5e4cf0 gc: Turn off debugging info again 2014-06-18 12:29:03 +02:00
stijn def10cecd1 gc: Keep debug statements at beginning of scope where possible 2014-06-18 10:20:41 +02:00
stijn bbcea3f62b gc: More verbose debugging
Add more DEBUG_printf statements to trace gc behaviour
2014-06-16 12:43:35 +02:00
Damien George c037694957 py, gc: Revert ret_ptr to void*, casting to byte* for memset. 2014-06-13 22:33:31 +01:00
stijn f33385f56d gc: Use byte* pointers instead of void* for pointer arithmetic
void* is of unknown size
2014-06-13 20:42:06 +02:00
Paul Sokolovsky 755a55f507 modgc: Implement return value for gc.collect(), enable on Unix. 2014-06-05 22:48:02 +03:00
Damien George 0fb80c303a py: Compress a little the bytecode emitter structure. 2014-05-10 18:16:21 +01:00
Damien George 04b9147e15 Add license header to (almost) all files.
Blanket wide to all .c and .h files.  Some files originating from ST are
difficult to deal with (license wise) so it was left out of those.

Also merged modpyb.h, modos.h, modstm.h and modtime.h in stmhal/.
2014-05-03 23:27:38 +01:00
Damien George 32bef315be py, gc: Only zero out the extra bytes at the end of the heap chunk.
This is a small optimisation to zero out only the additional bytes that
the caller did not ask for.
2014-04-26 22:23:42 +01:00
Damien George c492cf1f44 Merge branch 'master' of github.com:micropython/micropython 2014-04-25 23:45:52 +01:00
Damien George daab651c5c py, gc: Zero out newly allocated blocks in the GC.
Also add some more debugging output to gc_dump_alloc_table().

Now that newly allocated heap is always zero'd, maybe we just make this
a policy for the uPy API to keep it simple (ie any new implementation of
memory allocation must zero all allocations).  This follows the D
language philosophy.

Before this patch, a previously used memory block which had pointers in
it may still retain those pointers if the new user of that block does
not actually use the entire block.  Eg, if I want 5 blocks worth of
heap, I actually get 8 (round up to nearest 4).  Then I never use the
last 3, so they keep their old values, which may be pointers pointing to
the heap, hence preventing GC.

In rare (or maybe not that rare) cases, this leads to long, unintentional
"linked lists" within the GC'd heap, filling it up completely.  It's
pretty rare, because you have to reuse exactly that memory which is part
of this "linked list", and reuse it in just the right way.

This should fix issue #522, and might have something to do with
issue #510.
2014-04-25 23:37:55 +01:00
Damien George 410f30772f py, gc: Fix old gc_realloc for case when NULL is passed in as ptr. 2014-04-25 11:44:53 +00:00
Paul Sokolovsky 5b991ae2d3 gc: gc_realloc(): Fix byte-to-block calculation. 2014-04-20 20:46:39 +03:00
Damien George dde739d364 py, gc: Further simplify coding-style of gc_realloc.
No logic changes, just coding style to make it easy to read.
2014-04-20 18:16:25 +01:00
Paul Sokolovsky c86889dafb gc: "new" gc_realloc: Rewrite in plain C, fixing bunch of bugs.
There were typos, various rounding errors trying to do concurrent counting
in bytes vs blocks, complex conditional paths, superfluous variables, etc.,
etc., all leading to obscure segfaults.
2014-04-20 13:08:33 +03:00
Paul Sokolovsky ed162b5ef2 gc: Recover simple gc_realloc implementation, make easier to switch between. 2014-04-20 13:08:33 +03:00
Paul Sokolovsky 03b9ad7b01 gc.c: Remove superfluous typedef (bute defined in misc.h). 2014-04-09 04:13:21 +03:00
Damien George 443e018a3f py: Improve GC locking/unlocking, and make it part of the API. 2014-04-08 11:31:21 +00:00
Damien George 12bab72d93 Improve GC finalisation code; add option to disable it. 2014-04-05 20:35:48 +01:00
Damien George 8123a3339d Merge pull request #425 from iabdalkader/del
Implement del
2014-04-05 18:49:39 +01:00
mux cc849f70f4 Move del to locals 2014-04-05 15:49:03 +02:00
mux 4f7e9f5c44 Implement del 2014-04-03 23:55:12 +02:00
Paul Sokolovsky e807fa8d60 gc: Uses uint defined in misc.h. 2014-04-02 20:36:32 +03:00
Damien George 70f33cde48 py: Fix up so that it can compile without float. 2014-04-02 17:06:05 +01:00
xbe efe3422394 py: Clean up includes.
Remove unnecessary includes. Add includes that improve portability.
2014-03-17 02:43:40 -07:00
Damien George 470184e2c3 py: Cosmetic changes. 2014-03-12 22:44:11 +00:00
mux 8782676514 Fix realloc 2014-03-12 21:00:23 +02:00