Commit Graph

206 Commits

Author SHA1 Message Date
Jeff Epler f211a090e2 py.mk: Assume we want all C source files in ulab 2020-06-01 08:26:23 -05:00
Jeff Epler 1cc281b6a4 py.mk: Assume we want all C files from ulab 2020-06-01 08:20:23 -05:00
Dan Halbert 180f5c6a94 Merge remote-tracking branch 'adafruit/master' into ringbuf-fixes 2020-04-29 22:11:22 -04:00
Jeff Epler c016ea6f0a ulab: actually update the submodule
PR#2802 missed the submodule update itself.
2020-04-26 10:12:56 -05:00
Dan Halbert 38ec3bc574 further ringbuf cleanup 2020-04-21 17:38:20 -04:00
Jeff Epler 135fb5b887 py.mk: update warning flags needed for ulab 2020-04-14 15:37:36 -05:00
Jeff Epler 96f2288b84 ulab: include new 'extras' source file 2020-03-17 09:33:17 -05:00
Jeff Epler d6342af980 ulab: rename enable macro so it appears in the support matrix 2020-03-17 09:33:03 -05:00
Jeff Epler 39cfe32c34 Update ulab from upstream again 2020-02-27 14:14:05 -06:00
Jeff Epler fa3b9eba92 ulab: Incorporate it 2020-02-27 11:03:03 -06:00
Steve Theodore dd4b0f6e9c
Make all `PYTHON` env vars into `PYTHON3`
make file contained a mix of references to `PYTHON` and `PYTHON3`, and did not build on a fresh install of Ubuntu (under Windows LXSS)
2020-01-11 21:37:54 -08:00
Jeff Epler 201f4648c4 py.mk: Fix race condition building .mo files
By having an order-only dependency on the directory itself, the directory
is sure to be created before the rule to create a .mo file is.

This fixes a low-freqency error on github actions such as

> msgfmt: error while opening "build/genhdr/en_US.mo" for writing: No such file or directory
2019-12-16 15:33:55 -06:00
Dan Halbert 68ae47907c merge from upstream 2019-12-10 21:04:46 -05:00
Dan Halbert 40434d6919 wip 2019-12-05 22:45:53 -05:00
Jeff Epler 238e121236 protocols: Allow them to be (optionally) type-safe
Protocols are nice, but there is no way for C code to verify whether
a type's "protocol" structure actually implements some particular
protocol.  As a result, you can pass an object that implements the
"vfs" protocol to one that expects the "stream" protocol, and the
opposite of awesomeness ensues.

This patch adds an OPTIONAL (but enabled by default) protocol identifier
as the first member of any protocol structure.  This identifier is
simply a unique QSTR chosen by the protocol designer and used by each
protocol implementer.  When checking for protocol support, instead of
just checking whether the object's type has a non-NULL protocol field,
use `mp_proto_get` which implements the protocol check when possible.

The existing protocols are now named:
    protocol_framebuf
    protocol_i2c
    protocol_pin
    protocol_stream
    protocol_spi
    protocol_vfs
(most of these are unused in CP and are just inherited from MP; vfs and
stream are definitely used though)

I did not find any crashing examples, but here's one to give a flavor of what
is improved, using `micropython_coverage`.  Before the change,
the vfs "ioctl" protocol is invoked, and the result is not intelligible
as json (but it could have resulted in a hard fault, potentially):

    >>> import uos, ujson
    >>> u = uos.VfsPosix('/tmp')
    >>> ujson.load(u)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: syntax error in JSON

After the change, the vfs object is correctly detected as not supporting
the stream protocol:
    >>> ujson.load(p)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: stream operation not supported
2019-12-04 09:29:57 -06:00
Dan Halbert ba1b36a800 Uncomment vm.c SUPEROPT (debugging typo); trim a few builds 2019-06-12 13:09:09 -04:00
Dan Halbert 1bb4fccc3b Turn off SUPEROPT on gc.c instead of trying to squueze inline limit so much; reorganize mpconfigboard.mk files 2019-06-12 11:08:22 -04:00
Damiano Mazzella 7549326ceb
Update py.mk 2019-04-05 21:39:44 +02:00
Scott Shawcroft 12c8b00556
Don't build machine class we don't use. 2019-01-14 17:30:01 -08:00
Scott Shawcroft 9d07e95351
Add support for adding release info into adafruit/circuitpython-org
This also changes the build script to python with better output.
2018-11-30 00:30:57 -08:00
Scott Shawcroft 168e23e466
Build refinement to handle warnings and quiet output 2018-11-09 00:11:43 -08:00
Scott Shawcroft de5a9d72dc
Compress all translated strings with Huffman coding.
This saves code space in builds which use link-time optimization.
The optimization drops the untranslated strings and replaces them
with a compressed_string_t struct. It can then be decompressed to
a c string.

Builds without LTO work as well but include both untranslated
strings and compressed strings.

This work could be expanded to include QSTRs and loaded strings if
a compress method is added to C. Its tracked in #531.
2018-08-16 17:40:57 -07:00
Scott Shawcroft 24e53ad591
Rework escaping and fix ESP build. 2018-08-09 15:58:45 -07:00
Scott Shawcroft 933add6cd8
Support internationalisation. 2018-08-07 14:58:57 -07:00
Scott Shawcroft b50f46d4be
Rename to SRC_QSTR_PREPROCESSOR for clarity. 2018-08-02 11:07:22 -07:00
Scott Shawcroft f6d1b63ecd
Handle emitnative.c which is #included into other .c files. 2018-08-02 00:51:53 -07:00
Scott Shawcroft 8cf03d2d00
Speed up QSTR creation by pre-filtering files before pre-processing. 2018-08-02 00:51:52 -07:00
Dan Halbert 7c219600a2 WIP: after merge; before testing 2018-07-11 16:45:30 -04:00
Damien George 7ad04d17da py/mkrules.mk: Regenerate all qstrs when config files change.
A port can define QSTR_GLOBAL_DEPENDENCIES to add extra files.
2018-06-12 13:53:43 +10:00
Damien George 8d82b0edbd extmod: Add VfsPosix filesystem component.
This VFS component allows to mount a host POSIX filesystem within the uPy
VFS sub-system.  All traditional POSIX file access then goes through the
VFS, allowing to sandbox a uPy process to a certain sub-dir of the host
system, as well as mount other filesystem types alongside the host
filesystem.
2018-06-06 14:28:23 +10:00
Roy Hooper 92b1cb5743 move reload exception to reload.c 2018-05-14 17:41:17 -04:00
Damien George ef12a4bd05 py: Refactor how native emitter code is compiled with a file per arch.
Instead of emitnative.c having configuration code for each supported
architecture, and then compiling this file multiple times with different
macros defined, this patch adds a file per architecture with the necessary
code to configure the native emitter.  These files then #include the
emitnative.c file.

This simplifies emitnative.c (which is already very large), and simplifies
the build system because emitnative.c no longer needs special handling for
compilation and qstr extraction.
2018-04-10 15:06:47 +10:00
Damien George 638b860066 extmod/vfs_fat: Merge remaining vfs_fat_misc.c code into vfs_fat.c.
The only function left in vfs_fat_misc.c is fat_vfs_import_stat() which
can logically go into vfs_fat.c, allowing to remove vfs_fat_misc.c.
2018-02-23 17:24:57 +11:00
Damien George 65ef59a9b5 py/py.mk: Remove .. path component from list of extmod files.
This just makes it a bit cleaner in the output of the build process:
instead of "CC ../../py/../extmod/" there is now "CC ../../extmod/".
2018-02-22 12:48:51 +11:00
Damien George 8ca469cae2 py/py.mk: Split list of uPy sources into core and extmod files.
If a port only needs the core files then it can now use the $(PY_CORE_O)
variable instead of $(PY_O).  $(PY_EXTMOD_O) contains the list of extmod
files (including some files from lib/). $(PY_O) retains its original
definition as the list of all object file (including those for frozen code)
and is a convenience variable for ports that want everything.
2018-02-22 12:48:15 +11:00
Paul Sokolovsky 970eedce8f py/objdeque: Implement ucollections.deque type with fixed size.
So far, implements just append() and popleft() methods, required for
a normal queue. Constructor doesn't accept an arbitarry sequence to
initialize from (am empty deque is always created), so an empty tuple
must be passed as such. Only fixed-size deques are supported, so 2nd
argument (size) is required.

There's also an extension to CPython - if True is passed as 3rd argument,
append(), instead of silently overwriting the oldest item on queue
overflow, will throw IndexError. This behavior is desired in many
cases, where queues should store information reliably, instead of
silently losing some items.
2018-02-21 22:39:25 +11:00
Scott Shawcroft 416abe33ed Introduce a long lived section of the heap.
This adapts the allocation process to start from either end of the heap
when searching for free space. The default behavior is identical to the
existing behavior where it starts with the lowest block and looks higher.
Now it can also look from the highest block and lower depending on the
long_lived parameter to gc_alloc. As the heap fills, the two sections may
overlap. When they overlap, a collect may be triggered in order to keep
the long lived section compact. However, free space is always eligable
for each type of allocation.

By starting from either of the end of the heap we have ability to separate
short lived objects from long lived ones. This separation reduces heap
fragmentation because long lived objects are easy to densely pack.

Most objects are short lived initially but may be made long lived when
they are referenced by a type or module. This involves copying the
memory and then letting the collect phase free the old portion.

QSTR pools and chunks are always long lived because they are never freed.

The reallocation, collection and free processes are largely unchanged. They
simply also maintain an index to the highest free block as well as the lowest.
These indices are used to speed up the allocation search until the next collect.

In practice, this change may slightly slow down import statements with the
benefit that memory is much less fragmented afterwards. For example, a test
import into a 20k heap that leaves ~6k free previously had the largest
continuous free space of ~400 bytes. After this change, the largest continuous
free space is over 3400 bytes.
2018-01-24 10:33:46 -08:00
Damien George b25f92160b py/nlr: Factor out common NLR code to macro and generic funcs in nlr.c.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c, along with a helper macro in nlr.h.  This eliminates duplicated
code.
2017-12-28 16:46:30 +11:00
Paul Sokolovsky 096e967aad Revert "py/nlr: Factor out common NLR code to generic functions."
This reverts commit 6a3a742a6c.

The above commit has number of faults starting from the motivation down
to the actual implementation.

1. Faulty implementation.

The original code contained functions like:

NORETURN void nlr_jump(void *val) {
    nlr_buf_t **top_ptr = &MP_STATE_THREAD(nlr_top);
    nlr_buf_t *top = *top_ptr;
...
     __asm volatile (
    "mov    %0, %%edx           \n" // %edx points to nlr_buf
    "mov    28(%%edx), %%esi    \n" // load saved %esi
    "mov    24(%%edx), %%edi    \n" // load saved %edi
    "mov    20(%%edx), %%ebx    \n" // load saved %ebx
    "mov    16(%%edx), %%esp    \n" // load saved %esp
    "mov    12(%%edx), %%ebp    \n" // load saved %ebp
    "mov    8(%%edx), %%eax     \n" // load saved %eip
    "mov    %%eax, (%%esp)      \n" // store saved %eip to stack
    "xor    %%eax, %%eax        \n" // clear return register
    "inc    %%al                \n" // increase to make 1, non-local return
     "ret                        \n" // return
    :                               // output operands
    : "r"(top)                      // input operands
    :                               // clobbered registers
     );
}

Which clearly stated that C-level variable should be a parameter of the
assembly, whcih then moved it into correct register.

Whereas now it's:

NORETURN void nlr_jump_tail(nlr_buf_t *top) {
    (void)top;

    __asm volatile (
    "mov    28(%edx), %esi      \n" // load saved %esi
    "mov    24(%edx), %edi      \n" // load saved %edi
    "mov    20(%edx), %ebx      \n" // load saved %ebx
    "mov    16(%edx), %esp      \n" // load saved %esp
    "mov    12(%edx), %ebp      \n" // load saved %ebp
    "mov    8(%edx), %eax       \n" // load saved %eip
    "mov    %eax, (%esp)        \n" // store saved %eip to stack
    "xor    %eax, %eax          \n" // clear return register
    "inc    %al                 \n" // increase to make 1, non-local return
    "ret                        \n" // return
    );

    for (;;); // needed to silence compiler warning
}

Which just tries to perform operations on a completely random register (edx
in this case). The outcome is the expected: saving the pure random luck of
the compiler putting the right value in the random register above, there's
a crash.

2. Non-critical assessment.

The original commit message says "There is a small overhead introduced
(typically 1 machine instruction)". That machine instruction is a call
if a compiler doesn't perform tail optimization (happens regularly), and
it's 1 instruction only with the broken code shown above, fixing it
requires adding more. With inefficiencies already presented in the NLR
code, the overhead becomes "considerable" (several times more than 1%),
not "small".

The commit message also says "This eliminates duplicated code.". An
obvious way to eliminate duplication would be to factor out common code
to macros, not introduce overhead and breakage like above.

3. Faulty motivation.

All this started with a report of warnings/errors happening for a niche
compiler. It could have been solved in one the direct ways: a) fixing it
just for affected compiler(s); b) rewriting it in proper assembly (like
it was before BTW); c) by not doing anything at all, MICROPY_NLR_SETJMP
exists exactly to address minor-impact cases like thar (where a) or b) are
not applicable). Instead, a backwards "solution" was put forward, leading
to all the issues above.

The best action thus appears to be revert and rework, not trying to work
around what went haywire in the first place.
2017-12-26 19:27:58 +02:00
Damien George 6a3a742a6c py/nlr: Factor out common NLR code to generic functions.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c.  This eliminates duplicated code.

The factoring also allows to make the machine-specific NLR code pure
assembler code, thus allowing nlrthumb.c to use naked function attributes
in the correct way (naked functions can only have basic inline assembler
code in them).

There is a small overhead introduced (typically 1 machine instruction)
because now the generic nlr_jump() must call nlr_jump_tail() rather than
them being one combined function.
2017-12-20 15:42:06 +11:00
Damien George 02d830c035 py: Introduce a Python stack for scoped allocation.
This patch introduces the MICROPY_ENABLE_PYSTACK option (disabled by
default) which enables a "Python stack" that allows to allocate and free
memory in a scoped, or Last-In-First-Out (LIFO) way, similar to alloca().

A new memory allocation API is introduced along with this Py-stack.  It
includes both "local" and "nonlocal" LIFO allocation.  Local allocation is
intended to be equivalent to using alloca(), whereby the same function must
free the memory.  Nonlocal allocation is where another function may free
the memory, so long as it's still LIFO.

Follow-up patches will convert all uses of alloca() and VLA to the new
scoped allocation API.  The old behaviour (using alloca()) will still be
available, but when MICROPY_ENABLE_PYSTACK is enabled then alloca() is no
longer required or used.

The benefits of enabling this option are (or will be once subsequent
patches are made to convert alloca()/VLA):
- Toolchains without alloca() can use this feature to obtain correct and
  efficient scoped memory allocation (compared to using the heap instead
  of alloca(), which is slower).
- Even if alloca() is available, enabling the Py-stack gives slightly more
  efficient use of stack space when calling nested Python functions, due to
  the way that compilers implement alloca().
- Enabling the Py-stack with the stackless mode allows for even more
  efficient stack usage, as well as retaining high performance (because the
  heap is no longer used to build and destroy stackless code states).
- With Py-stack and stackless enabled, Python-calling-Python is no longer
  recursive in the C mp_execute_bytecode function.

The micropython.pystack_use() function is included to measure usage of the
Python stack.
2017-12-11 13:49:09 +11:00
Scott Shawcroft 73c15dcf8b Merge commit 'f869d6b2e339c04469c6c9ea3fb2fabd7bbb2d8c' into nrf2_merge
This is prep for merging in the NRF5 pull request.
2017-10-24 22:31:16 -07:00
Paul Sokolovsky 9355cca610 esp8266: Set DEFPSIZE=1024, MINCACHE=3 for "btree" module.
Defaults of 4096 and 5 respectively are too high to esp8266, causing
out of memory with a database beyond couple of pages.
2017-09-10 13:54:00 +03:00
Dan Halbert ef61b5ecb5 Initial merge of micropython v1.9.2 into circuitpython 2.0.0 (in development) master.
cpx build compiles and loads and works in repl; test suite not run yet
esp8266 not tested yet
2017-08-25 22:17:07 -04:00
Dan Halbert 74cfdeb316 Added ability to freeze multiple directories; freeze neopixel library in cpx build (#199)
Reworked frozen module support: clean up makefiles and handle multiple directories.
Modules to freeze are included as git submodules.

Add neopixel to circuitplayground express build.

Fixes #56
2017-08-23 11:05:59 -07:00
Damien George 7d4a2f773c all: Make use of $(TOP) variable in Makefiles, instead of "..".
$(TOP) is defined in py/mkenv.mk and should be used to refer to the top
level of this repository.
2017-08-11 12:22:19 +10:00
Damien George 1ed3356540 py/py.mk: Make berkeley-db C-defs apply only to relevant source files.
Otherwise they can interfere (eg redefinition of "abort") with other source
files in a given uPy port.
2017-07-24 15:50:47 +10:00
Ville Skyttä ca16c38210 various: Spelling fixes 2017-05-29 11:36:05 +03:00
Damien George b6c7e4b143 all: Use full path name when including mp-readline/timeutils/netutils.
This follows the pattern of how all other headers are now included, and
makes it explicit where the header file comes from.  This patch also
removes -I options from Makefile's that specify the mp-readline/timeutils/
netutils directories, which are no longer needed.
2017-03-31 22:29:39 +11:00
Damien George 6e74d24f30 py: Add micropython.schedule() function and associated runtime code. 2017-03-20 15:20:26 +11:00