Calling it from mp_init() is too late for some ports (like Unix), and leads
to incomplete stack frame being captured, with following GC issues. So, now
each port should call mp_stack_ctrl_init() on its own, ASAP after startup,
and taking special precautions so it really was called before stack variables
get allocated (because if such variable with a pointer is missed, it may lead
to over-collecting (typical symptom is segfaulting)).
When using newer glibc's the compiler automatically sets
_FORTIFY_SOURCE when building with -O1 and this causes
a special inlined version of printf to be declared which
then bypasses our version of printf.
Functions added are:
- randint
- randrange
- choice
- random
- uniform
They are enabled with configuration variable
MICROPY_PY_URANDOM_EXTRA_FUNCS, which is disabled by default. It is
enabled for unix coverage build and stmhal.
Seedable and reproducible pseudo-random number generator. Implemented
functions are getrandbits(n) (n <= 32) and seed().
The algorithm used is Yasmarang by Ilya Levin:
http://www.literatecode.com/yasmarang
The first argument to the type.make_new method is naturally a uPy type,
and all uses of this argument cast it directly to a pointer to a type
structure. So it makes sense to just have it a pointer to a type from
the very beginning (and a const pointer at that). This patch makes
such a change, and removes all unnecessary casting to/from mp_obj_t.
This patch changes the type signature of .make_new and .call object method
slots to use size_t for n_args and n_kw (was mp_uint_t. Makes code more
efficient when mp_uint_t is larger than a machine word. Doesn't affect
ports when size_t and mp_uint_t have the same size.
POSIX doesn't guarantee something like that to work, but it works on any
system with careful signal implementation. Roughly, the requirement is
that signal handler is executed in the context of the process, its main
thread, etc. This is true for Linux. Also tested to work without issues
on MacOSX.
This basically introduces the MICROPY_MACHINE_MEM_GET_READ_ADDR
and MICROPY_MACHINE_MEM_GET_WRITE_ADDR macros. If one of them is
not defined, then a default identity function is provided.
To let unix port implement "machine" functionality on Python level, and
keep consistent naming in other ports (baremetal ports will use magic
module "symlinking" to still load it on "import machine").
Fixes#1701.
This solves long-standing non-deterministic bug, which manifested itself
on x86 32-bit (at least of reported cases) - segfault on Ctrl+C (i.e.
SIGINT).
ilistdir() returns iterator which yields triples of (name, type, ino)
where ino is inode number for entry's data, type of entry (file/dir/etc.),
and name of file/dir. listdir() can be easily implemented in terms of this
iterator (which is otherwise more efficient in terms of memory use and may
save expensive call to stat() for each returned entry).
CPython has os.scandir() which also returns an iterator, but it yields
more complex objects of DirEntry type. scandir() can also be easily
implemented in terms of ilistdir().
After an I/O event is triggered for fd, event flags are automatically reset,
so no further events are reported until new event flags are set. This is
an optimization for uasyncio, required to account for coroutine semantics:
each coroutine issues explicit read/write async call, and once that trigger,
no events should be reported to coroutine, unless it again explicitly
requests it. One-shot mode saves one linear scan over the poll array.
Per CPython docs, "Registering a file descriptor that’s already registered
is not an error, and has the same effect as registering the descriptor
exactly once."
https://docs.python.org/3/library/select.html#select.poll.register
That's somewhat ambiguous, what's implemented here is that if fd si not
yet registered, it is registered. Otherwise, the effect is equivalent to
modify() method.
Usually this checking is done by VM on jump instructions, but for linear
sequences of instructions and builtin functions this won't happen. Particular
target of this change is long-running builtin functions like time.sleep().
As set by signal handler. This assumes that exception will be raised
somewhere else, which so far doesn't happen for single function call.
Still, it makes sense to handle that in some common place.
This allows the mp_obj_t type to be configured to something other than a
pointer-sized primitive type.
This patch also includes additional changes to allow the code to compile
when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of
mp_uint_t, and various casts.
THis is required to deal well with signals, signals being the closest
analogue of hardware interrupts for POSIX. This is also CPython 3.5
compliant behavior (PEP 475).
The main problem implementing this is to figure out how much time was
spent in waiting so far/how much is remaining. It's well-known fact that
Linux updates select()'s timeout value when returning with EINTR to the
remaining wait time. Here's what POSIX-based standards say about this:
(http://pubs.opengroup.org/onlinepubs/9699919799/functions/pselect.html):
"Upon successful completion, the select() function may modify the object
pointed to by the timeout argument."
I.e. it allows to modify timeout value, but doesn't say how exactly it is
modified. And actually, it allows such modification only "upon successful
completion", which returning with EINTR error hardly is.
POSIX also allows to request automatic EINTR restart for system calls using
sigaction call with SA_RESTART flag, but here's what the same document says
about it:
"If SA_RESTART has been set for the interrupting signal, it is
implementation-defined whether the function restarts or returns with
[EINTR]."
In other words, POSIX doesn't leave room for both portable and efficient
handling of this matter, so the code just allows to manually select
Linux-compatible behavior with MICROPY_SELECT_REMAINING_TIME option,
or otherwise will just raise OSError. When systems with non-Linux behavior
are found, they can be handled separately.
In other words, unix port now uses overriden printf(), instead of using
libc's. This should remove almost all dependency on libc stdio (which
is bloated).
Return tuple of (address_family, net_addr, [port, [extra_data]]). net_addr
is still raw network address as bytes object, but suitable for passing to
inet_ntop() function. At the very least, sockaddr() will separate address
family value from binary socket address (and currently, only AF_INET family
is decoded).
Now, if we build for an architecture which doesn't have dedicated support
for getting registers for GC scanning, fallback to setjmp-based method
automatically. It's still possible to force setjmp-based implementation
on archs with dedicated support (e.g. for testing, or for peculiar calling
conventions/optimizations).
py/mphal.h contains declarations for generic mp_hal_XXX functions, such
as stdio and delay/ticks, which ports should provide definitions for. A
port will also provide mphalport.h with further HAL declarations.
These MPHAL functions are intended to replace previously used HAL_Delay(),
HAL_GetTick() to provide better naming and MPHAL separation (they are
fully equivalent otherwise).
Also, refactor extmod/modlwip to use them.
This is required to properly select among overloaded methods. It however
relies on java.lang.Object-overloaded method to come last, which appears
to be the case for OpenJDK.
Another function (like stat) which is problematic to deal with on ABI level
(FFI), as struct statvfs layout may differ unpredictably between OSes and
even different versions of a same OS. So, implement it in C, returning a
10-element tuple of f_bsize, f_frsize, f_blocks, f_bfree, f_bavail, f_files,
f_ffree, f_favail, f_flag, f_namemax. This is exactly the order described
in Python3 docs, https://docs.python.org/3/library/os.html#os.statvfs
(but note that os.statvfs() should make these values available as
attributes).
As we dn't export constants for TCSANOW, etc., zero makes a good "don't
care" param, and now it will work also under Android Bionic and any other
libc.
Use CTRL-E to enter paste mode. Prompt starts with "===" and accepts
all characters verbatim, echoing them back. Only control characters are
CTRL-C which cancels the input and returns to normal REPL, and CTRL-D
which ends the input and executes it. The input is executed as though
it were a file. The input is not added to the prompt history.
With this patch parse nodes are allocated sequentially in chunks. This
reduces fragmentation of the heap and prevents waste at the end of
individually allocated parse nodes.
Saves roughly 20% of RAM during parse stage.
This fixes errors like these ones:
modffi.c: In function 'return_ffi_value':
modffi.c:143:29: error: cast to pointer from integer of different size
[-Werror=int-to-pointer-cast]
const char *s = (const char *)val;
^
modffi.c:162:20: error: cast to pointer from integer of different size
[-Werror=int-to-pointer-cast]
return (mp_obj_t)val;
^
modffi.c: In function 'ffifunc_call':
modffi.c:358:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)a;
^
modffi.c:373:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)s;
^
modffi.c:381:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)bufinfo.buf;
^
modffi.c:384:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)p->func;
^
These errors can be highlighted when building micropython from MIPS64
n32 because ffi_arg is 64-bit wide and the pointers on MIPS64 n32 are
32-bit wide, so it's trying to case an integer to a pointer (or
vice-versa) of a different size. We should cast first the pointer (or the
integer) to a pointer sized integer (intptr_t) to fix that problem.
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Linking against local libffi (and other libs in future) is triggered by
"make MICROPY_STANDALONE=1". Before that, dependent libs should be built
with "make deplibs".
Indeed, this flag efectively selects architecture target, and must
consistently apply to all compiles and links, including 3rd-party
libraries, unlike CFLAGS, which have MicroPython-specific setting.
inet_pton supports both ipv4 and ipv6 addresses. Interface is also extensible
for other address families, but underlying libc inet_pton() function isn't
really extensible (e.g., it doesn't return length of binary address, i.e. it's
really hardcoded to AF_INET and AF_INET6). But anyway, on Python side, we could
extend it to support other addresses.
sendto() turns out to be mandatory function to work with UDP. It may seem
that connect(addr) + send() would achieve the same effect, but what connect()
appears to do is to set source address filter on a socket to its argument.
Then everything falls apart: socket sends to a broad-/multi-cast address,
but reply is sent from real peer address, which doesn't match filter set
by connect(), so local socket never sees a reply.
This requires root access. And on recent Linux kernels, with
CONFIG_STRICT_DEVMEM option enabled, only address ranges listed in
/proc/iomem can be accessed. The above compiled-time option can be
however overriden with boot-time option "iomem=relaxed".
This also removed separate read/write paths - there unlikely would
be a case when they're different.
Previous to this patch a call such as list.append(1, 2) would lead to a
seg fault. This is because list.append is a builtin method and the first
argument to such methods is always assumed to have the correct type.
Now, when a builtin method is extracted like this it is wrapped in a
checker object which checks the the type of the first argument before
calling the builtin function.
This feature is contrelled by MICROPY_BUILTIN_METHOD_CHECK_SELF_ARG and
is enabled by default.
See issue #1216.
MicroPython doesn't come with standard library included, so it is important
to be able to easily install needed package in a seamless manner. Bundling
package manager (upip) inside an executable solves this issue.
upip is bundled only with standard executable, not "minimal" or "fast"
builds.
Using MICROPY_PY_SYS_PATH_DEFAULT macro define. A usecase is building a
distribution package, which should not have user home path by default in
sys.path. In such case, MICROPY_PY_SYS_PATH_DEFAULT can be defined on
make command-line (using CFLAGS_EXTRA).
This gets uPy readline working with unix port, with tab completion and
history. GNU readline is still supported, configure using
MICROPY_USE_READLINE variable.
The function and corresponding command-line option are only enabled for
the coverage build. They are used to exercise uPy features that can't
be properly tested by Python scripts.
From https://docs.python.org/3/library/constants.html#NotImplemented :
"Special value which should be returned by the binary special methods
(e.g. __eq__(), __lt__(), __add__(), __rsub__(), etc.) to indicate
that the operation is not implemented with respect to the other type;
may be returned by the in-place binary special methods (e.g. __imul__(),
__iand__(), etc.) for the same purpose. Its truth value is true."
Some people however appear to abuse it to mean "no value" when None is
a legitimate value (don't do that).
The implementation is very basic and non-compliant and provided solely for
CPython compatibility. The function itself is bad Python2 heritage, its
usage is discouraged.
Previous to this patch the printing mechanism was a bit of a tangled
mess. This patch attempts to consolidate printing into one interface.
All (non-debug) printing now uses the mp_print* family of functions,
mainly mp_printf. All these functions take an mp_print_t structure as
their first argument, and this structure defines the printing backend
through the "print_strn" function of said structure.
Printing from the uPy core can reach the platform-defined print code via
two paths: either through mp_sys_stdout_obj (defined pert port) in
conjunction with mp_stream_write; or through the mp_plat_print structure
which uses the MP_PLAT_PRINT_STRN macro to define how string are printed
on the platform. The former is only used when MICROPY_PY_IO is defined.
With this new scheme printing is generally more efficient (less layers
to go through, less arguments to pass), and, given an mp_print_t*
structure, one can call mp_print_str for efficiency instead of
mp_printf("%s", ...). Code size is also reduced by around 200 bytes on
Thumb2 archs.
splitlines() occurs ~179 times in CPython3 standard library, so was
deemed worthy to implement. The method has subtle semantic differences
from just .split("\n"). It is also defined as working for any end-of-line
combination, but this is currently not implemented - it works only with
LF line-endings (which should be OK for text strings on any platforms,
but not OK for bytes).
Given that there's already support for "fixed table" maps, which are
essentially ordered maps, the implementation of OrderedDict just extends
"fixed table" maps by adding an "is ordered" flag and add/remove
operations, and reuses 95% of objdict code, just making methods tolerant
to both dict and OrderedDict.
Some things are missing so far, like CPython-compatible repr and comparison.
OrderedDict is Disabled by default; enabled on unix and stmhal ports.
These allow to fine-tune the compiler to select whether it optimises
tuple assignments of the form a, b = c, d and a, b, c = d, e, f.
Sensible defaults are provided.
This is rarely used feature which takes enough code to implement, so is
controlled by MICROPY_PY_ARRAY_SLICE_ASSIGN config setting, default off.
But otherwise it may be useful, as allows to update arbitrary-sized data
buffers in-place.
Slice is yet to implement, and actually, slice assignment implemented in
such a way that RHS of assignment should be array of the exact same item
typecode as LHS. CPython has it more relaxed, where RHS can be any sequence
of compatible types (e.g. it's possible to assign list of int's to a
bytearray slice).
Overall, when all "slice write" features are implemented, it may cost ~1KB
of code.
The implementation of these functions is very large (order 4k) and they
are rarely used, so we don't enable them by default.
They are however enabled in stmhal and unix, since we have the room.
To enable parsing constants more efficiently, mp_parse should be allowed
to raise an exception, and mp_compile can already raise a MemoryError.
So these functions need to be protected by an nlr push/pop block.
This patch adds that feature in all places. This allows to simplify how
mp_parse and mp_compile are called: they now raise an exception if they
have an error and so explicit checking is not needed anymore.
Native code has GC-heap pointers in it so it must be scanned. But on
unix port memory for native functions is mmap'd, and so it must have
explicit code to scan it for root pointers.
Compiler optimises lookup of module.CONST when enabled (an existing
feature). Disabled by default; enabled for unix, windows, stmhal.
Costs about 100 bytes ROM on stmhal.
This allows to enable mem-info functions in micropython module, even if
MICROPY_MEM_STATS is not enabled. In this case, you get mem_info and
qstr_info but not mem_{total,current,peak}.
GC for unix/windows builds doesn't make use of the bss section anymore,
so we do not need the (sometimes complicated) build features and code related to it
This is a simple optimisation inspired by JITing technology: we cache in
the bytecode (using 1 byte) the offset of the last successful lookup in
a map. This allows us next time round to check in that location in the
hash table (mp_map_t) for the desired entry, and if it's there use that
entry straight away. Otherwise fallback to a normal map lookup.
Works for LOAD_NAME, LOAD_GLOBAL, LOAD_ATTR and STORE_ATTR opcodes.
On a few tests it gives >90% cache hit and greatly improves speed of
code.
Disabled by default. Enabled for unix and stmhal ports.
This patch consolidates all global variables in py/ core into one place,
in a global structure. Root pointers are all located together to make
GC tracing easier and more efficient.
This patch makes MICROPY_PY_BUILTINS_SET compile-time option fully
disable the builtin set object (when set to 0). This includes removing
set constructor/comprehension from the grammar, the compiler and the
emitters. Now, enabling set costs 8168 bytes on unix x64, and 3576
bytes on stmhal.
system() is the basic function to support automation of tasks, so have it
available builtin, for example, for bootstrapping rest of micropython
environment.
