ilistdir() returns iterator which yields triples of (name, type, ino)
where ino is inode number for entry's data, type of entry (file/dir/etc.),
and name of file/dir. listdir() can be easily implemented in terms of this
iterator (which is otherwise more efficient in terms of memory use and may
save expensive call to stat() for each returned entry).
CPython has os.scandir() which also returns an iterator, but it yields
more complex objects of DirEntry type. scandir() can also be easily
implemented in terms of ilistdir().
After an I/O event is triggered for fd, event flags are automatically reset,
so no further events are reported until new event flags are set. This is
an optimization for uasyncio, required to account for coroutine semantics:
each coroutine issues explicit read/write async call, and once that trigger,
no events should be reported to coroutine, unless it again explicitly
requests it. One-shot mode saves one linear scan over the poll array.
Per CPython docs, "Registering a file descriptor that’s already registered
is not an error, and has the same effect as registering the descriptor
exactly once."
https://docs.python.org/3/library/select.html#select.poll.register
That's somewhat ambiguous, what's implemented here is that if fd si not
yet registered, it is registered. Otherwise, the effect is equivalent to
modify() method.
Usually this checking is done by VM on jump instructions, but for linear
sequences of instructions and builtin functions this won't happen. Particular
target of this change is long-running builtin functions like time.sleep().
As set by signal handler. This assumes that exception will be raised
somewhere else, which so far doesn't happen for single function call.
Still, it makes sense to handle that in some common place.
This allows the mp_obj_t type to be configured to something other than a
pointer-sized primitive type.
This patch also includes additional changes to allow the code to compile
when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of
mp_uint_t, and various casts.
THis is required to deal well with signals, signals being the closest
analogue of hardware interrupts for POSIX. This is also CPython 3.5
compliant behavior (PEP 475).
The main problem implementing this is to figure out how much time was
spent in waiting so far/how much is remaining. It's well-known fact that
Linux updates select()'s timeout value when returning with EINTR to the
remaining wait time. Here's what POSIX-based standards say about this:
(http://pubs.opengroup.org/onlinepubs/9699919799/functions/pselect.html):
"Upon successful completion, the select() function may modify the object
pointed to by the timeout argument."
I.e. it allows to modify timeout value, but doesn't say how exactly it is
modified. And actually, it allows such modification only "upon successful
completion", which returning with EINTR error hardly is.
POSIX also allows to request automatic EINTR restart for system calls using
sigaction call with SA_RESTART flag, but here's what the same document says
about it:
"If SA_RESTART has been set for the interrupting signal, it is
implementation-defined whether the function restarts or returns with
[EINTR]."
In other words, POSIX doesn't leave room for both portable and efficient
handling of this matter, so the code just allows to manually select
Linux-compatible behavior with MICROPY_SELECT_REMAINING_TIME option,
or otherwise will just raise OSError. When systems with non-Linux behavior
are found, they can be handled separately.
In other words, unix port now uses overriden printf(), instead of using
libc's. This should remove almost all dependency on libc stdio (which
is bloated).
Return tuple of (address_family, net_addr, [port, [extra_data]]). net_addr
is still raw network address as bytes object, but suitable for passing to
inet_ntop() function. At the very least, sockaddr() will separate address
family value from binary socket address (and currently, only AF_INET family
is decoded).
Now, if we build for an architecture which doesn't have dedicated support
for getting registers for GC scanning, fallback to setjmp-based method
automatically. It's still possible to force setjmp-based implementation
on archs with dedicated support (e.g. for testing, or for peculiar calling
conventions/optimizations).
py/mphal.h contains declarations for generic mp_hal_XXX functions, such
as stdio and delay/ticks, which ports should provide definitions for. A
port will also provide mphalport.h with further HAL declarations.
These MPHAL functions are intended to replace previously used HAL_Delay(),
HAL_GetTick() to provide better naming and MPHAL separation (they are
fully equivalent otherwise).
Also, refactor extmod/modlwip to use them.
This is required to properly select among overloaded methods. It however
relies on java.lang.Object-overloaded method to come last, which appears
to be the case for OpenJDK.
Another function (like stat) which is problematic to deal with on ABI level
(FFI), as struct statvfs layout may differ unpredictably between OSes and
even different versions of a same OS. So, implement it in C, returning a
10-element tuple of f_bsize, f_frsize, f_blocks, f_bfree, f_bavail, f_files,
f_ffree, f_favail, f_flag, f_namemax. This is exactly the order described
in Python3 docs, https://docs.python.org/3/library/os.html#os.statvfs
(but note that os.statvfs() should make these values available as
attributes).
As we dn't export constants for TCSANOW, etc., zero makes a good "don't
care" param, and now it will work also under Android Bionic and any other
libc.
Use CTRL-E to enter paste mode. Prompt starts with "===" and accepts
all characters verbatim, echoing them back. Only control characters are
CTRL-C which cancels the input and returns to normal REPL, and CTRL-D
which ends the input and executes it. The input is executed as though
it were a file. The input is not added to the prompt history.
With this patch parse nodes are allocated sequentially in chunks. This
reduces fragmentation of the heap and prevents waste at the end of
individually allocated parse nodes.
Saves roughly 20% of RAM during parse stage.
This fixes errors like these ones:
modffi.c: In function 'return_ffi_value':
modffi.c:143:29: error: cast to pointer from integer of different size
[-Werror=int-to-pointer-cast]
const char *s = (const char *)val;
^
modffi.c:162:20: error: cast to pointer from integer of different size
[-Werror=int-to-pointer-cast]
return (mp_obj_t)val;
^
modffi.c: In function 'ffifunc_call':
modffi.c:358:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)a;
^
modffi.c:373:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)s;
^
modffi.c:381:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)bufinfo.buf;
^
modffi.c:384:25: error: cast from pointer to integer of different size
[-Werror=pointer-to-int-cast]
values[i] = (ffi_arg)p->func;
^
These errors can be highlighted when building micropython from MIPS64
n32 because ffi_arg is 64-bit wide and the pointers on MIPS64 n32 are
32-bit wide, so it's trying to case an integer to a pointer (or
vice-versa) of a different size. We should cast first the pointer (or the
integer) to a pointer sized integer (intptr_t) to fix that problem.
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Linking against local libffi (and other libs in future) is triggered by
"make MICROPY_STANDALONE=1". Before that, dependent libs should be built
with "make deplibs".
Indeed, this flag efectively selects architecture target, and must
consistently apply to all compiles and links, including 3rd-party
libraries, unlike CFLAGS, which have MicroPython-specific setting.
inet_pton supports both ipv4 and ipv6 addresses. Interface is also extensible
for other address families, but underlying libc inet_pton() function isn't
really extensible (e.g., it doesn't return length of binary address, i.e. it's
really hardcoded to AF_INET and AF_INET6). But anyway, on Python side, we could
extend it to support other addresses.
sendto() turns out to be mandatory function to work with UDP. It may seem
that connect(addr) + send() would achieve the same effect, but what connect()
appears to do is to set source address filter on a socket to its argument.
Then everything falls apart: socket sends to a broad-/multi-cast address,
but reply is sent from real peer address, which doesn't match filter set
by connect(), so local socket never sees a reply.
This requires root access. And on recent Linux kernels, with
CONFIG_STRICT_DEVMEM option enabled, only address ranges listed in
/proc/iomem can be accessed. The above compiled-time option can be
however overriden with boot-time option "iomem=relaxed".
This also removed separate read/write paths - there unlikely would
be a case when they're different.
Previous to this patch a call such as list.append(1, 2) would lead to a
seg fault. This is because list.append is a builtin method and the first
argument to such methods is always assumed to have the correct type.
Now, when a builtin method is extracted like this it is wrapped in a
checker object which checks the the type of the first argument before
calling the builtin function.
This feature is contrelled by MICROPY_BUILTIN_METHOD_CHECK_SELF_ARG and
is enabled by default.
See issue #1216.
MicroPython doesn't come with standard library included, so it is important
to be able to easily install needed package in a seamless manner. Bundling
package manager (upip) inside an executable solves this issue.
upip is bundled only with standard executable, not "minimal" or "fast"
builds.
Using MICROPY_PY_SYS_PATH_DEFAULT macro define. A usecase is building a
distribution package, which should not have user home path by default in
sys.path. In such case, MICROPY_PY_SYS_PATH_DEFAULT can be defined on
make command-line (using CFLAGS_EXTRA).
This gets uPy readline working with unix port, with tab completion and
history. GNU readline is still supported, configure using
MICROPY_USE_READLINE variable.
The function and corresponding command-line option are only enabled for
the coverage build. They are used to exercise uPy features that can't
be properly tested by Python scripts.
From https://docs.python.org/3/library/constants.html#NotImplemented :
"Special value which should be returned by the binary special methods
(e.g. __eq__(), __lt__(), __add__(), __rsub__(), etc.) to indicate
that the operation is not implemented with respect to the other type;
may be returned by the in-place binary special methods (e.g. __imul__(),
__iand__(), etc.) for the same purpose. Its truth value is true."
Some people however appear to abuse it to mean "no value" when None is
a legitimate value (don't do that).
The implementation is very basic and non-compliant and provided solely for
CPython compatibility. The function itself is bad Python2 heritage, its
usage is discouraged.
Previous to this patch the printing mechanism was a bit of a tangled
mess. This patch attempts to consolidate printing into one interface.
All (non-debug) printing now uses the mp_print* family of functions,
mainly mp_printf. All these functions take an mp_print_t structure as
their first argument, and this structure defines the printing backend
through the "print_strn" function of said structure.
Printing from the uPy core can reach the platform-defined print code via
two paths: either through mp_sys_stdout_obj (defined pert port) in
conjunction with mp_stream_write; or through the mp_plat_print structure
which uses the MP_PLAT_PRINT_STRN macro to define how string are printed
on the platform. The former is only used when MICROPY_PY_IO is defined.
With this new scheme printing is generally more efficient (less layers
to go through, less arguments to pass), and, given an mp_print_t*
structure, one can call mp_print_str for efficiency instead of
mp_printf("%s", ...). Code size is also reduced by around 200 bytes on
Thumb2 archs.
splitlines() occurs ~179 times in CPython3 standard library, so was
deemed worthy to implement. The method has subtle semantic differences
from just .split("\n"). It is also defined as working for any end-of-line
combination, but this is currently not implemented - it works only with
LF line-endings (which should be OK for text strings on any platforms,
but not OK for bytes).
Given that there's already support for "fixed table" maps, which are
essentially ordered maps, the implementation of OrderedDict just extends
"fixed table" maps by adding an "is ordered" flag and add/remove
operations, and reuses 95% of objdict code, just making methods tolerant
to both dict and OrderedDict.
Some things are missing so far, like CPython-compatible repr and comparison.
OrderedDict is Disabled by default; enabled on unix and stmhal ports.
These allow to fine-tune the compiler to select whether it optimises
tuple assignments of the form a, b = c, d and a, b, c = d, e, f.
Sensible defaults are provided.
This is rarely used feature which takes enough code to implement, so is
controlled by MICROPY_PY_ARRAY_SLICE_ASSIGN config setting, default off.
But otherwise it may be useful, as allows to update arbitrary-sized data
buffers in-place.
Slice is yet to implement, and actually, slice assignment implemented in
such a way that RHS of assignment should be array of the exact same item
typecode as LHS. CPython has it more relaxed, where RHS can be any sequence
of compatible types (e.g. it's possible to assign list of int's to a
bytearray slice).
Overall, when all "slice write" features are implemented, it may cost ~1KB
of code.
The implementation of these functions is very large (order 4k) and they
are rarely used, so we don't enable them by default.
They are however enabled in stmhal and unix, since we have the room.
To enable parsing constants more efficiently, mp_parse should be allowed
to raise an exception, and mp_compile can already raise a MemoryError.
So these functions need to be protected by an nlr push/pop block.
This patch adds that feature in all places. This allows to simplify how
mp_parse and mp_compile are called: they now raise an exception if they
have an error and so explicit checking is not needed anymore.
Native code has GC-heap pointers in it so it must be scanned. But on
unix port memory for native functions is mmap'd, and so it must have
explicit code to scan it for root pointers.
Compiler optimises lookup of module.CONST when enabled (an existing
feature). Disabled by default; enabled for unix, windows, stmhal.
Costs about 100 bytes ROM on stmhal.
This allows to enable mem-info functions in micropython module, even if
MICROPY_MEM_STATS is not enabled. In this case, you get mem_info and
qstr_info but not mem_{total,current,peak}.
GC for unix/windows builds doesn't make use of the bss section anymore,
so we do not need the (sometimes complicated) build features and code related to it
This is a simple optimisation inspired by JITing technology: we cache in
the bytecode (using 1 byte) the offset of the last successful lookup in
a map. This allows us next time round to check in that location in the
hash table (mp_map_t) for the desired entry, and if it's there use that
entry straight away. Otherwise fallback to a normal map lookup.
Works for LOAD_NAME, LOAD_GLOBAL, LOAD_ATTR and STORE_ATTR opcodes.
On a few tests it gives >90% cache hit and greatly improves speed of
code.
Disabled by default. Enabled for unix and stmhal ports.
This patch consolidates all global variables in py/ core into one place,
in a global structure. Root pointers are all located together to make
GC tracing easier and more efficient.
This patch makes MICROPY_PY_BUILTINS_SET compile-time option fully
disable the builtin set object (when set to 0). This includes removing
set constructor/comprehension from the grammar, the compiler and the
emitters. Now, enabling set costs 8168 bytes on unix x64, and 3576
bytes on stmhal.
system() is the basic function to support automation of tasks, so have it
available builtin, for example, for bootstrapping rest of micropython
environment.
This patch adds a configuration option (MICROPY_CAN_OVERRIDE_BUILTINS)
which, when enabled, allows to override all names within the builtins
module. A builtins override dict is created the first time the user
assigns to a name in the builtins model, and then that dict is searched
first on subsequent lookups. Note that this implementation doesn't
allow deleting of names.
This patch also does some refactoring of builtins code, creating the
modbuiltins.c file.
Addresses issue #959.
The function is modeled after traceback.print_exception(), but unbloated,
and put into existing module to save overhead on adding another module.
Compliant traceback.print_exception() is intended to be implemented in
micropython-lib in terms of sys.print_exception().
This change required refactoring mp_obj_print_exception() to take pfenv_t
interface arguments.
Addresses #751.
mp_obj_int_get_truncated is used as a "fast path" int accessor that
doesn't check for overflow and returns the int truncated to the machine
word size, ie mp_int_t.
Use mp_obj_int_get_truncated to fix struct.pack when packing maximum word
sized values.
Addresses issues #779 and #998.
mp_lexer_t type is exposed, mp_token_t type is removed, and simple lexer
functions (like checking current token kind) are now inlined.
This saves 784 bytes ROM on 32-bit unix, 348 bytes on stmhal, and 460
bytes on bare-arm. It also saves a tiny bit of RAM since mp_lexer_t
is a bit smaller. Also will run a bit more efficiently.
The specifier should go after the number, before size suffix like 'k' or 'm'.
E.g.: "-X heapsize=100wk" will use 100K heap on 32-bit system and 200K - on
64-bit.
This build is primarily intended for benchmarking, and may have random
features enabled/disabled to get high scores in synthetic benchmarks.
The intent is to show/prove that MicroPython codebase can compete with
CPython, when configured appropriately. But the main MicroPython aim
still remains to optimize for memory usage (which inevitibly leads to
performance degradation in some areas on some workloads).
gc.enable/disable are now the same as CPython: they just control whether
automatic garbage collection is enabled or not. If disabled, you can
still allocate heap memory, and initiate a manual collection.
Support for packages as argument not implemented, but otherwise error and
exit handling should be correct. This for example will allow to do:
pip-micropython install micropython-test.pystone
micropython -m test.pystone
This allows to implement KeyboardInterrupt on unix, and a much safer
ctrl-C in stmhal port. First ctrl-C is a soft one, with hope that VM
will notice it; second ctrl-C is a hard one that kills anything (for
both unix and stmhal).
One needs to check for a pending exception in the VM only for jump
opcodes. Others can't produce an infinite loop (infinite recursion is
caught by stack check).
This makes open() and _io.FileIO() more CPython compliant.
The mode kwarg is fully iplemented.
The encoding kwarg is allowed but not implemented; mainly to allow
the tests to specify encoding for CPython, see #874
Also, usocket.readinto(). Known issue is that .readinto() should be available
only for binary files, but micropython uses single method table for both
binary and text files.
Per new conventions, we'd like to consistently use "u*" naming conventions
for modules which don't offer complete CPython compatibility, while offer
subset or similar API.
