unix-cpy was originally written to get semantic equivalent with CPython
without writing functional tests. When writing the initial
implementation of uPy it was a long way between lexer and functional
tests, so the half-way test was to make sure that the bytecode was
correct. The idea was that if the uPy bytecode matched CPython 1-1 then
uPy would be proper Python if the bytecodes acted correctly. And having
matching bytecode meant that it was less likely to miss some deep
subtlety in the Python semantics that would require an architectural
change later on.
But that is all history and it no longer makes sense to retain the
ability to output CPython bytecode, because:
1. It outputs CPython 3.3 compatible bytecode. CPython's bytecode
changes from version to version, and seems to have changed quite a bit
in 3.5. There's no point in changing the bytecode output to match
CPython anymore.
2. uPy and CPy do different optimisations to the bytecode which makes it
harder to match.
3. The bytecode tests are not run. They were never part of Travis and
are not run locally anymore.
4. The EMIT_CPYTHON option needs a lot of extra source code which adds
heaps of noise, especially in compile.c.
5. Now that there is an extensive test suite (which tests functionality)
there is no need to match the bytecode. Some very subtle behaviour is
tested with the test suite and passing these tests is a much better
way to stay Python-language compliant, rather than trying to match
CPy bytecode.
- subprocess.check_output can only handle strings on windows, not bytes,
so convert the arguments as such
- the pty module is for posix systems only so skip the tests needing it
in case it is not available
The adapter class "TelnetToSerial" is used to access the Telnet
connection using the same API as with the serial connection. The
function pyboard.run-test() has been removed to made the module
generic and because this small test is no longer needed.
This requires root access. And on recent Linux kernels, with
CONFIG_STRICT_DEVMEM option enabled, only address ranges listed in
/proc/iomem can be accessed. The above compiled-time option can be
however overriden with boot-time option "iomem=relaxed".
This also removed separate read/write paths - there unlikely would
be a case when they're different.
Previous to this patch each time a bytes object was referenced a new
instance (with the same data) was created. With this patch a single
bytes object is created in the compiler and is loaded directly at execute
time as a true constant (similar to loading bignum and float objects).
This saves on allocating RAM and means that bytes objects can now be
used when the memory manager is locked (eg in interrupts).
The MP_BC_LOAD_CONST_BYTES bytecode was removed as part of this.
Generated bytecode is slightly larger due to storing a pointer to the
bytes object instead of the qstr identifier.
Code size is reduced by about 60 bytes on Thumb2 architectures.
The --pyboard param has been replaced by --target which defaults to
'unix'. Possible values at this moment are 'unix', 'pyboard' and
'wipy'. Now is also possible to select the baud rate of the serial
device when calling the script.
Previous to this patch a call such as list.append(1, 2) would lead to a
seg fault. This is because list.append is a builtin method and the first
argument to such methods is always assumed to have the correct type.
Now, when a builtin method is extracted like this it is wrapped in a
checker object which checks the the type of the first argument before
calling the builtin function.
This feature is contrelled by MICROPY_BUILTIN_METHOD_CHECK_SELF_ARG and
is enabled by default.
See issue #1216.
If heap allocation for the Python-stack of a function fails then we may
as well allocate the Python-stack on the C stack. This will allow to
run more code without using the heap.
Now address comes first, and args related to struct type are groupped next.
Besides clear groupping, should help catch errors eagerly (e.g. forgetting
to pass address will error out).
Also, improve args number checking/reporting overall.
This allows to do "ar[i]" and "ar[i] = val" in viper when ar is a Python
object and i and/or val are native viper types (eg ints).
Patch also includes tests for this feature.
C's printf will pad nan/inf differently to CPython. Our implementation
originally conformed to C, now it conforms to CPython's way.
Tests for this are also added in this patch.
This gets uPy readline working with unix port, with tab completion and
history. GNU readline is still supported, configure using
MICROPY_USE_READLINE variable.
In particular, dates prior to Mar 1, 2000 are screwed up.
The easiest way to see this is to do:
>>> import time
>>> time.localtime(0)
(2000, 1, 1, 0, 0, 0, 5, 1)
>>> time.localtime(1)
(2000, 1, 2, 233, 197, 197, 6, 2)
With this patch, we instead get:
>>> import time
>>> time.localtime(1)
(2000, 1, 1, 0, 0, 1, 5, 1)
Doh - In C % is NOT a modulo operator, it's a remainder operator.
This allows using (almost) the same code for printing floats everywhere,
removes the dependency on sprintf and uses just snprintf and
applies an msvc-specific fix for snprintf in a single place so
nan/inf are now printed correctly.
Hashing is now done using mp_unary_op function with MP_UNARY_OP_HASH as
the operator argument. Hashing for int, str and bytes still go via
fast-path in mp_unary_op since they are the most common objects which
need to be hashed.
This lead to quite a bit of code cleanup, and should be more efficient
if anything. It saves 176 bytes code space on Thumb2, and 360 bytes on
x86.
The only loss is that the error message "unhashable type" is now the
more generic "unsupported type for __hash__".
User instances are hashable by default (using __hash__ inherited from
"object"). But if __eq__ is defined and __hash__ not defined in particular
class, instance is not hashable.
This doesn't handle case fo enclosed except blocks, but once again,
sys.exc_info() support is a workaround for software which uses it
instead of properly catching exceptions via variable in except clause.
The implementation is very basic and non-compliant and provided solely for
CPython compatibility. The function itself is bad Python2 heritage, its
usage is discouraged.
Before this patch a "with" block needed to create a bound method object
on the heap for the __exit__ call. Now it doesn't because we use
load_method instead of load_attr, and save the method+self on the stack.
Adds support for the following Thumb2 VFP instructions, via the option
MICROPY_EMIT_INLINE_THUMB_FLOAT:
vcmp
vsqrt
vneg
vcvt_f32_to_s32
vcvt_s32_to_f32
vmrs
vmov
vldr
vstr
vadd
vsub
vmul
vdiv
This patch gets full function argument passing working with native
emitter. Includes named args, keyword args, default args, var args
and var keyword args. Fully Python compliant.
It reuses the bytecode mp_setup_code_state function to do all the hard
work. This function is slightly adjusted to accommodate native calls,
and the native emitter is forced a bit to emit similar prelude and
code-info as bytecode.
This makes exception traceback info self contained (ie doesn't rely on
list object, which was a bit of a hack), reduces code size, and reduces
RAM footprint of exception by eliminating the list object.
Addresses part of issue #1126.
Previous to this patch, a big-int, float or imag constant was interned
(made into a qstr) and then parsed at runtime to create an object each
time it was needed. This is wasteful in RAM and not efficient. Now,
these constants are parsed straight away in the parser and turned into
objects. This allows constants with large numbers of digits (so
addresses issue #1103) and takes us a step closer to #722.
Bytecode also needs a pass to compute the stack size. This is because
the state size of the bytecode function is encoded as a variable uint,
so we must know the value of this uint before we encode it (otherwise
the size of the generated code changes from one pass to the next).
Having an entire pass for this seems wasteful (in time). Alternative is
to allocate fixed space for the state size (would need 3-4 bytes to be
general, when 1 byte is usually sufficient) which uses a bit of extra
RAM per bytecode function, and makes the code less elegant in places
where this uint is encoded/decoded.
So, for now, opt for an extra pass.
Previously to this patch all constant string/bytes objects were
interned by the compiler, and this lead to crashes when the qstr was too
long (noticeable now that qstr length storage defaults to 1 byte).
With this patch, long string/bytes objects are never interned, and are
referenced directly as constant objects within generated code using
load_const_obj.
This fixes conversion when float type has more mantissa bits than small int,
and float value has small exponent. This is for example the case of 32-bit
platform using doubles, and converting value of time.time(). Conversion of
floats with larg exponnet is still not handled correctly.
This script is rewrite of run-tests-exp.sh, and tries to achieve self-hosted
testsuite running in environments where neither CPython nor unix shell is
available. As run-tests-exp.sh, it requires complete set of .exp files
pre-generated with ./run-test --write-exp.
Just adjust line-endings of micropython.exe output, the rest should be
handled by Wine (automagically on properly configured distro).
To run:
MICROPY_MICROPYTHON=../windows/micropython.exe ./run-tests
acoshf, asinhf, atanhf were added from musl. mathsincos.c was
split up into its original, separate files (from newlibe-nano-2).
tan was added.
All of the important missing float functions are now implemented,
and pyboard now passes tests/float/math_fun.py (finally!).
You can now assign to the range end variable and the for-loop still
works correctly. This fully addresses issue #565.
Also fixed a bug with the stack not being fully popped when breaking out
of an optimised for-loop (and it's actually impossible to write a test
for this case!).
Issue was with uPy: on local machine with micropython-lib installed, io
module is available. Not the case on Travis CI, where only _io module
is available in uPy.
This patch adds a configuration option (MICROPY_CAN_OVERRIDE_BUILTINS)
which, when enabled, allows to override all names within the builtins
module. A builtins override dict is created the first time the user
assigns to a name in the builtins model, and then that dict is searched
first on subsequent lookups. Note that this implementation doesn't
allow deleting of names.
This patch also does some refactoring of builtins code, creating the
modbuiltins.c file.
Addresses issue #959.
mp_obj_int_get_truncated is used as a "fast path" int accessor that
doesn't check for overflow and returns the int truncated to the machine
word size, ie mp_int_t.
Use mp_obj_int_get_truncated to fix struct.pack when packing maximum word
sized values.
Addresses issues #779 and #998.
Behaviour of array initialisation is subtly different for bytes,
bytearray and array.array when argument has buffer protocol. This patch
gets us CPython conformant (except we allow initialisation of
array.array by buffer with length not a multiple of typecode).
Eg b"123" + bytearray(2) now works. This patch actually decreases code
size while adding functionality: 32-bit unix down by 128 bytes, stmhal
down by 84 bytes.
gc.enable/disable are now the same as CPython: they just control whether
automatic garbage collection is enabled or not. If disabled, you can
still allocate heap memory, and initiate a manual collection.
Before, sizeof() could be applied to a structure field only if that field
was itself a structure. Now it can be applied to PTR and ARRAY fields too.
It's not possible to apply it to scalar fields though, because as soon as
scalar field (int or float) is dereferenced, its value is converted into
Python int/float value, and all original type info is lost. Moreover, we
allow sizeof of type definitions too, and there int is used to represent
(scalar) types. So, we have ambiguity what int may be - either dereferenced
scalar structure field, or encoded scalar type. So, rather throw an error
if user tries to apply sizeof() to int.
UART object now uses a stream-like interface: read, readall, readline,
readinto, readchar, write, writechar.
Timeouts are configured when the UART object is initialised, using
timeout and timeout_char keyword args.
The object includes optional read buffering, using interrupts. You can set
the buffer size dynamically using read_buf_len keyword arg. A size of 0
disables buffering.
mode argument is used to assert it works
encoding argument is used to make sure CPython uses the correct encoding
as it does not automatically use utf-8
Also, usocket.readinto(). Known issue is that .readinto() should be available
only for binary files, but micropython uses single method table for both
binary and text files.
If micropython.native decorator doesn't compile, then we skill all
native/viper tests.
This patch also re-enables the ujson_loads test on NT.
Addresses issue #861, and partially addresses issue #856.
In CPython IOError (and EnvironmentError) is deprecated and aliased to
OSError. All modules that used to raise IOError now raise OSError (or a
derived exception).
In Micro Python we never used IOError (except 1 place, incorrectly) and
so don't need to keep it.
See http://legacy.python.org/dev/peps/pep-3151/ for background.
Viper can now do the following:
def store(p:ptr8, c:int):
p[0] = c
This does a store of c to the memory pointed to by p using a machine
instructions inline in the code.
For the sake of older versions of gcc (and other compilers), don't use
the #warning CPP directive, nor the -Wno-error=cpp option.
Also, fix a strict alias warning in modffi.c for older compilers, and
add a test for ffi module.
Addresses issue #847.
Previously, mpz was restricted to using at most 15 bits in each digit,
where a digit was a uint16_t.
With this patch, mpz can use all 16 bits in the uint16_t (improvement
to mpn_div was required). This gives small inprovements in speed and
RAM usage. It also yields savings in ROM code size because all of the
digit masking operations become no-ops.
Also, mpz can now use a uint32_t as the digit type, and hence use 32
bits per digit. This will give decent improvements in mpz speed on
64-bit machines.
Test for big integer division added.
Because (for Thumb) a function pointer has the LSB set, pointers to
dynamic functions in RAM (eg native, viper or asm functions) were not
being traced by the GC. This patch is a comprehensive fix for this.
Addresses issue #820.
Multiplication of a tuple, list, str or bytes now yields an empty
sequence (instead of crashing). Addresses issue #799
Also added ability to mult bytes on LHS by integer.
Can now index ranges with integers and slices, and reverse ranges
(although reversing is not very efficient).
Not sure how useful this stuff is, but gets us closer to having all of
Python's builtins.
reversed function now implemented, and works for tuple, list, str, bytes
and user objects with __len__ and __getitem__.
Renamed mp_builtin_len to mp_obj_len to make it publically available (eg
for reversed).
With unicode enabled, this patch allows reading a fixed number of
characters from text-mode streams; eg file.read(5) will read 5 unicode
chars, which can made of more than 5 bytes.
For an ASCII stream (ie no chars > 127) it only needs to do 1 read. If
there are lots of non-ASCII chars in a stream, then it needs multiple
reads of the underlying object.
Adds a new test for this case. Enables unicode support by default on
unix and stmhal ports.
This script uses expected test results as generated by run-tests --write-exp,
and requires only standard unix shell funtionality (no bash). It is useful
to run testsuite on embedded systems, where there's no CPython and Bash.
Both "bound" (like, length known) and "unbound" (length unknown) are tested.
All of list, tuple, bytes, bytesarray offer approximately the same
performance, with "unbound" case being 30 times slower.
This will allow roughly the same behavior as Python3 for non-ASCII strings,
for example, print("<phrase in non-Latin script>".split()) will print list
of words, not weird hex dump (like Python2 behaves). (Of course, that it
will print list of words, if there're "words" in that phrase at all, separated
by ASCII-compatible whitespace; that surely won't apply to every human
language in existence).
Functionality we provide in builtin io module is fairly minimal. Some
code, including CPython stdlib, depends on more functionality. So, there's
a choice to either implement it in C, or move it _io, and let implement other
functionality in Python. 2nd choice is pursued. This setup matches CPython
too (_io is builtin, io is Python-level).
See http://bugs.python.org/issue19995 for rationale. As micropython currently
aims for Python 3.3 compatibility I have not changed behaviour, but this
change allows the test suite to be run against a newer Python without having
spurious failures.
Add keyword args to dict.update(), and ability to take a dictionary as
argument.
dict() class constructor can now use dict.update() directly.
This patch loses fast path for dict(other_dict), but is that really
needed? Any anyway, this idiom will now re-hash the dictionary, so is
arguably more memory efficient.
Addresses issue #647.
This is not fully correct re: error handling, because we should check that
that types are used consistently (only str's or only bytes), but magically
makes lot of functions support bytes.
Two things are handled here: allow to compare native subtypes of tuple,
e.g. namedtuple (TODO: should compare type too, currently compared
duck-typedly by content). Secondly, allow user sunclasses of tuples
(and its subtypes) be compared either. "Magic" I did previously in
objtype.c covers only one argument (lhs is many), so we're in trouble
when lhs is native type - there's no other option besides handling
rhs in special manner. Fortunately, this patch outlines approach with
fast path for native types.
This was hit when trying to make urlparse.py from stdlib run. Took
quite some time to debug.
TODO: Reconsile bound method creation process better, maybe callable is
to generic type to bind at all?
"object" type in MicroPython currently doesn't implement any methods, and
hopefully, we'll try to stay like that for as long as possible. Even if we
have to add something eventually, look up from there might be handled in
adhoc manner, as last resort (that's not compliant with Python3 MRO, but
we're already non-compliant). Hence: 1) no need to spend type trying to
lookup anything in object; 2) no need to allocate subobject when explicitly
inheriting from object; 3) and having multiple bases inheriting from object
is not a case of incompatible multiple inheritance.
You can now do:
X = const(123)
Y = const(456 + X)
and the compiler will replace X and Y with their values.
See discussion in issue #266 and issue #573.
... and we have not that bad mapping type after all - lookup time is ~ the
same as in one-attr instance. My namedtuple implementation on the other
hand degrades awfully.
So, need to rework it. First observation is that named tuple fields are
accessed as attributes, so all names are interned at the program start.
Then, really should store field array as qstr[], and do quick 32/64 bit
scan thru it.
Need to have a policy as to how far we go adding keyword support to
built ins. It's nice to have, and gets better CPython compatibility,
but hurts the micro nature of uPy.
Addresses issue #577.
Motivation is optimizing handling of various constructs as well as
understanding which constructs are more efficient in MicroPython.
More info: http://forum.micropython.org/viewtopic.php?f=3&t=77
Results are wildly unexpected. For example, "optimization" of range
iteration into while loop makes it twice as slow. Generally, the more
bytecodes, the slower the code.
In tests/pyb is now a suite of tests that tests the pyb module on the
pyboard. They include expected output files because we can't run
CPython on the pyboard to compare against.
run-tests script has now been updated to allow pyboard tests to be run.
Just pass the option --pyboard. This runs all basic, float and pyb
tests. Note that float/math-fun.py currently fails because not all math
functions are implemented in stmhal/.
Tests in basics (which should probably be renamed to core) should not
rely on float, or import any non-built-in files. This way these tests
can be run when those features are not available.
All test in basics now pass on the pyboard using stmhal port, except for
string-repr which has some issues with character hex printing.
This is necessary to catch all cases where locals are referenced before
assignment. We still keep the _0, _1, _2 versions of LOAD_FAST to help
reduced the byte code size in RAM.
Addresses issue #457.
I'm pretty sure these are never reached, since NOT_EQUAL is always
converted into EQUAL in mp_binary_op. No one should call
type.binary_op directly, they should always go through mp_binary_op
(or mp_obj_is_equal).
Only calcsize() and unpack() functions provided so far, for little-endian
byte order. Format strings don't support repition spec (like "2b3i").
Unfortunately, dealing with all the various binary type sizes and alignments
will lead to quite a bloated "binary" helper functions - if optimizing for
speed. Need to think if using dynamic parametrized algos makes more sense.
These two are apprerently the most concise and efficient way to convert
int to/from bytes in Python. The alternatives are struct and array modules,
but methods using them are more verbose in Python code and less efficient
in memory/cycles.
Damien George
Paul Sokolovsky
blmorris
Tom Soulanille
stijn
Daniel Campora
Delio Brignoli
Sebastian Plamauer
Galen Hazelwood
Dave Hylands
stijn
=
Henrik
Peter D. Gray
David Steinberg
Chris Angelico
Kim Bauters
Antonin ENFRUN
Andrew Scheller
xbe