Commit Graph

341 Commits

Author SHA1 Message Date
Jim Mussared fb2a57800a all: Simplify buffer protocol to just a "get buffer" callback.
The buffer protocol type only has a single member, and this existing layout
creates problems for the upcoming split/slot-index mp_obj_type_t layout
optimisations.

If we need to make the buffer protocol more sophisticated in the future
either we can rely on the mp_obj_type_t optimisations to just add
additional slots to mp_obj_type_t or re-visit the buffer protocol then.

This change is a no-op in terms of generated code.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 18:40:39 +10:00
Jim Mussared 6c3d8d38bf py/objstr: Always validate utf-8 for mp_obj_new_str.
All uses of this are either tiny strings or not-known-to-be-safe.

Update comments for mp_obj_new_str_copy and mp_obj_new_str_of_type.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:45:46 +10:00
Jim Mussared 3a910b1565 py/objstr: Optimise mp_obj_new_str_from_vstr for known-safe strings.
The new `mp_obj_new_str_from_utf8_vstr` can be used when you know you
already have a unicode-safe string.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:44:35 +10:00
Jim Mussared 88864587f5 py/objstr: Always ensure mp_obj_str_from_vstr is unicode-safe.
Now that we have `mp_obj_new_str_type_from_vstr` (private helper used by
objstr.c) split from the public API (`mp_obj_new_str_from_vstr`), we can
enforce a unicode check at the public API without incurring a performance
cost on the various objstr.c methods (which are already working on known
unicode-safe strings).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:44:20 +10:00
Jim Mussared 8a0ee5a5c0 py/objstr: Split mp_obj_str_from_vstr into bytes/str versions.
Previously the desired output type was specified.  Now make the type part
of the function name.  Because this function is used in a few places this
saves code size due to smaller call-site.

This makes `mp_obj_new_str_type_from_vstr` a private function of objstr.c
(which is almost the only place where the output type isn't a compile-time
constant).

This saves ~140 bytes on PYBV11.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:43:55 +10:00
Jim Mussared 28aaab9590 py/objstr: Add hex/fromhex to bytes/memoryview/bytearray.
These were added in Python 3.5.

Enabled via MICROPY_PY_BUILTINS_BYTES_HEX, and enabled by default for all
ports that currently have ubinascii.

Rework ubinascii to use the implementation of these methods.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-12 12:44:30 +10:00
Andrew Leech f7f56d4285 py/objstr: Consolidate methods for str/bytes/bytearray/array.
This commit adds the bytes methods to bytearray, matching CPython.  The
existing implementations of these methods for str/bytes are reused for
bytearray with minor updates to match CPython return types.

For details on the CPython behaviour see
https://docs.python.org/3/library/stdtypes.html#bytes-and-bytearray-operations

The work to merge locals tables for str/bytes/bytearray/array was done by
@jimmo.  Because of this merging of locals the change in code size for this
commit is mostly negative:

       bare-arm:    +0 +0.000%
    minimal x86:   +29 +0.018%
       unix x64:  -792 -0.128% standard[incl -448(data)]
    unix nanbox:  -436 -0.078% nanbox[incl -448(data)]
          stm32:   -40 -0.010% PYBV10
         cc3200:   -32 -0.017%
        esp8266:   -28 -0.004% GENERIC
          esp32:   -72 -0.005% GENERIC[incl -200(data)]
         mimxrt:   -40 -0.011% TEENSY40
     renesas-ra:   -40 -0.006% RA6M2_EK
            nrf:   -16 -0.009% pca10040
            rp2:   -64 -0.013% PICO
           samd:  +148 +0.105% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2022-08-11 23:18:02 +10:00
Yonatan Goldschmidt 2a6ba47110 py/obj: Add static safety checks to mp_obj_is_type().
Commit d96cfd13e3 introduced a regression by breaking existing
users of mp_obj_is_type(.., &mp_obj_bool).  This function (and associated
helpers like mp_obj_is_int()) have some specific nuances, and mistakes like
this one can happen again.

This commit adds mp_obj_is_exact_type() which behaves like the the old
mp_obj_is_type().  The new mp_obj_is_type() has the same prototype but it
attempts to statically assert that it's not called with types which should
be checked using mp_obj_is_type().  If called with any of these types: int,
str, bool, NoneType - it will cause a compilation error.  Additional
checked types (e.g function types) can be added in the future.

Existing users of mp_obj_is_type() with the now "invalid" types, were
translated to use mp_obj_is_exact_type().

The use of MP_STATIC_ASSERT() is not bulletproof - usually GCC (and other
compilers) can't statically check conditions that are only known during
link-time (like variables' addresses comparison).  However, in this case,
GCC is able to statically detect these conditions, probably because it's
the exact same object - `&mp_type_int == &mp_type_int` is detected.
Misuses of this function with runtime-chosen types (e.g:
`mp_obj_type_t *x = ...; mp_obj_is_type(..., x);` won't be detected.  MSC
is unable to detect this, so we use MP_STATIC_ASSERT_NOT_MSC().

Compiling with this commit and without the fix for d96cfd13e3 shows
that it detects the problem.

Signed-off-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com>
2022-07-18 11:17:46 +10:00
Jim Mussared 0e7bfc88c6 all: Use mp_obj_malloc everywhere it's applicable.
This replaces occurences of

    foo_t *foo = m_new_obj(foo_t);
    foo->base.type = &foo_type;

with

    foo_t *foo = mp_obj_malloc(foo_t, &foo_type);

Excludes any places where base is a sub-field or when new0/memset is used.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-03 22:28:14 +10:00
Jeff Epler 037b2c72a1 py/objstr: Support '{:08}'.format("Jan") like Python 3.10.
The new test has an .exp file, because it is not compatible with Python 3.9
and lower.

See CPython version of the issue at https://bugs.python.org/issue27772

Signed-off-by: Jeff Epler <jepler@gmail.com>
2022-01-19 15:34:32 +11:00
Damien George 38a204ed96 py: Introduce and use mp_raise_type_arg helper.
To reduce code size.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-15 00:12:41 +10:00
Damien George d4b706c4d0 py: Add option to compile without any error messages at all.
This introduces a new option, MICROPY_ERROR_REPORTING_NONE, which
completely disables all error messages.  To be used in cases where
MicroPython needs to fit in very limited systems.

Signed-off-by: Damien George <damien@micropython.org>
2021-04-27 23:51:52 +10:00
Joris Peeraer 5020b14d54 py/mpprint: Fix length calculation for strings with precision-modifier.
Two issues are tackled:

1. The calculation of the correct length to print is fixed to treat the
   precision as a maximum length instead as the exact length.
   This is done for both qstr (%q) and for regular str (%s).

2. Fix the incorrect use of mp_printf("%.*s") to mp_print_strn().

   Because of the fix of above issue, some testcases that would print
   an embedded null-byte (^@ in test-output) would now fail.
   The bug here is that "%s" was used to print null-bytes. Instead,
   mp_print_strn is used to make sure all bytes are outputted and the
   exact length is respected.

Test-cases are added for both %s and %q with a combination of precision
and padding specifiers.
2020-12-07 23:32:06 +11:00
Iyassou Shimels ca017841d6 py/objstr: Make bytes(bytes_obj) return bytes_obj.
Calling the bytes constructor on a bytes object returns the original bytes
object.  This saves allocating a new instance, and matches CPython.

Signed-off-by: Iyassou Shimels <s.iyassou@gmail.com>
2020-09-24 11:04:58 +10:00
stijn 84fa3312cf all: Format code to add space after C++-style comment start.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
2020-04-23 11:24:25 +10:00
Jim Mussared def76fe4d9 all: Use MP_ERROR_TEXT for all error messages. 2020-04-05 15:02:06 +10:00
Jim Mussared a9a745e4b4 py: Use preprocessor to detect error reporting level (terse/detailed).
Instead of compiler-level if-logic.  This is necessary to know what error
strings are included in the build at the preprocessor stage, so that string
compression can be implemented.
2020-04-05 14:11:51 +10:00
Tom Collins fccf17521a py/objstr: Remove duplicate % in error string.
The double-% was added in 11de8399fe (Jun
2014) when such errors were formatted with printf.  But then
55830dd9bf (Dec 2018) changed
mp_obj_new_exception_msg() to not format the message, as discussed
in #3004.  So such error strings are no longer formatted and a % is just
that.
2020-03-11 14:31:29 +11:00
Damien George 69661f3343 all: Reformat C and Python source code with tools/codeformat.py.
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-28 10:33:03 +11:00
Damien George ad7213d3c3 py: Add mp_raise_msg_varg helper and use it where appropriate.
This commit adds mp_raise_msg_varg(type, fmt, ...) as a helper for
nlr_raise(mp_obj_new_exception_msg_varg(type, fmt, ...)).  It makes the
C-level API for raising exceptions more consistent, and reduces code size
on most ports:

   bare-arm:   +28 +0.042%
minimal x86:  +100 +0.067%
   unix x64:   -56 -0.011%
unix nanbox:  -300 -0.068%
      stm32:  -204 -0.054% PYBV10
     cc3200:    +0 +0.000%
    esp8266:   -64 -0.010% GENERIC
      esp32:  -104 -0.007% GENERIC
        nrf:  -136 -0.094% pca10040
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2020-02-13 11:52:40 +11:00
Yonatan Goldschmidt d9433d3e94 py/obj.h: Add and use mp_obj_is_bool() helper.
Commit d96cfd13e3 introduced a regression in
testing for bool objects, that such objects were in some cases no longer
recognised and bools, eg when using mp_obj_is_type(o, &mp_type_bool), or
mp_obj_is_integer(o).

This commit fixes that problem by adding mp_obj_is_bool(o).  Builds with
MICROPY_OBJ_IMMEDIATE_OBJS enabled check if the object is any of the const
True or False objects.  Builds without it use the old method of ->type
checking, which compiles to smaller code (compared with the former
mentioned method).

Fixes #5538.
2020-01-24 10:53:45 +11:00
Damien George bfbd94401d py: Make mp_obj_get_type() return a const ptr to mp_obj_type_t.
Most types are in rodata/ROM, and mp_obj_base_t.type is a constant pointer,
so enforce this const-ness throughout the code base.  If a type ever needs
to be modified (eg a user type) then a simple cast can be used.
2020-01-09 11:25:26 +11:00
Damien George 4c0176d13f py/objstr: Don't use inline GET_STR_DATA_LEN for object-repr D.
Changing to use the helper function mp_obj_str_get_data_no_check() reduces
code size of nan-boxing builds by about 1000 bytes.
2019-12-27 23:15:52 +11:00
Jim Mussared c7ae8c5a99 py/objstr: Size-optimise failure path for mp_obj_str_get_buffer.
These fields are never looked at if the function returns non-zero.
2019-10-22 13:54:09 +11:00
Josh Lloyd 7d58a197cf py: Rename MP_QSTR_NULL to MP_QSTRnull to avoid intern collisions.
Fixes #5140.
2019-09-26 16:04:56 +10:00
Damien George eee1e8841a py: Downcase all MP_OBJ_IS_xxx macros to make a more consistent C API.
These macros could in principle be (inline) functions so it makes sense to
have them lower case, to match the other C API functions.

The remaining macros that are upper case are:
- MP_OBJ_TO_PTR, MP_OBJ_FROM_PTR
- MP_OBJ_NEW_SMALL_INT, MP_OBJ_SMALL_INT_VALUE
- MP_OBJ_NEW_QSTR, MP_OBJ_QSTR_VALUE
- MP_OBJ_FUN_MAKE_SIG
- MP_DECLARE_CONST_xxx
- MP_DEFINE_CONST_xxx

These must remain macros because they are used when defining const data (at
least, MP_OBJ_NEW_SMALL_INT is so it makes sense to have
MP_OBJ_SMALL_INT_VALUE also a macro).

For those macros that have been made lower case, compatibility macros are
provided for the old names so that users do not need to change their code
immediately.
2019-02-12 14:54:51 +11:00
Paul Sokolovsky 8fea833e3f py: Update my copyright info on some files.
Based on git history.
2019-02-06 00:19:00 +11:00
Paul Sokolovsky 5a91fce9f8 py/objstr: Make str.count() method configurable.
Configurable via MICROPY_PY_BUILTINS_STR_COUNT.  Default is enabled.
Disabled for bare-arm, minimal, unix-minimal and zephyr ports.  Disabling
it saves 408 bytes on x86.
2018-10-22 22:49:05 +11:00
Paul Sokolovsky a135bca4a1 py/objstr: format: Return bytes result for bytes format string.
This is an improvement over previous behavior when str was returned for
both str and bytes input format.  This new behaviour is also consistent
with how the % operator works, as well as many other str/bytes methods.

It should be noted that it's not how current versions of CPython work,
where there's a gap in the functionality and bytes.format() is not
supported.
2018-09-26 15:29:41 +10:00
Paul Sokolovsky 2da5d41350 py/objstr: Make % (__mod__) formatting operator configurable.
Default is enabled, disabled for minimal builds. Saves 1296 bytes on x86,
976 bytes on ARM.
2018-09-20 14:41:08 +10:00
Damien George b01f66c5f1 py: Shorten error messages by using contractions and some rewording. 2018-09-20 14:33:10 +10:00
Damien George aec6fa9160 py/objstr: In format error message, use common string with %s for type.
This error message did not consume all of its variable args, a bug
introduced long ago in baf6f14deb.  By fixing
it to use %s (instead of keeping the string as-is and deleting the last
arg) the same error message string is now reused three times in this format
function and gives a code size reduction of around 130 bytes.  It also now
gives a better error message when a non-string is passed in as an argument
to format, eg '{:d}'.format([]).
2018-07-30 12:46:47 +10:00
Jeff Epler d6cf5c6749 py/objstr: In find/rfind, don't crash when end < start. 2018-04-05 16:14:17 +10:00
Damien George 3280788195 py/runtime: Check that keys in dicts passed as ** args are strings.
Prior to this patch the code would crash if a key in a ** dict was anything
other than a str or qstr.  This is because mp_setup_code_state() assumes
that keys in kwargs are qstrs (for efficiency).

Thanks to @jepler for finding the bug.
2018-03-30 11:13:32 +11:00
Damien George 8769049e93 py/objstr: Remove unnecessary check for positive splits variable.
At this point in the code the variable "splits" is guaranteed to be
positive due to the check for "splits == 0" above it.
2018-02-20 19:19:02 +11:00
Damien George 4e469085c1 py/objstr: Protect against creating bytes(n) with n negative.
Prior to this patch uPy (on a 32-bit arch) would have severe issues when
calling bytes(-1): such a call would call vstr_init_len(vstr, -1) which
would then +1 on the len and call vstr_init(vstr, 0), which would then
round this up and allocate a small amount of memory for the vstr.  The
bytes constructor would then attempt to zero out all this memory, thinking
it had allocated 2^32-1 bytes.
2018-02-19 16:25:30 +11:00
Damien George 19aee9438a py/unicode: Clean up utf8 funcs and provide non-utf8 inline versions.
This patch provides inline versions of the utf8 helper functions for the
case when unicode is disabled (MICROPY_PY_BUILTINS_STR_UNICODE set to 0).
This saves code size.

The unichar_charlen function is also renamed to utf8_charlen to match the
other utf8 helper functions, and the signature of this function is adjusted
for consistency (const char* -> const byte*, mp_uint_t -> size_t).
2018-02-14 18:19:22 +11:00
Damien George 3990a52c0f py: Annotate func defs with NORETURN when their corresp decls have it. 2017-11-29 15:43:40 +11:00
Damien George 5e34a113ea py/runtime: Add MP_BINARY_OP_CONTAINS as reverse of MP_BINARY_OP_IN.
Before this patch MP_BINARY_OP_IN had two meanings: coming from bytecode it
meant that the args needed to be swapped, but coming from within the
runtime meant that the args were already in the correct order.  This lead
to some confusion in the code and comments stating how args were reversed.
It also lead to 2 bugs: 1) containment for a subclass of a native type
didn't work; 2) the expression "{True} in True" would illegally succeed and
return True.  In both of these cases it was because the args to
MP_BINARY_OP_IN ended up being reversed twice.

To fix these things this patch introduces MP_BINARY_OP_CONTAINS which
corresponds exactly to the __contains__ special method, and this is the
operator that built-in types should implement.  MP_BINARY_OP_IN is now only
emitted by the compiler and is converted to MP_BINARY_OP_CONTAINS by
swapping the arguments.
2017-11-24 14:48:23 +11:00
Damien George 8d956c26d1 py/objstr: When constructing str from bytes, check for existing qstr.
This patch uses existing qstr data where possible when constructing a str
from a bytes object.
2017-11-16 14:02:28 +11:00
Damien George 1f1d5194d7 py/objstr: Make mp_obj_new_str_of_type check for existing interned qstr.
The function mp_obj_new_str_of_type is a general str object constructor
used in many places in the code to create either a str or bytes object.
When creating a str it should first check if the string data already exists
as an interned qstr, and if so then return the qstr object.  This patch
makes the function have such behaviour, which helps to reduce heap usage by
reusing existing interned data where possible.

The old behaviour of mp_obj_new_str_of_type (which didn't check for
existing interned data) is made available through the function
mp_obj_new_str_copy, but should only be used in very special cases.

One consequence of this patch is that the following expression is now True:

    'abc' is ' abc '.split()[0]
2017-11-16 13:53:04 +11:00
Damien George 4601759bf5 py/objstr: Remove "make_qstr_if_not_already" arg from mp_obj_new_str.
This patch simplifies the str creation API to favour the common case of
creating a str object that is not forced to be interned.  To force
interning of a new str the new mp_obj_new_str_via_qstr function is added,
and should only be used if warranted.

Apart from simplifying the mp_obj_new_str function (and making it have the
same signature as mp_obj_new_bytes), this patch also reduces code size by a
bit (-16 bytes for bare-arm and roughly -40 bytes on the bare-metal archs).
2017-11-16 13:17:51 +11:00
Damien George dfa563c71f py/objstr: Make empty bytes object have a null-terminating byte.
Because a lot of string processing functions assume there is a null
terminating byte, so they can work in an efficient way.

Fixes issue #3334.
2017-10-04 17:59:22 +11:00
Damien George a3dc1b1957 all: Remove inclusion of internal py header files.
Header files that are considered internal to the py core and should not
normally be included directly are:
    py/nlr.h - internal nlr configuration and declarations
    py/bc0.h - contains bytecode macro definitions
    py/runtime0.h - contains basic runtime enums

Instead, the top-level header files to include are one of:
    py/obj.h - includes runtime0.h and defines everything to use the
        mp_obj_t type
    py/runtime.h - includes mpstate.h and hence nlr.h, obj.h, runtime0.h,
        and defines everything to use the general runtime support functions

Additional, specific headers (eg py/objlist.h) can be included if needed.
2017-10-04 12:37:50 +11:00
Paul Sokolovsky fc9a6dd09e py/objstr: strip: Don't strip "\0" by default.
An issue was due to incorrectly taking size of default strip characters
set.
2017-09-19 21:21:12 +03:00
tll 68c28174d0 py/objstr: Add check for valid UTF-8 when making a str from bytes.
This patch adds a function utf8_check() to check for a valid UTF-8 encoded
string, and calls it when constructing a str from raw bytes.  The feature
is selectable at compile time via MICROPY_PY_BUILTINS_STR_UNICODE_CHECK and
is enabled if unicode is enabled.  It costs about 110 bytes on Thumb-2, 150
bytes on Xtensa and 170 bytes on x86-64.
2017-09-06 16:43:09 +10:00
Damien George 58321dd985 all: Convert mp_uint_t to mp_unary_op_t/mp_binary_op_t where appropriate
The unary-op/binary-op enums are already defined, and there are no
arithmetic tricks used with these types, so it makes sense to use the
correct enum type for arguments that take these values.  It also reduces
code size quite a bit for nan-boxing builds.
2017-08-29 13:16:30 +10:00
Paul Sokolovsky 37379a2974 py/objstr: startswith, endswith: Check arg to be a string.
Otherwise, it will silently get incorrect result on other values types,
including CPython tuple form like "foo.png".endswith(("png", "jpg"))
(which MicroPython doesn't support for unbloatedness).
2017-08-29 00:06:21 +03:00
Javier Candeira 35a1fea90b all: Raise exceptions via mp_raise_XXX
- Changed: ValueError, TypeError, NotImplementedError
  - OSError invocations unchanged, because the corresponding utility
    function takes ints, not strings like the long form invocation.
  - OverflowError, IndexError and RuntimeError etc. not changed for now
    until we decide whether to add new utility functions.
2017-08-13 22:52:33 +10:00
Damien George 3d25d9c7d9 py/objstr: Raise an exception for wrong type on RHS of str binary op.
The main case to catch is invalid types for the containment operator, of
the form str.__contains__(non-str).
2017-08-09 21:25:48 +10:00