Commit Graph

39 Commits

Author SHA1 Message Date
Jeff Epler f1c2dee1c0 style 2020-01-08 16:36:43 -06:00
Jeff Epler 5baaac55ce vstr_init_len: Don't crash if (size_t)-1 is passed
In this unusual case, (len + 1) is zero, the allocation in vstr_init
succeeds (allocating 1 byte), and then the caller is likely to erroneously
access outside the allocated region, for instance with a memset().

This could be triggered with os.urandom(-1) after it was converted to use
mp_obj_new_bytes_of_zeros.
2020-01-08 09:42:44 -06:00
Damien George ede8a0235b py/vstr: Raise a RuntimeError if fixed vstr buffer overflows.
Current users of fixed vstr buffers (building file paths) assume that there
is no overflow and do not check for overflow after building the vstr.  This
has the potential to lead to NULL pointer dereferences
(when vstr_null_terminated_str returns NULL because it can't allocate RAM
for the terminating byte) and stat'ing and loading invalid path names (due
to the path being truncated).  The safest and simplest thing to do in these
cases is just raise an exception if a write goes beyond the end of a fixed
vstr buffer, which is what this patch does.  It also simplifies the vstr
code.
2017-09-21 20:29:41 +10:00
Alexander Steffen 55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Tom Collins 145796f037 py,extmod: Some casts and minor refactors to quiet compiler warnings. 2017-07-07 11:32:22 +10:00
Damien George 824f5c5a32 py/vstr: Combine vstr_new_size with vstr_new since they are rarely used.
Now there is just one function to allocate a new vstr, namely vstr_new
(in addition to vstr_init etc).  The caller of this function should know
what initial size to allocate for the buffer, or at least have some policy
or config option, instead of leaving it to a default (as it was before).
2016-10-14 16:46:34 +11:00
Damien George 5da0d29d3c py/vstr: Remove vstr.had_error flag and inline basic vstr functions.
The vstr.had_error flag was a relic from the very early days which assumed
that the malloc functions (eg m_new, m_renew) returned NULL if they failed
to allocate.  But that's no longer the case: these functions will raise an
exception if they fail.

Since it was impossible for had_error to be set, this patch introduces no
change in behaviour.

An alternative option would be to change the malloc calls to the _maybe
variants, which return NULL instead of raising, but then a lot of code
will need to explicitly check if the vstr had an error and raise if it
did.

The code-size savings for this patch are, in bytes: bare-arm:188,
minimal:456, unix(NDEBUG,x86-64):368, stmhal:228, esp8266:360.
2016-09-19 12:28:55 +10:00
Paul Sokolovsky 6f34e138f1 py/vstr: Change allocation policy, +16 to requested size, instead of *2.
Effect measured on esp8266 port:

Before:
>>> pystone_lowmem.main(10000)
Pystone(1.2) time for 10000 passes = 44214 ms
This machine benchmarks at 226 pystones/second
>>> pystone_lowmem.main(10000)
Pystone(1.2) time for 10000 passes = 44246 ms
This machine benchmarks at 226 pystones/second

After:
>>> pystone_lowmem.main(10000)
Pystone(1.2) time for 10000 passes = 44343ms
This machine benchmarks at 225 pystones/second
>>> pystone_lowmem.main(10000)
Pystone(1.2) time for 10000 passes = 44376ms
This machine benchmarks at 225 pystones/second
2016-05-10 00:56:51 +03:00
Paul Sokolovsky a1f2245a81 py/vstr: vstr_null_terminated_str(): Extend string by at most one byte.
vstr_null_terminated_str is almost certainly a vstr finalization operation,
so it should add the requested NUL byte, and not try to pre-allocate more.
The previous implementation could actually allocate double of the buffer
size.
2016-05-09 22:39:57 +03:00
Dave Hylands 9f76dcd682 py: Prevent many extra vstr allocations.
I checked the entire codebase, and every place that vstr_init_len
was called, there was a call to mp_obj_new_str_from_vstr after it.

mp_obj_new_str_from_vstr always tries to reallocate a new buffer
1 byte larger than the original to store the terminating null
character.

In many cases, if we allocated the initial buffer to be 1 byte
longer, we can prevent this extra allocation, and just reuse
the originally allocated buffer.

Asking to read 256 bytes and only getting 100 will still cause
the extra allocation, but if you ask to read 256 and get 256
then the extra allocation will be optimized away.

Yes - the reallocation is optimized in the heap to try and reuse
the buffer if it can, but it takes quite a few cycles to figure
this out.

Note by Damien: vstr_init_len should now be considered as a
string-init convenience function and used only when creating
null-terminated objects.
2015-07-06 17:29:27 +01:00
Damien George 1c9a499135 py/vstr.c: Allow vstr_printf to print correctly to a fixed buffer.
This patch allows vstr_printf to use all the available space of a fixed
vstr buffer.  vstr_printf is a good alternative to snprintf.
2015-04-18 14:23:13 +01:00
Damien George 7f9d1d6ab9 py: Overhaul and simplify printf/pfenv mechanism.
Previous to this patch the printing mechanism was a bit of a tangled
mess.  This patch attempts to consolidate printing into one interface.

All (non-debug) printing now uses the mp_print* family of functions,
mainly mp_printf.  All these functions take an mp_print_t structure as
their first argument, and this structure defines the printing backend
through the "print_strn" function of said structure.

Printing from the uPy core can reach the platform-defined print code via
two paths: either through mp_sys_stdout_obj (defined pert port) in
conjunction with mp_stream_write; or through the mp_plat_print structure
which uses the MP_PLAT_PRINT_STRN macro to define how string are printed
on the platform.  The former is only used when MICROPY_PY_IO is defined.

With this new scheme printing is generally more efficient (less layers
to go through, less arguments to pass), and, given an mp_print_t*
structure, one can call mp_print_str for efficiency instead of
mp_printf("%s", ...).  Code size is also reduced by around 200 bytes on
Thumb2 archs.
2015-04-16 14:30:16 +00:00
Damien George 827b0f747b py: Change vstr_null_terminate -> vstr_null_terminated_str, returns str. 2015-01-29 13:57:23 +00:00
Damien George 0d3cb6726d py: Change vstr so that it doesn't null terminate buffer by default.
This cleans up vstr so that it's a pure "variable buffer", and the user
can decide whether they need to add a terminating null byte.  In most
places where vstr is used, the vstr did not need to be null terminated
and so this patch saves code size, a tiny bit of RAM, and makes vstr
usage more efficient.  When null termination is needed it must be
done explicitly using vstr_null_terminate.
2015-01-28 23:43:01 +00:00
Damien George 77089bebd4 py: Add comments for vstr_init and mp_obj_new_str. 2015-01-21 23:18:02 +00:00
Damien George 05005f679e py: Remove mp_obj_str_builder and use vstr instead.
With this patch str/bytes construction is streamlined.  Always use a
vstr to build a str/bytes object.  If the size is known beforehand then
use vstr_init_len to allocate only required memory.  Otherwise use
vstr_init and the vstr will grow as needed.  Then use
mp_obj_new_str_from_vstr to create a str/bytes object using the vstr
memory.

Saves code ROM: 68 bytes on stmhal, 108 bytes on bare-arm, and 336 bytes
on unix x64.
2015-01-21 23:18:02 +00:00
Damien George 0b9ee86133 py: Add mp_obj_new_str_from_vstr, and use it where relevant.
This patch allows to reuse vstr memory when creating str/bytes object.
This improves memory usage.

Also saves code ROM: 128 bytes on stmhal, 92 bytes on bare-arm, and 88
bytes on unix x64.
2015-01-21 23:17:27 +00:00
Damien George 963a5a3e82 py, unix: Allow to compile with -Wsign-compare.
See issue #699.
2015-01-16 17:47:07 +00:00
Damien George 51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Damien George 969a6b37bf py: Make functions static where appropriate. 2014-12-10 22:08:14 +00:00
Damien George 42f3de924b py: Convert [u]int to mp_[u]int_t where appropriate.
Addressing issue #50.
2014-10-03 17:44:14 +00:00
Damien George b0261341d3 py: For malloc and vstr functions, use size_t exclusively for int type.
It seems most sensible to use size_t for measuring "number of bytes" in
malloc and vstr functions (since that's what size_t is for).  We don't
use mp_uint_t because malloc and vstr are not Micro Python specific.
2014-09-25 15:49:26 +01:00
Paul Sokolovsky 165eb69b86 vstr: Restore bytestr compatibility. 2014-06-27 00:04:18 +03:00
Chris Angelico 2ba2299d28 lexer, vstr: Add unicode support. 2014-06-27 00:04:18 +03:00
Paul Sokolovsky 59c675a64c py: Include mpconfig.h before all other includes.
It defines types used by all other headers.

Fixes #691.
2014-06-21 22:43:22 +03:00
Damien George 04b9147e15 Add license header to (almost) all files.
Blanket wide to all .c and .h files.  Some files originating from ST are
difficult to deal with (license wise) so it was left out of those.

Also merged modpyb.h, modos.h, modstm.h and modtime.h in stmhal/.
2014-05-03 23:27:38 +01:00
Damien George 8cd72bdf92 py: Fix vstr_init for case that alloc = 0. 2014-03-31 17:10:59 +01:00
xbe efe3422394 py: Clean up includes.
Remove unnecessary includes. Add includes that improve portability.
2014-03-17 02:43:40 -07:00
Damien George ecd58aec08 py: Fix bug in vstr_ins_blank_bytes. 2014-03-15 16:54:06 +00:00
Damien George 280e7208d8 Add vstr_ins and vstr_cut_out; improve stmhal readline. 2014-03-15 14:33:09 +00:00
Paul Sokolovsky 520e2f58a5 Replace global "static" -> "STATIC", to allow "analysis builds". Part 2. 2014-02-12 18:31:30 +02:00
Damien George 354d15a964 Implement fixed buffer vstrs; use them for import path. 2014-02-06 21:11:19 +00:00
Damien George f64086f80f Fix 1 warning and 1 bug. 2014-01-22 23:18:50 +00:00
Paul Sokolovsky 5d2499c638 Add "buffer management" and "shrink" API calls to vstr.
vstr is initially intended to deal with arbitrary-length strings. By
providing a bit lower-level API calls, it will be also useful to deal
with arbitrary-length I/O buffers (the difference from strings is that
buffers are filled from "outside", via I/O).

Another issue, especially aggravated by I/O buffer use, is alloc size
vs actual size length. If allocated 1Mb for buffer, but actually
read 1 byte, we don't want to keep rest of 1Mb be locked by this I/O
result, but rather return it to heap ASAP ("shrink" buffer before passing
it to qstr_from_str_take()).
2014-01-13 23:31:06 +02:00
Damien George 66028ab6dc Basic implementation of import.
import works for simple cases.  Still work to do on finding the right
script, and setting globals/locals correctly when running an imported
function.
2014-01-03 14:03:48 +00:00
Damien 732407f1bf Change memory allocation API to require size for free and realloc. 2013-12-29 19:33:23 +00:00
Damien 2f06c57f8a Add simple var-arg functions; add simple string.format. 2013-11-03 18:20:56 +00:00
Damien 8b3a7c2237 Fix func decls with no arguments: () -> (void). 2013-10-23 20:20:17 +01:00
Damien 68f59a96e3 Add vstr and its functions. 2013-10-20 14:39:58 +01:00