Commit Graph

32 Commits

Author SHA1 Message Date
Dan Halbert 367e13c69f change CIRCUITPY change markers to CIRCUITPY-CHANGE 2023-10-19 16:42:36 -04:00
Dan Halbert f2ebe6839c Initial MicroPython v1.21.0 merge; not compiled yet 2023-10-18 17:49:14 -04:00
Daniël van de Giessen d1f288c041 py/modstruct: Support pad bytes in struct format.
This adds support for the x format code in struct.pack and struct.unpack.

The primary use case for this is ignoring bytes while unpacking.  When
interfacing with existing systems, it may often happen that you either have
fields in a struct that aren't properly specified or you simply don't care
about them.  Being able to easily skip them is useful.

Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2023-09-01 14:50:49 +10:00
Dan Halbert 88c22d5052 remove last uses of 'u' prefix 2023-08-22 12:57:47 -04:00
Jim Mussared 4216bc7d13 tests: Replace umodule with module everywhere.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Damien George b1229efbd1 all: Fix spelling mistakes based on codespell check.
Signed-off-by: Damien George <damien@micropython.org>
2023-04-27 18:03:06 +10:00
hathach fecc1bdedb
fix typos (partial) detected by codepell 2023-03-18 22:17:02 +07:00
Scott Shawcroft 5ccd5dc08b
Fix struct.pack with padding bytes
It used to validate the following arg could fit in a single byte.
Now, it always uses zero to pad.
2021-04-16 12:39:23 -07:00
Jeff Epler 54d97251fe modstruct: Improve compliance with python3
While checking whether we can enable -Wimplicit-fallthrough, I encountered
a diagnostic in mp_binary_set_val_array_from_int which led to discovering
the following bug:
```
>>> struct.pack("xb", 3)
b'\x03\x03'
```
That is, the next value (3) was used as the value of a padding byte, while
standard Python always fills "x" bytes with zeros.  I initially thought
this had to do with the unintentional fallthrough, but it doesn't.
Instead, this code would relate to an array.array with a typecode of
padding ('x'), which is ALSO not desktop Python compliant:
```
>>> array.array('x', (1, 2, 3))
array('x', [1, 0, 0])
```
Possibly this is dead code that used to be shared between struct-setting
and array-setting, but it no longer is.

I also discovered that the argument list length for struct.pack
and struct.pack_into were not checked, and that the length of binary data
passed to array.array was not checked to be a multiple of the element
size.

I have corrected all of these to conform more closely to standard Python
and revised some tests where necessary.  Some tests for micropython-specific
behavior that does not conform to standard Python and is not present
in CircuitPython was deleted outright.
2020-09-12 14:07:23 -05:00
Damien George 2daacc5cee py/modstruct: Check and prevent buffer-write overflow in struct packing.
Prior to this patch, the size of the buffer given to pack_into() was checked
for being too small by using the count of the arguments, not their actual
size.  For example, a format spec of '4I' would only check that there was 4
bytes available, not 16; and 'I' would check for 1 byte, not 4.

The pack() function is ok because its buffer is created to be exactly the
correct size.

The fix in this patch calculates the total size of the format spec at the
start of pack_into() and verifies that the buffer is large enough.  This
adds some computational overhead, to iterate through the whole format spec.
The alternative is to check during the packing, but that requires extra
code to handle alignment, and the check is anyway not needed for pack().
So to maintain minimal code size the check is done using struct_calcsize.
2017-09-01 11:11:09 +10:00
Damien George 79d5acbd01 py/modstruct: Check and prevent buffer-read overflow in struct unpacking
Prior to this patch, the size of the buffer given to unpack/unpack_from was
checked for being too small by using the count of the arguments, not their
actual size.  For example, a format spec of '4I' would only check that
there was 4 bytes available, not 16; and 'I' would check for 1 byte, not 4.

This bug is fixed in this patch by calculating the total size of the format
spec at the start of the unpacking function.  This function anyway needs to
calculate the number of items at the start, so calculating the total size
can be done at the same time.
2017-09-01 10:53:29 +10:00
Paul Sokolovsky a2803b74f4 tests/basics: Convert "sys.exit()" to "raise SystemExit". 2017-06-10 20:03:01 +03:00
Paul Sokolovsky 3ab6aa3a6d tests/basic: Split tests into working with small ints and not working.
Tests which don't work with small ints are suffixed with _intbig.py. Some
of these may still work with long long ints and need to be reclassified
later.
2017-03-04 00:13:27 +03:00
Rami Ali 65574f817a tests/basics: Add tests to improve coverage of binary.c. 2016-12-28 16:11:54 +11:00
Paul Sokolovsky 91359c8690 tests/struct*: Make skippable. 2016-12-19 19:41:12 +03:00
Damien George 82af4d6749 tests: Improve coverage of struct with test for non-compliant behaviour. 2016-10-07 12:57:25 +11:00
Paul Sokolovsky b85bcd671c tests/struct1: Test "l" specifier to improve coverage. 2016-09-19 17:01:02 +03:00
Paul Sokolovsky 719f8c044a tests/struct1: Add testcase for an unknown type char. 2016-05-14 15:54:09 +03:00
Dave Hylands a17755ee8b py: Add ustruct.pack_into and unpack_from 2016-01-19 12:25:28 +00:00
Paul Sokolovsky 0bd3f3291d tests/base/struct1.py: Add test for repetition counters. 2015-10-31 18:55:31 +03:00
Damien George a81539db25 tests: Add further tests for mpz code. 2015-10-01 18:49:37 +01:00
Damien George 25afc7da0d tests: Add tests to improve coverage of objstr.c. 2015-09-03 23:06:18 +01:00
Paul Sokolovsky 3d3ef36e97 modstruct: Rename module to "ustruct", to allow full Python-level impl. 2015-05-04 16:53:52 +03:00
Damien George 271d18eb08 py: Support conversion of bignum to bytes.
This gets int.to_bytes working for bignum, and also struct.pack with 'q'
and 'Q' args on 32-bit machines.

Addresses issue #1155.
2015-04-25 23:16:39 +01:00
Damien George e5c4362a98 tests: Add some more tests to improve code coverage of corner cases. 2015-04-05 00:03:43 +01:00
Damien George be6d8be91e py: Rename mp_obj_int_get to mp_obj_int_get_truncated; fix struct.pack.
mp_obj_int_get_truncated is used as a "fast path" int accessor that
doesn't check for overflow and returns the int truncated to the machine
word size, ie mp_int_t.

Use mp_obj_int_get_truncated to fix struct.pack when packing maximum word
sized values.

Addresses issues #779 and #998.
2014-12-05 23:13:52 +00:00
Paul Sokolovsky 7a2f166949 modstruct: Fix alignment handling issues.
Also, factor out mp_binary_get_int() function.
2014-06-25 23:34:44 +03:00
Paul Sokolovsky df94b717b4 modstruct: Implement count specifier for strings (e.g. "100s").
Infra for counts of other types is there, need last mile to be implemented.
2014-05-12 23:45:50 +03:00
Paul Sokolovsky 6204460461 modstruct: Initial implementation of struct.pack(). 2014-04-19 03:22:08 +03:00
Paul Sokolovsky 0c43cf9154 modstruct: Basic implementation of native struct alignment and types. 2014-04-11 03:59:16 +03:00
Paul Sokolovsky 6582d64d01 modstruct: Refactor to support both LE and BE packed structs. 2014-04-11 03:58:49 +03:00
Paul Sokolovsky e9db840480 py: Start implementing "struct" module.
Only calcsize() and unpack() functions provided so far, for little-endian
byte order. Format strings don't support repition spec (like "2b3i").

Unfortunately, dealing with all the various binary type sizes and alignments
will lead to quite a bloated "binary" helper functions  - if optimizing for
speed. Need to think if using dynamic parametrized algos makes more sense.
2014-04-10 03:58:03 +03:00