djsundog/circuitpython
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

extmod/moddeflate: Change default window size.

Browse Source 
The primary purpose of this commit is to make decompress default to
wbits=15 when the format is gzip (or auto format with gzip detected). The
idea is that someone decompressing a gzip stream should be able to use the
default `deflate.DeflateIO(f)` and it will "just work" for any input
stream, even though it uses a lot of memory.

This is done by making uzlib report gzip files as having wbits set to 15
in their header (where it previously only set the wbits out parameter for
zlib files), and then fixing up the logic in `deflateio_init_read`.

Updates the documentation to match.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This commit is contained in:
Jim Mussared 2023-08-31 14:29:49 +10:00
parent 81c19d93bc
commit 32db4c58f7
3 changed files with 68 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

79
docs/library/deflate.rst
View File

@ -41,9 +41,15 @@ Classes
to 1024 bytes. Valid values are ``5`` to ``15`` inclusive (corresponding to
window sizes of 32 to 32k bytes).
If *wbits* is set to ``0`` (the default), then a window size of 256 bytes
will be used (corresponding to *wbits* set to ``8``), except when
:ref:`decompressing a zlib stream <deflate_wbits_zlib>`.
If *wbits* is set to ``0`` (the default), then for compression a window size
of 256 bytes will be used (as if *wbits* was set to 8). For decompression, it
depends on the format:
* ``RAW`` will use 256 bytes (corresponding to *wbits* set to 8).
* ``ZLIB`` (or ``AUTO`` with zlib detected) will use the value from the zlib
header.
* ``GZIP`` (or ``AUTO`` with gzip detected) will use 32 kilobytes
(corresponding to *wbits* set to 15).
See the :ref:`window size <deflate_wbits>` notes below for more information
about the window size, zlib, and gzip streams.
@ -134,44 +140,43 @@ Deflate window size
-------------------
The window size limits how far back in the stream the (de)compressor can
reference. Increasing the window size will improve compression, but will
require more memory.
reference. Increasing the window size will improve compression, but will require
more memory and make the compressor slower.
However, just because a given window size is used for compression, this does not
mean that the stream will require the same size window for decompression, as
the stream may not reference data as far back as the window allows (for example,
if the length of the input is smaller than the window size).
If an input stream was compressed a given window size, then `DeflateIO`
using a smaller window size will fail mid-way during decompression with
:exc:`OSError`, but only if a back-reference actually refers back further
than the decompressor's window size. This means it may be possible to decompress
with a smaller window size. For example, this would trivially be the case if the
original uncompressed data is shorter than the window size.
If the decompressor uses a smaller window size than necessary for the input data
stream, it will fail mid-way through decompression with :exc:`OSError`.
Decompression
~~~~~~~~~~~~~
.. _deflate_wbits_zlib:
The zlib format includes a header which specifies the window size that was used
to compress the data. This indicates the maximum window size required to
decompress this stream. If this header value is less than the specified *wbits*
value (or if *wbits* is unset), then the header value will be used.
The zlib format includes a header which specifies the window size used to
compress the data (which due to the above, may be larger than the size required
for the decompressor).
The gzip format does not include the window size in the header, and assumes that
all gzip compressors (e.g. the ``gzip`` utility, or CPython's implementation of
:class:`gzip.GzipFile`) use the maximum window size of 32kiB. For this reason,
if the *wbits* parameter is not set, the decompressor will use a 32 kiB window
size (corresponding to *wbits* set to 15). This means that to be able to
decompress an arbitrary gzip stream, you must have at least this much RAM
available. If you control the source data, consider instead using the zlib
format with a smaller window size.
If this header value is lower than the specified *wbits* value, then the header
value will be used instead in order to reduce the memory allocation size. If
the *wbits* parameter is zero (the default), then the header value will only be
used if it is less than the maximum value of ``15`` (which is default value
used by most compressors [#f1]_).
The raw format has no header and therefore does not include any information
about the window size. If *wbits* is not set, then it will default to a window
size of 256 bytes, which may not be large enough for a given stream. Therefore
it is recommended that you should always explicitly set *wbits* if using the raw
format.
In other words, if the source zlib stream has been compressed with a custom window
size (i.e. less than ``15``), then using the default *wbits* parameter of zero
will decompress any such stream.
Compression
~~~~~~~~~~~
The gzip file format does not include the window size in the header.
Additionally, most compressor libraries (including CPython's implementation
of :class:`gzip.GzipFile`) will default to the maximum possible window size.
This makes it difficult to decompress most gzip streams on MicroPython unless
your board has a lot of free RAM.
If you control the source of the compressed data, then prefer to use the zlib
format, with a window size that is suitable for your target device.
.. rubric:: Footnotes
.. [#f1] The assumption here is that if the header value is the default used by
most compressors, then nothing is known about the likely required window
size and we should ignore it.
For compression, MicroPython will default to a window size of 256 bytes for all
formats. This provides a reasonable amount of compression with minimal memory
usage and fast compression time, and will generate output that will work with
any decompressor.

33
extmod/moddeflate.c
View File

@ -54,6 +54,8 @@ typedef enum {
DEFLATEIO_FORMAT_MAX = DEFLATEIO_FORMAT_GZIP,
} deflateio_format_t;
// This is used when the wbits is unset in the DeflateIO constructor. Default
// to the smallest window size (faster compression, less RAM usage, etc).
const int DEFLATEIO_DEFAULT_WBITS = 8;
typedef struct {
@ -114,24 +116,32 @@ STATIC bool deflateio_init_read(mp_obj_deflateio_t *self) {
// Don't modify self->window_bits as it may also be used for write.
int wbits = self->window_bits;
// Parse the header if we're in NONE/ZLIB/GZIP modes.
if (self->format != DEFLATEIO_FORMAT_RAW) {
int header_wbits = wbits;
if (self->format == DEFLATEIO_FORMAT_RAW) {
if (wbits == 0) {
// The docs recommends always setting wbits explicitly when using
// RAW, but we still allow a default.
wbits = DEFLATEIO_DEFAULT_WBITS;
}
} else {
// Parse the header if we're in NONE/ZLIB/GZIP modes.
int header_wbits;
int header_type = uzlib_parse_zlib_gzip_header(&self->read->decomp, &header_wbits);
if ((self->format == DEFLATEIO_FORMAT_ZLIB && header_type != UZLIB_HEADER_ZLIB) || (self->format == DEFLATEIO_FORMAT_GZIP && header_type != UZLIB_HEADER_GZIP)) {
if (header_type < 0) {
// Stream header was invalid.
return false;
}
if (wbits == 0 && header_wbits < 15) {
// If the header specified something lower than the default, then
// use that instead.
if ((self->format == DEFLATEIO_FORMAT_ZLIB && header_type != UZLIB_HEADER_ZLIB) || (self->format == DEFLATEIO_FORMAT_GZIP && header_type != UZLIB_HEADER_GZIP)) {
// Not what we expected.
return false;
}
// header_wbits will either be 15 (gzip) or 8-15 (zlib).
if (wbits == 0 || header_wbits < wbits) {
// If the header specified something lower, then use that instead.
// No point doing a bigger allocation than we need to.
wbits = header_wbits;
}
}
if (wbits == 0) {
wbits = DEFLATEIO_DEFAULT_WBITS;
}
size_t window_len = 1 << wbits;
self->read->window = m_new(uint8_t, window_len);
@ -163,6 +173,7 @@ STATIC bool deflateio_init_write(mp_obj_deflateio_t *self) {
int wbits = self->window_bits;
if (wbits == 0) {
// Same default wbits for all formats.
wbits = DEFLATEIO_DEFAULT_WBITS;
}
size_t window_len = 1 << wbits;

4
lib/uzlib/header.c
View File

@ -108,6 +108,10 @@ int uzlib_parse_zlib_gzip_header(uzlib_uncomp_t *d, int *wbits)
d->checksum_type = UZLIB_CHKSUM_CRC;
d->checksum = ~0;
/* gzip does not include the window size in the header, as it is expected that a
compressor will use wbits=15 (32kiB).*/
*wbits = 15;
return UZLIB_HEADER_GZIP;
} else {
/* check checksum */