c4feb806e0
The compression algorithm implemented in this commit uses much less memory compared to the standard way of implementing it using a hash table and large look-back window. In particular the algorithm here doesn't allocate hash table to store indices into the history of the previously seen text. Instead it simply does a brute-force-search of the history text to find a match for the compressor. This is slower (linear search vs hash table lookup) but with a small enough history (eg 512 bytes) it's not that slow. And a small history does not impact the compression too much. To give some more concrete numbers comparing memory use between the approaches: - Standard approach: inplace compression, all text to compress must be in RAM (or at least memory addressable), and then an additional 16k bytes RAM of hash table pointers, pointing into the text - The approach in this commit: streaming compression, only a limited amount of previous text must be in RAM (user selectable, defaults to 512 bytes). To compress, say, 1k of data, the standard approach requires all that data to be in RAM, plus an additional 16k of RAM for the hash table pointers. With this commit, you only need the 1k of data in RAM. Or if it's streaming from a file (or elsewhere), you could get away with only 256 bytes of RAM for the sliding history and still get very decent compression. In summary: because compression takes such a large amount of RAM (in the standard algorithm) and it's not really suitable for microcontrollers, the approach taken in this commit is to minimise RAM usage as much as possible, and still have acceptable performance (speed and compression ratio). Signed-off-by: Damien George <damien@micropython.org> |
||
|---|---|---|
| .. | ||
| asf4@84f56af132 | ||
| axtls@531cab9c27 | ||
| berkeley-db-1.xx@35aaec4418 | ||
| btstack@77e752abd6 | ||
| cmsis/inc | ||
| crypto-algorithms | ||
| cyw43-driver@8ef38a6d32 | ||
| fsp@e78939d32d | ||
| libffi@e9de7e35f2 | ||
| libhydrogen@5c5d513093 | ||
| libm | ||
| libm_dbl | ||
| littlefs | ||
| lwip@6ca936f6b5 | ||
| mbedtls@981743de6f | ||
| mbedtls_errors | ||
| micropython-lib@c113611765 | ||
| mynewt-nimble@42849560ba | ||
| nrfx@7a4c9d946c | ||
| nxp_driver@fa5a554c79 | ||
| oofatfs | ||
| pico-sdk@6a7db34ff6 | ||
| re1.5 | ||
| stm32lib@928df866e4 | ||
| tinytest | ||
| tinyusb@868f2bcda0 | ||
| uzlib | ||
| wiznet5k@0803fc519a | ||
| README.md | ||
This directory contains third-party, low-level C libraries and SDKs. Libraries that do not target any specific platform are generally chosen based on them being independent and efficient.